From anthony at ekit-inc.com Mon Dec 1 01:15:46 2003 From: anthony at ekit-inc.com (Anthony Baxter) Date: Mon Dec 1 01:16:10 2003 Subject: [Python-Dev] backticks delenda est In-Reply-To: Message from "Raymond Hettinger" of "Sun, 30 Nov 2003 23:42:55 CDT." <003401c3b7c5$96b39e20$e841fea9@oemcomputer> Message-ID: <200312010615.hB16FlJ8019622@maxim.off.ekorp.com> >>> "Raymond Hettinger" wrote > Advisory from a micro-performance hawk: Backticks are faster than > repr() > > >>> from timeit import Timer > >>> min(Timer('`x`', 'x=1').repeat(3)) > 1.4857213496706265 > >>> min(Timer('repr(x)', 'x=1').repeat(3)) > 1.7748914665012876 Presumably because the backtick version doesn't have to look up 'repr', then call it. >>> def f1(a): ... repr(a) ... >>> >>> def f2(a): ... `a` ... >>> import dis >>> dis.dis(f1) 2 0 LOAD_GLOBAL 0 (repr) 3 LOAD_FAST 0 (a) 6 CALL_FUNCTION 1 9 POP_TOP 10 LOAD_CONST 0 (None) 13 RETURN_VALUE >>> dis.dis(f2) 2 0 LOAD_FAST 0 (a) 3 UNARY_CONVERT 4 POP_TOP 5 LOAD_CONST 0 (None) 8 RETURN_VALUE This is presumably something that a future optimiser could "fix", once we have the no-shadowing-builtins rule enforced. I don't think this is a serious enough problem to merit leaving the backticks in the std lib. I find them annoying, and hard to read. In general, the backticks are in debugging and error handling code, so the performance penalty should be negligible. Anthony From theller at python.net Mon Dec 1 06:43:43 2003 From: theller at python.net (Thomas Heller) Date: Mon Dec 1 06:43:57 2003 Subject: [Python-Dev] "groupby" iterator In-Reply-To: <200312010018.hB10IbS29532@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Sun, 30 Nov 2003 16:18:37 -0800") References: <20031130073109.GA1560@hishome.net> <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <002701c3b606$c61304a0$e841fea9@oemcomputer> <20031130073109.GA1560@hishome.net> <5.1.0.14.0.20031130184217.02e3c1d0@mail.telecommunity.com> <200312010018.hB10IbS29532@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: >> How about: >> >> extract(attr='grade') >> extract(item=2) >> extract(method='foo') # returns the result of calling 'ob.foo()' >> >> And following the pattern of Zope's old "query" package: >> >> extract(extract(attr='foo'), attr='bar') # extracts ob.foo.bar >> extract(extract(item=10), method='spam') # extracts ob[10].spam() >> >> i.e., the first (optional) positional argument to extract is a function >> that's called on the outer extract's argument, and the return value is then >> used to perform the main extract operation on. > > I'm not sure what the advantage of this is. It seems more typing, > more explanation, probably more code to implement (to check for > contradicting keyword args). Hm, couldn't "lambda ob: ob.foo.bar" return exactly the same thing as "extract(extract(attr='foo'), attr='bar')" ? In other words: return specialized C implemented functions for simple lambda expressions? Thomas From walter at livinglogic.de Mon Dec 1 07:59:44 2003 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon Dec 1 07:59:52 2003 Subject: [Python-Dev] Banishing apply(), buffer(), coerce(), and intern() In-Reply-To: <000201c3b71b$9facae40$e841fea9@oemcomputer> References: <000201c3b71b$9facae40$e841fea9@oemcomputer> Message-ID: <3FCB3B40.5070703@livinglogic.de> Raymond Hettinger wrote: > [...] >>As always, be careful with doing peephole changes to the standard >>library -- historically, we've seen a 1-5% error rate in these change >>sets that persists for months or years afterwards. > > FWIW, Walter and I did a bunch of these for Py2.3 and had excellent > success because of a good process. If you're willing to review the patch, I'm going to give it a try. Bye, Walter D?rwald From python at rcn.com Mon Dec 1 08:59:01 2003 From: python at rcn.com (Raymond Hettinger) Date: Mon Dec 1 08:59:49 2003 Subject: [Python-Dev] Banishing apply(), buffer(), coerce(), and intern() In-Reply-To: <3FCB3B40.5070703@livinglogic.de> Message-ID: <000501c3b813$46249d80$e841fea9@oemcomputer> > Raymond Hettinger wrote: > > > [...] > >>As always, be careful with doing peephole changes to the standard > >>library -- historically, we've seen a 1-5% error rate in these change > >>sets that persists for months or years afterwards. > > > > FWIW, Walter and I did a bunch of these for Py2.3 and had excellent > > success because of a good process. [Walter] > If you're willing to review the patch, I'm going to give it a try. The honor of second review should go to Anthony ;-) This is his itch. If a third review is needed to make it error free, I would be happy to accommodate. Raymond From anthony at ekit-inc.com Mon Dec 1 09:16:36 2003 From: anthony at ekit-inc.com (Anthony Baxter) Date: Mon Dec 1 09:16:48 2003 Subject: [Python-Dev] Banishing apply(), buffer(), coerce(), and intern() In-Reply-To: Message from "Raymond Hettinger" of "Mon, 01 Dec 2003 08:59:01 CDT." <000501c3b813$46249d80$e841fea9@oemcomputer> Message-ID: <200312011416.hB1EGaDC023723@maxim.off.ekorp.com> > [Walter] > > If you're willing to review the patch, I'm going to give it a try. > > The honor of second review should go to Anthony ;-) This is his itch. > If a third review is needed to make it error free, I would be happy to > accommodate. I'll see how I go for time - with the 2.3.3 release coming up, I'm unlikely to have a whole lot of spare cycles. Anthony From anthony at interlink.com.au Mon Dec 1 09:22:36 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Dec 1 09:22:50 2003 Subject: [Python-Dev] 2.3.3 cycle Message-ID: <200312011422.hB1EMa9d023801@maxim.off.ekorp.com> Ok, my email mailbox is still on tape at the moment, so I'm going from memory here. My current plan (assuming this meets with everyone's approval) is to shoot for a 2.3.3rc1 either Friday this week or else on this weekend, and a final 2.3.3 release around the 17th. How's this fit in with other people's plans? In particular Thomas or Tim, whoever's going to need to do the Windows release. I'll be spending Wednesday working on autoconf magic - I'll see how the latest autoconf-generated stuff works on the HP testdrive farm. If it shows no problems and fixes a bunch of the existing build problems, I'll look at checking it in. I'm not entirely sold on this, though, so could probably be convinced to leave it until 2.3.4, whenever we decide to do that. Given the long time until 2.4, I could see a 2.3.4 around May next year. I'm going to start hanging out on #python-dev for the next few weeks if people want more real-time communications. Anthony From skip at pobox.com Mon Dec 1 09:56:36 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 1 09:56:49 2003 Subject: [Python-Dev] Tutorial: Brief Introduction to the Standard Libary In-Reply-To: <000901c3b45f$b600bba0$5ab0958d@oemcomputer> References: <000901c3b45f$b600bba0$5ab0958d@oemcomputer> Message-ID: <16331.22180.131561.468695@montanaro.dyndns.org> Raymond> Some other candidates (let's pick just a two or three): Raymond> - csv (basic tool for sharing data with other applications) -0. I think basic usage is covered pretty well in the libref. At best, I'd just mention that it exists and link to the libref. Raymond> - datetime (comes up frequently in real apps and admin tasks) +1. Discussing how datetime and time integrate would be useful. Raymond> - ftplib (because the examples are so brief) -1. I would think it's rarely used. Raymond> - getopt or optparse (because the task is common) +1. Raymond> - operator (because otherwise, the functionals can be a PITA) -1. The most common case people needs is now covered by a builtin (sum). Raymond> - pprint (because beauty counts) +0. Brief mention at best. Raymond> - struct (because fixed record layouts are common) -0. Only for propeller heads. Raymond> - threading/Queue (because without direction people grab thread Raymond> and mutexes) +0. I agree that stumbling on thread is too common. OTOH, threads in general are a pretty advanced topic, and probably not real suitable for the tutorial. Raymond> - timeit (because it answers most performance questions in a Raymond> jiffy) -1. Newbies should probably not be worried about performance too much. In addition, I think most performance questions are deeper than those which can be answered by timeit (think naive O(n^2) algorithms). Raymond> - unittest (because TDD folks like myself live by it) +1. There's no time like the present to start adding tests. Raymond> I've avoided XML because it is a can of worms and because short Raymond> examples don't do it justice. OTOH, it *is* the hot topic of Raymond> the day and seems to be taking over the world one angle bracket Raymond> at a time. -1. XML is too complex for tutorial material. Skip From skip at pobox.com Mon Dec 1 10:12:25 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 1 10:12:49 2003 Subject: [Python-Dev] possible backward incompatibility in test.regrtest In-Reply-To: <200311300124.hAU1O1N21082@grad.sccs.chukyo-u.ac.jp> References: <200311300124.hAU1O1N21082@grad.sccs.chukyo-u.ac.jp> Message-ID: <16331.23129.160126.321827@montanaro.dyndns.org> Tamito> It seems that the test.regrtest module has a possible backward Tamito> incompatibility with regard to pre-Python 2.3 releases. I have Tamito> a test suit implemented using the test.regrtest module. ... Tamito> Unless I miss something trivial (I hope so), I'd have to give up Tamito> using the test.regrtest module. I think this is the best course. The advertised (and supported) way to build test suites in Python is with the unittest module. Skip From pycon at python.org Mon Dec 1 10:55:36 2003 From: pycon at python.org (Steve Holden) Date: Mon Dec 1 10:55:38 2003 Subject: [Python-Dev] PyCon DC 2004 - Registration about to open! Message-ID: Dear Python User: You have probably been wondering why you haven't yet heard that PyCon has opened its doors for registration. Well, we are a volunteer organization, and it often takes more time than anticipated to put things together. Welcome to the world of open source :-) I am happy to be able to tell you that thanks to sterling work by Trevor Toenjes we are now in a position to accept your registrations. All you need to do is navigate to http://dc2004reg.pycon.org/ and make the appropriate entries. This year we all have the opportunity to add a PSF donation to the registration fee, and I hope many of you will follow my example and donate a little extra to assist the Foundation in its goals. There is still lots to be done -- for example, we are *also* late with the system to accept submissions, so the original December 1 deadline will be extended by at least a month to ensure that everyone gets plenty of chance to submit papers telling us what they've been up to with Python. We are also encouraging students to attend the conference by giving them a significant reduction from the standard fee, and we hope that this will result in even wider participation by the educational community. Last year was an exciting departure from the traditional "professional" conference format, and I hope that we will be able to improve on it next time. I look forward to seeing you there! Sincerely Steve Holden Chairman PyCon DC 2004 From guido at python.org Mon Dec 1 11:15:20 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 1 11:15:27 2003 Subject: [Python-Dev] "groupby" iterator In-Reply-To: Your message of "Sun, 30 Nov 2003 21:20:16 EST." <5.1.0.14.0.20031130210357.02f62d80@mail.telecommunity.com> References: <20031130073109.GA1560@hishome.net> <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <002701c3b606$c61304a0$e841fea9@oemcomputer> <20031130073109.GA1560@hishome.net> <5.1.0.14.0.20031130184217.02e3c1d0@mail.telecommunity.com> <5.1.0.14.0.20031130210357.02f62d80@mail.telecommunity.com> Message-ID: <200312011615.hB1GFKf30570@c-24-5-183-134.client.comcast.net> > The chaining part, or the idea at all? Sorry, I didn't even see the chaining idea, so it was the keyword attribute instead of two different functions I disliked. The chaining idea seems unnecessary, you can do this with a general currying facility. (I'm not so keen on copying too many such ideas from Zope in to Python -- together they feel more like a bag of clever tricks than like a well-thought-out language.) Anyway, it's all moot -- Raymond just added operator.itemgetter and operator.attrgetter. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 1 11:18:20 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 1 11:18:33 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: Your message of "Sun, 30 Nov 2003 23:35:56 EST." <003301c3b7c4$b9f1a400$e841fea9@oemcomputer> References: <003301c3b7c4$b9f1a400$e841fea9@oemcomputer> Message-ID: <200312011618.hB1GILD30610@c-24-5-183-134.client.comcast.net> > P.S. I'm leaning toward Alex's suggested argument order. Having a > default identity function is too attractive to pass up. So the choice > is between a style like map(None, s) or something closer to > list.sorted(s, key=). Though the latter is not consistent with other > itertools, it wins in the beauty department and its similarity with the > key= is a accurate, helpful analogy. Good. Have you thought about the name yet? Will it be grouped() or grouby()? --Guido van Rossum (home page: http://www.python.org/~guido/) From gerrit at nl.linux.org Mon Dec 1 13:30:30 2003 From: gerrit at nl.linux.org (Gerrit Holl) Date: Mon Dec 1 13:32:20 2003 Subject: [Python-Dev] adding save() and load() function for interactive use Message-ID: <20031201183030.GA5043@nl.linux.org> Hello, I have a proposal to enhance the usefulness of Python for learning purposes. RFE#852222: I think that for learning Python, it would be nice to save the current namespace to a file using a builtin save() function (and corresponding load()). For beginning programmers, it would be nice to be able to tinker interactively with the shell, defining functions and maybe even classes, with which they can continue later on. save() is easily implemented by pickling locals(); load() is easily implemented by updating locals() while unpickling it. The functions would take a filename (string) as an argument. In Idle, they could be in the menu. Like the _ variable, load() and save() are only present in the interactive shell (some global .pythonrc?). Example usage: >>> def foo(): ... raw_input("Do you like it?" ) ... print "Great!" >>> save("~/session") >>> load("~/session") >>> foo() Do you like it? yes Great! Note that this code is untested. I think this would be a nice addition to Python 2.4. yours, Gerrit. -- 105. If the agent is careless, and does not take a receipt for the money which he gave the merchant, he can not consider the unreceipted money as his own. -- 1780 BC, Hammurabi, Code of Law -- Asperger's Syndrome - a personal approach: http://people.nl.linux.org/~gerrit/english/ From mcherm at mcherm.com Mon Dec 1 14:00:30 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Mon Dec 1 14:00:47 2003 Subject: [Python-Dev] Tutorial: Brief Introduction to the Standard Library Message-ID: <1070305230.3fcb8fce50934@mcherm.com> Raymond> - timeit (because it answers most performance questions in a Raymond> jiffy) Skip> -1. Newbies should probably not be worried about performance too Skip> much. In addition, I think most performance questions are deeper Skip> than those which can be answered by timeit (think naive O(n^2) Skip> algorithms). Skip is right on both counts: newbies DO worry too much about performance, and they WON'T be able to think deeply about performance issues like how an algorithm scales (unless they're new to Python but not to programming). But I still agree with Raymond that timeit should be included. The fact is, newbies ARE going to worry too much about performance, whether we want them to or not. And a VERY frequently asked question on c.l.py is "what's the fastest way to do X". IMHO, learning to answer this kind of question themselves is MORE important than learning not to ask the question when it isn't needed -- just like learning to use the interactive interpreter to answer "does the syntax X work?", it makes them more self sufficient, and therefore more self confident. So I'd say DO include a (brief!) tutorial example on using timeit. But perhaps craft the example so it shows two approaches, one readable and one obscure, and the obscure one is shown to be FASTER but not MUCH faster, and is therefore DROPPED in favor of the more readable. (Hey... rigging the problem set in order to model good practices is an old teacher's trick I've used often.) And having piped up with my own 2c, I'd better pay up. Raymond... if you're working on this and need help, send me an example or two of other modules (so I can try to follow your style) and I'll volunteer to write one for timeit. -- Michael Chermside From guido at python.org Mon Dec 1 14:07:11 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 1 14:07:32 2003 Subject: [Python-Dev] adding save() and load() function for interactive use In-Reply-To: Your message of "Mon, 01 Dec 2003 19:30:30 +0100." <20031201183030.GA5043@nl.linux.org> References: <20031201183030.GA5043@nl.linux.org> Message-ID: <200312011907.hB1J7BU31066@c-24-5-183-134.client.comcast.net> > I think that for learning Python, it would be nice to > save the current namespace to a file using a builtin > save() function (and corresponding load()). For > beginning programmers, it would be nice to be able to > tinker interactively with the shell, defining functions > and maybe even classes, with which they can continue > later on. save() is easily implemented by pickling > locals(); load() is easily implemented by updating > locals() while unpickling it. The functions would take > a filename (string) as an argument. In Idle, they could be > in the menu. Like the _ variable, load() and save() are > only present in the interactive shell (some global > .pythonrc?). Example usage: [omitted] Saving classes and functions with pickle doesn't work; it *appeared* to work only because you didn't actually quit your session. IDLE has a feature designed to further the same goal without having to save a "workspace" (a vague concept at best and one that doesn't scale to larger Python apps). You can configure IDLE to come up with a blank untitled Python file instead of a Python Shell window. Then the student's work cycle is intended to be as follows: - Type some code into the file. - Save it (the first time the student has to pick a filename). - Hit F5 which will run the file (which is now a module); this will open a Python Shell (if one isn't already open) so you can experiment with the code defined in the file, but these experiments won't be saved. - Go back to the top to edit the file and run it again. There's also an option to avoid the save step (F5 will save automatically once a filename is picked). --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Mon Dec 1 14:06:57 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Dec 1 14:08:32 2003 Subject: [Python-Dev] adding save() and load() function for interactive use In-Reply-To: <20031201183030.GA5043@nl.linux.org> Message-ID: <5.1.1.6.0.20031201140528.024c3b90@telecommunity.com> At 07:30 PM 12/1/03 +0100, Gerrit Holl wrote: >save() is easily implemented by pickling > locals(); load() is easily implemented by updating > locals() while unpickling it. Functions and classes are pickled as references to a module, so this wouldn't actually work. When you try to load(), the unpickling will fail if the function or class doesn't already exist in the module. From python at rcn.com Mon Dec 1 14:22:32 2003 From: python at rcn.com (Raymond Hettinger) Date: Mon Dec 1 14:23:22 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <200312011618.hB1GILD30610@c-24-5-183-134.client.comcast.net> Message-ID: <002a01c3b840$783a24c0$e841fea9@oemcomputer> [David Eppstein] > My implementation will skip or partially consume the subiterators, with > only a very temporary additional use of memory, if you don't keep a > reference to them. > But I can see your arguments about visible vs silent failure modes and > code > complexity. I'll add note saying that the behavior may change. Then, if I was wrong, we can add support for delinquent subiterators. [Guido] >Have you thought about the name yet? Will it be grouped() or > grouby()? This question is still open. The grouped() name fits well with its list.sorted() and reversed(). Also, it is more natural in context of a default identity function. The groupby() name better suggests what's going on under the hood. The strong association with SQL is a plus because the analogy is accurate. I'm leaning away from grouped() because it seems vague and lifeless. Other tools like izip() and the prospective window() and roundrobin() functions could also be said to return groups. [After Guido suggests introducing an SQLlike itertools.count()] I think this belongs in a new module for accumulator functions including average, standard deviation, and such. In the meantime, len(list(grp)) will suffice. That is even more reasonable when multiple accumulation functions are used because a list has to be created anyway: for k, g in groupby(s, key=keyfunc): data = list(g) print s, len(data), sum(data), average(data) [Guido on chained attribute lookups] >Anyway, it's all moot -- Raymond just added operator.itemgetter > and operator.attrgetter. The implementation does not preclude someone adding support for chained attribute lookups. IMO, the non-chained version covers the most common use cases simply and directly. For complex structures, lambda already has plenty of versatility: key = lambda r: r.roster[3].personalinfo.physicaladdress.zipcode IIRC, Tim recently yearned for the days when the lack of support for nested structures pushed people towards flatter designs. With him already looking forward to death, we dare not add to his angst. Raymond Hettinger From gerrit at nl.linux.org Mon Dec 1 14:30:55 2003 From: gerrit at nl.linux.org (Gerrit Holl) Date: Mon Dec 1 14:31:25 2003 Subject: [Python-Dev] adding save() and load() function for interactive use In-Reply-To: <5.1.1.6.0.20031201140528.024c3b90@telecommunity.com> References: <20031201183030.GA5043@nl.linux.org> <5.1.1.6.0.20031201140528.024c3b90@telecommunity.com> Message-ID: <20031201193054.GA5336@nl.linux.org> Phillip J. Eby wrote: > At 07:30 PM 12/1/03 +0100, Gerrit Holl wrote: > >save() is easily implemented by pickling > > locals(); load() is easily implemented by updating > > locals() while unpickling it. > > Functions and classes are pickled as references to a module, so this > wouldn't actually work. When you try to load(), the unpickling will fail > if the function or class doesn't already exist in the module. Ouch. Of course... I should have thought it through better... sorry! yours, Gerrit. -- 219. If a physician make a large incision in the slave of a freed man, and kill him, he shall replace the slave with another slave. -- 1780 BC, Hammurabi, Code of Law -- Asperger's Syndrome - a personal approach: http://people.nl.linux.org/~gerrit/english/ From mwh at python.net Mon Dec 1 14:35:13 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 1 14:35:21 2003 Subject: [Python-Dev] CVS bustage? Message-ID: <2mwu9gxrf2.fsf@starship.python.net> I'm seeing test_trace dump core from current CVS. Anyone else? Slightly worryingly, working backwards in CVS isn't making it stop, so it might just be something silly at my end... Cheers, mwh -- I have no disaster recovery plan for black holes, I'm afraid. Also please be aware that if it one looks imminent I will be out rioting and setting fire to McDonalds (always wanted to do that) and probably not reading email anyway. -- Dan Barlow From guido at python.org Mon Dec 1 14:47:17 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 1 14:48:46 2003 Subject: [Python-Dev] CVS bustage? In-Reply-To: Your message of "Mon, 01 Dec 2003 19:35:13 GMT." <2mwu9gxrf2.fsf@starship.python.net> References: <2mwu9gxrf2.fsf@starship.python.net> Message-ID: <200312011947.hB1JlHo31202@c-24-5-183-134.client.comcast.net> > I'm seeing test_trace dump core from current CVS. Anyone else? Not here (Red Hat 9, production build). > Slightly worryingly, working backwards in CVS isn't making it stop, so > it might just be something silly at my end... Most likely. What platform? What build (debug or prod.)? --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh at python.net Mon Dec 1 14:54:11 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 1 14:54:15 2003 Subject: [Python-Dev] CVS bustage? In-Reply-To: <200312011947.hB1JlHo31202@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Mon, 01 Dec 2003 11:47:17 -0800") References: <2mwu9gxrf2.fsf@starship.python.net> <200312011947.hB1JlHo31202@c-24-5-183-134.client.comcast.net> Message-ID: <2m8ylwjov0.fsf@starship.python.net> Guido van Rossum writes: >> I'm seeing test_trace dump core from current CVS. Anyone else? > > Not here (Red Hat 9, production build). Nor anyone on #python, either. >> Slightly worryingly, working backwards in CVS isn't making it stop, so >> it might just be something silly at my end... > > Most likely. Yes, sigh. I think maybe I should give up on today. > What platform? Redhat 9... > What build (debug or prod.)? Both. Cheers, mwh -- If your telephone company installs a system in the woods with no one around to see them, do they still get it wrong? -- Robert Moir, alt.sysadmin.recovery From tim.one at comcast.net Mon Dec 1 15:07:03 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 1 15:07:13 2003 Subject: [Python-Dev] CVS bustage? In-Reply-To: <2mwu9gxrf2.fsf@starship.python.net> Message-ID: [Michael Hudson] > I'm seeing test_trace dump core from current CVS. Anyone else? Works on Windows, under release and debug builds. test_winreg fails, though: http://www.python.org/sf/852281 From mcherm at mcherm.com Mon Dec 1 15:12:08 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Mon Dec 1 15:13:13 2003 Subject: [Python-Dev] possible backward incompatibility in test.regrtest Message-ID: <1070309528.3fcba09899c9e@mcherm.com> > Unless I miss something trivial (I hope so), I'd have to give up > using the test.regrtest module. I appreciate any comment. Maybe the thing to do is to change: > if test.startswith('test.'): > abstest = test > else: > # Always import it from the test package > abstest = 'test.' + test > the_package = __import__(abstest, globals(), locals(), []) to this: > if test.startswith('mytest.'): > abstest = test > else: > # Always import it from the test package > abstest = 'mytest.' + test > the_package = __import__(abstest, globals(), locals(), []) in your own copy of regrtest. Converting to unittest would be better, but this is a fix that would at least prevent immediate breakage. -- Michael Chermside From skip at pobox.com Mon Dec 1 15:35:33 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 1 15:35:44 2003 Subject: [Python-Dev] save()/load() Message-ID: <16331.42517.265132.36606@montanaro.dyndns.org> I lost the original thread, but my first thought was to use the readline module's history capability to save and reload a "session". This hack partially works for what little testing I've done (place it in sitecustomize.py): import readline import os import sys _start_len = len(open(os.path.expanduser("~/.pyhist")).readlines()) _session_file = os.path.expanduser("~/.pysession") def save(f=_session_file, start=_start_len): sf = open(f, 'w') while True: line = readline.get_history_item(start) start += 1 if line is None: break if line.startswith('save(') or line.startswith('load('): continue sf.write(line+"\n") def load(f=_session_file): execfile(f, sys._getframe(-1).f_globals) import __builtin__ __builtin__.save = save __builtin__.load = load My readline history is stored in ~/.pyhist. I can't use readline.get_history_length() to mark the start of a session for reasons which are not obvious to me. When I first started futzing around with it, readline.get_history_length() returned 9934. Now, even though my ~/.pyhist file has more than 9960 lines, get_history_length() seems to always return -1, thus making it useless in this context. Here's a simple session showing save() and load() within the same session: >>> import math >>> print math.sin(47) 0.123573122745 >>> def f(): ... return math.sin(48) ... >>> save() >>> load() 0.123573122745 >>> f >>> f() -0.76825466132366682 Note that f may not be a new f, however. A call to load() in a fresh session fails to place 'f' in the session's globals. I'm not sure why. Any ideas? Skip From pje at telecommunity.com Mon Dec 1 16:04:07 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Dec 1 16:07:09 2003 Subject: [Python-Dev] save()/load() In-Reply-To: <16331.42517.265132.36606@montanaro.dyndns.org> Message-ID: <5.1.1.6.0.20031201160258.01e12700@telecommunity.com> At 02:35 PM 12/1/03 -0600, Skip Montanaro wrote: >My readline history is stored in ~/.pyhist. I can't use >readline.get_history_length() to mark the start of a session for reasons >which are not obvious to me. get_history_length() is the length you've set for truncation, not the actual length. You want get_current_history_length(), IIRC. From python at rcn.com Mon Dec 1 16:30:19 2003 From: python at rcn.com (Raymond Hettinger) Date: Mon Dec 1 16:31:26 2003 Subject: [Python-Dev] Tutorial: Brief Introduction to the Standard Library In-Reply-To: <1070305230.3fcb8fce50934@mcherm.com> Message-ID: <003101c3b852$51f2c6c0$e841fea9@oemcomputer> [Michael Chermside] > So I'd say DO include a (brief!) tutorial example on using timeit. But > perhaps craft the example so it shows two approaches, one readable > and one obscure, and the obscure one is shown to be FASTER but not > MUCH faster, and is therefore DROPPED in favor of the more readable. > (Hey... rigging the problem set in order to model good practices is an > old teacher's trick I've used often.) Especially, in the tutorial, stacking the deck is fair play. > And having piped up with my own 2c, I'd better pay up. Raymond... if > you're working on this and need help, send me an example or two of other > modules (so I can try to follow your style) and I'll volunteer to write > one for timeit. I expect to be able to wrap it up this weekend. Raymond From perky at i18n.org Mon Dec 1 16:38:18 2003 From: perky at i18n.org (Hye-Shik Chang) Date: Mon Dec 1 16:39:32 2003 Subject: [Python-Dev] "groupby" iterator In-Reply-To: <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> References: <200311272149.hARLneU16201@c-24-5-183-134.client.comcast.net> <20031128195420.GA63319@i18n.org> <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> Message-ID: <20031201213818.GA20382@i18n.org> On Fri, Nov 28, 2003 at 01:46:53PM -0800, Guido van Rossum wrote: > I would make one change: after looking at another use case, I'd like > to change the outer iterator to produce (key, grouper) tuples. This > way, you can write things like > > totals = {} > for key, group in sequence: > totals[key] = sum(group) > I think this would be helpful for lazy coders if some function of itertools cover the use case: (`LIMIT' keyword of SQL) >>> from groupby import groupby >>> alwaystrue = lambda n: True >>> for k, g in groupby(alwaystrue, range(20), 5): ... print list(g) ... [0, 1, 2, 3, 4] [5, 6, 7, 8, 9] [10, 11, 12, 13, 14] [15, 16, 17, 18, 19] prototype of this case is groupby(keyfunc, iterable, limit=None). Either, groupby(5, range(20)) is okay but 5 is not a sort of `key'. :) Hye-Shik From walter at livinglogic.de Mon Dec 1 16:42:09 2003 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon Dec 1 16:42:26 2003 Subject: [Python-Dev] Banishing apply(), buffer(), coerce(), and intern() In-Reply-To: <000501c3b813$46249d80$e841fea9@oemcomputer> References: <000501c3b813$46249d80$e841fea9@oemcomputer> Message-ID: <3FCBB5B1.8000602@livinglogic.de> Raymond Hettinger wrote: > [...] >>>FWIW, Walter and I did a bunch of these for Py2.3 and had excellent >>>success because of a good process. > > [Walter] > >>If you're willing to review the patch, I'm going to give it a try. > > The honor of second review should go to Anthony ;-) This is his itch. > If a third review is needed to make it error free, I would be happy to > accommodate. OK, the patch is done (http://www.python.org/sf/852334) and assigned to Anthony. I couldn't upload the patch itself, as it's too big, so the patch can be found at http://styx.livinglogic.de/~walter/backticks2repr.txt Bye, Walter D?rwald From skip at pobox.com Mon Dec 1 17:10:15 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 1 17:11:21 2003 Subject: [Python-Dev] save()/load() In-Reply-To: <5.1.1.6.0.20031201160258.01e12700@telecommunity.com> References: <16331.42517.265132.36606@montanaro.dyndns.org> <5.1.1.6.0.20031201160258.01e12700@telecommunity.com> Message-ID: <16331.48199.502155.614605@montanaro.dyndns.org> >> My readline history is stored in ~/.pyhist. I can't use >> readline.get_history_length() to mark the start of a session for >> reasons which are not obvious to me. Phillip> get_history_length() is the length you've set for truncation, Phillip> not the actual length. You want get_current_history_length(), Phillip> IIRC. Yes, thanks. I decided to dispense with accessing the "start" of the session and force the user to simply mark() where a session is to start. That's one extra function to remember, but seems a lot more flexible. Appended is my current incarnation. Still doesn't load() right, though save() seems to work. Usage is: mark() ... bunch of commands at the interpreter prompt ... save() Any clues about what I'm doing wrong in load()? I want to stuff the evaluated objects into the globals() of the caller (that is, the globals for the interactive session in most situations). Skip import readline import os import sys # needs to match the filename the user uses! _histfile = os.path.expanduser("~/.pyhist") # where we save sessions _session_file = os.path.expanduser("~/.pysession") # mark with something valid at the interpreter but unlikely to be executed # by the user _marker = '((1.0+999.0j, "mark", 999.0-1.0j))' def save(): end = readline.get_current_history_length() - 1 session = [] item = readline.get_history_item(end) while item != _marker: session.insert(0, item+"\n") end -= 1 item = readline.get_history_item(end) file(_session_file, 'w').writelines(session) print >> sys.stderr, "saved session to", _session_file def load(): execfile(_session_file, sys._getframe(-1).f_globals) print >> sys.stderr, "loaded session from", _session_file def mark(): readline.add_history(_marker) import __builtin__ __builtin__.save = save __builtin__.load = load __builtin__.mark = mark From pje at telecommunity.com Mon Dec 1 17:26:47 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Dec 1 17:28:31 2003 Subject: [Python-Dev] save()/load() In-Reply-To: <16331.48199.502155.614605@montanaro.dyndns.org> References: <5.1.1.6.0.20031201160258.01e12700@telecommunity.com> <16331.42517.265132.36606@montanaro.dyndns.org> <5.1.1.6.0.20031201160258.01e12700@telecommunity.com> Message-ID: <5.1.1.6.0.20031201172602.024c1cd0@telecommunity.com> At 04:10 PM 12/1/03 -0600, Skip Montanaro wrote: >def load(): > execfile(_session_file, sys._getframe(-1).f_globals) > print >> sys.stderr, "loaded session from", _session_file What does getframe with a negative number do? I've only ever used positive numbers when calling getframe. I think maybe you want 1 here, not -1. From guido at python.org Mon Dec 1 18:57:28 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 1 18:57:47 2003 Subject: [Python-Dev] "groupby" iterator In-Reply-To: Your message of "Tue, 02 Dec 2003 06:38:18 +0900." <20031201213818.GA20382@i18n.org> References: <200311272149.hARLneU16201@c-24-5-183-134.client.comcast.net> <20031128195420.GA63319@i18n.org> <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <20031201213818.GA20382@i18n.org> Message-ID: <200312012357.hB1NvSI32310@c-24-5-183-134.client.comcast.net> > I think this would be helpful for lazy coders if some function of > itertools cover the use case: (`LIMIT' keyword of SQL) > > >>> from groupby import groupby > >>> alwaystrue = lambda n: True > >>> for k, g in groupby(alwaystrue, range(20), 5): > ... print list(g) > ... > [0, 1, 2, 3, 4] > [5, 6, 7, 8, 9] > [10, 11, 12, 13, 14] > [15, 16, 17, 18, 19] > > prototype of this case is groupby(keyfunc, iterable, limit=None). > Either, groupby(5, range(20)) is okay but 5 is not a sort of `key'. :) I'd rather not weigh down groupby() with more options. If you really want only the first 5 of each group, you can use itertools.islice(): for k, g in groupby(alwaystrue, range(20)): print list(itertools.islice(g, 0, 5)) --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Mon Dec 1 20:00:23 2003 From: aahz at pythoncraft.com (Aahz) Date: Mon Dec 1 20:00:26 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200311300006.hAU06Lp19846@c-24-5-183-134.client.comcast.net> References: <200311240434.hAO4Y4L06979@c-24-5-183-134.client.comcast.net> <20031125203258.GA29814@i92.ryd.student.liu.se> <200311300006.hAU06Lp19846@c-24-5-183-134.client.comcast.net> Message-ID: <20031202010023.GA4439@panix.com> On Sat, Nov 29, 2003, Guido van Rossum wrote: > > (2) PEP 237 promises that repr() of a long should no longer show a > trailing 'L'. This is not yet implemented (i.e., repr() of a long > still has a trailing 'L'). First, past experience suggests that > quite a bit of end user code will break, and it may easily break > silently: there used to be code that did str(x)[:-1] (knowing x > was a long) to strip the 'L', which broke when str() of a long no > longer returned a trailing 'L'. Apparently some of this code was > "fixed" by changing str() into repr(), and this code will now > break again. Second, I *like* seeing a trailing L on longs, > especially when there's no reason for it to be a long: if some > expression returns 1L, I know something fishy may have gone on. That makes sense to me; there should be an easy way from Python to detect what kind of object you've got (as a string representation), and repr() is precisely the place for it. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From guido at python.org Mon Dec 1 20:10:25 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 1 20:10:33 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: Your message of "Mon, 01 Dec 2003 20:00:23 EST." <20031202010023.GA4439@panix.com> References: <200311240434.hAO4Y4L06979@c-24-5-183-134.client.comcast.net> <20031125203258.GA29814@i92.ryd.student.liu.se> <200311300006.hAU06Lp19846@c-24-5-183-134.client.comcast.net> <20031202010023.GA4439@panix.com> Message-ID: <200312020110.hB21APb32451@c-24-5-183-134.client.comcast.net> [me] > > (2) PEP 237 promises that repr() of a long should no longer show a > > trailing 'L'. This is not yet implemented (i.e., repr() of a long > > still has a trailing 'L'). First, past experience suggests that > > quite a bit of end user code will break, and it may easily break > > silently: there used to be code that did str(x)[:-1] (knowing x > > was a long) to strip the 'L', which broke when str() of a long no > > longer returned a trailing 'L'. Apparently some of this code was > > "fixed" by changing str() into repr(), and this code will now > > break again. Second, I *like* seeing a trailing L on longs, > > especially when there's no reason for it to be a long: if some > > expression returns 1L, I know something fishy may have gone on. [Aahz] > That makes sense to me; there should be an easy way from Python to > detect what kind of object you've got (as a string representation), and > repr() is precisely the place for it. OK. I've got one weak (-0 from Raymond) opposition to this idea, and two strong agreements (+1 from Tim and Aahz), so I'm going to go ahead and make this change to the PEP. This pretty much makes PEP 237 finished except for the complete and utter eradication of the trailing 'L' (and probably of the 'long' type altogether) in Python 3.0. --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Mon Dec 1 20:27:05 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Dec 1 20:27:12 2003 Subject: [Python-Dev] save()/load() In-Reply-To: <5.1.1.6.0.20031201172602.024c1cd0@telecommunity.com> Message-ID: <200312020127.hB21R5626908@oma.cosc.canterbury.ac.nz> "Phillip J. Eby" > What does getframe with a negative number do? Obviously it uses the time machine to look into the future a little and return a frame that is *going* to be called from the current function... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From ozabluda at yahoo.com Mon Dec 1 21:53:18 2003 From: ozabluda at yahoo.com (Oleg Zabluda) Date: Mon Dec 1 21:53:21 2003 Subject: [Python-Dev] Possible bugs and security holes in getpass.py Message-ID: <20031202025318.97602.qmail@web12208.mail.yahoo.com> I am comparing getpass.py (http://cvs.osafoundation.org/index.cgi/osaf/chandler/python/Lib/getpass.py) and getpass.c from glibc (http://sources.redhat.com/cgi-bin/cvsweb.cgi/libc/misc/getpass.c?rev=1.18&content-type=text/x-cvsweb-markup&cvsroot=glibc). Here are the differences that I identified, some of which may or may not be bugs in getpass.py: 1. getpass.c sets ~ISIG in addition to ~ECHO. 2. getpass.c locks "stdin". 3. getpass.c makes sure the "stdin" is closed even if the thread is cancelled. 4. getpass.c explicitly flushes "stdin" after outputting the promt and before reading the password. 5. getpass.c opens "stdin" in "c" mode. This sets _IO_FLAGS2_NOTCANCEL, whatever that means. Maybe it has something to do with thread cancellation, maybe not. (http://sources.redhat.com/cgi-bin/cvsweb.cgi/libc/libio/fileops.c?rev=1.101&content-type=text/x-cvsweb-markup&cvsroot=glibc) 1,2,3,5 are possible security holes in addition to being possible bugs. Although I don't completely understand all the details, it appears to me that getpass.c is more correct then getpass.py. Suggestion/RFC: either implement the same functionality or a portion thereof in getpass.py, or implement it using getpass(3) directly, at least when linking with glibc. More references: http://www.python.org/doc/current/lib/module-getpass.html http://www.opengroup.org/onlinepubs/007908799/xsh/getpass.html 'info getpass' It would help if we knew whether login, passwd, chfn, su, sudo, yppasswd, etc, use getpass(3) or not. If yes, the answer would be a no-brainer to me. Oleg. From jepler at unpythonic.net Mon Dec 1 22:55:16 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Mon Dec 1 22:55:27 2003 Subject: [Python-Dev] Possible bugs and security holes in getpass.py In-Reply-To: <20031202025318.97602.qmail@web12208.mail.yahoo.com> References: <20031202025318.97602.qmail@web12208.mail.yahoo.com> Message-ID: <20031202035508.GA4023@unpythonic.net> On Mon, Dec 01, 2003 at 06:53:18PM -0800, Oleg Zabluda wrote: > I am comparing getpass.py > (http://cvs.osafoundation.org/index.cgi/osaf/chandler/python/Lib/getpass.py) > and getpass.c from glibc > (http://sources.redhat.com/cgi-bin/cvsweb.cgi/libc/misc/getpass.c?rev=1.18&content-type=text/x-cvsweb-markup&cvsroot=glibc). Many of the differences you cite seem related to glibc-specific internals (I don't know what it means to 'lock "stdin"', open 'in "c" mode', or 'set _IO_FLAGS2_NOTCANCEL', when talking about portable stdio code, though I'm not a posix/SuS expert). glibc-specific code is unlikely to be included in Python, for obvious reasons. > 4. getpass.c explicitly flushes "stdin" after outputting the promt and before > reading the password. In 2.2, I think the marked lines are supposed to perform those functions: try: termios.tcsetattr(fd, termios.TCSADRAIN, new) ### HERE passwd = _raw_input(prompt) finally: termios.tcsetattr(fd, termios.TCSADRAIN, old) ### HERE > Suggestion/RFC: either implement the same functionality or a portion > thereof in getpass.py, or implement it using getpass(3) directly, at least > when linking with glibc. I think that a _getpass module on systems that provide getpass(3) would be appropriate. The block at the bottom of getpass.py would have another level added to deal with the import of _getpass and its absence: try: import _getpass except: existing block, but indented more else: getpass = _getpass.getpass There's probably nothing subtle about writing _getpassmodule.c, and distutils should be able to handle its absence gracefully (?). Before I'm +1 on doing this, though, here's what the linux (redhat 9) manpage has to say about getpass: SYNOPSIS #include char *getpass( const char * prompt ); DESCRIPTION This function is obsolete. Do not use it. ^^^^^^^^ The opengroup web page you mention says it is "LEGACY", with this explanation: This function was marked LEGACY since it provides no functionality which a user could not easily implement, and its name is misleading. ... how important is getpass, and did the opengroup really underestimate the subtlety of its implementation that greatly? Jeff From pje at telecommunity.com Mon Dec 1 23:11:20 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Dec 1 23:09:29 2003 Subject: [Python-Dev] save()/load() In-Reply-To: <200312020127.hB21R5626908@oma.cosc.canterbury.ac.nz> References: <5.1.1.6.0.20031201172602.024c1cd0@telecommunity.com> Message-ID: <5.1.0.14.0.20031201231000.02c97b60@mail.telecommunity.com> At 02:27 PM 12/2/03 +1300, Greg Ewing wrote: >"Phillip J. Eby" > > > What does getframe with a negative number do? > >Obviously it uses the time machine to look into the >future a little and return a frame that is *going* >to be called from the current function... Actually, it appears to return the same result as sys._getframe(0). Maybe it should raise an exception for a negative value. (OTOH, maybe leaving it so that it silently does something meaningless will help discourage people from using it so often... ;) ) From skip at pobox.com Mon Dec 1 23:16:46 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 1 23:16:51 2003 Subject: [Python-Dev] save()/load() In-Reply-To: <5.1.0.14.0.20031201231000.02c97b60@mail.telecommunity.com> References: <5.1.1.6.0.20031201172602.024c1cd0@telecommunity.com> <5.1.0.14.0.20031201231000.02c97b60@mail.telecommunity.com> Message-ID: <16332.4654.696644.838775@montanaro.dyndns.org> >> > What does getframe with a negative number do? >> Obviously it uses the time machine to look into the future... Phillip> Maybe it should raise an exception for a negative value. Phillip> (OTOH, maybe leaving it so that it silently does something Phillip> meaningless will help discourage people from using it so Phillip> often... ;) ) Yeah, whatever. ;-) At any rate, now that Phillip and Tim straightened me out on my number line basics, there's a simple session save/restore available at http://www.musi-cal.com/~skip/python/save_session.py Making it work on Windows (where it's probably needed more) is left as an exercise for the reader. Skip From guido at python.org Mon Dec 1 23:31:56 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 1 23:32:04 2003 Subject: [Python-Dev] Possible bugs and security holes in getpass.py In-Reply-To: Your message of "Mon, 01 Dec 2003 18:53:18 PST." <20031202025318.97602.qmail@web12208.mail.yahoo.com> References: <20031202025318.97602.qmail@web12208.mail.yahoo.com> Message-ID: <200312020431.hB24Vu932651@c-24-5-183-134.client.comcast.net> > 1,2,3,5 are possible security holes in addition to being possible bugs. > > Although I don't completely understand all the details, it appears > to me that getpass.c is more correct then getpass.py. Sorry, but this just doesn't make sense. There are so many differences between C and Python that you can't just compare a C and a Python version of a function and pointing at the differences as possible security holes or bugs. If you want to be helpful, I please try to understand the details, and then see if there are *actual* bugs or security holes instead of just "possible" ones. Looking for security issues is serious business. (It pays my bills. :-) But people shouldn't go around pointing out "possible" security holes without understanding what they are talking about -- spreading fear doesn't help real security. It is unlikely that a beginning programmer can find a security hole in a piece of software without dumb luck. --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh at python.net Tue Dec 2 06:49:16 2003 From: mwh at python.net (Michael Hudson) Date: Tue Dec 2 06:49:20 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib site.py, 1.53.6.1, 1.53.6.2 In-Reply-To: (jhylton@users.sourceforge.net's message of "Mon, 01 Dec 2003 22:47:39 -0800") References: Message-ID: <2m1xrnjv7n.fsf@starship.python.net> jhylton@users.sourceforge.net writes: > Update of /cvsroot/python/python/dist/src/Lib > In directory sc8-pr-cvs1:/tmp/cvs-serv30842 > > Modified Files: > Tag: release23-maint > site.py > Log Message: > A hack: Disable import of re, because sre doesn't compile right. > > DON'T MERGE THIS CHANGE TO THE TRUNK. Was this meant to be on the ast-branch? If not, what's going on? I don't think breaking uninstalled builds on any platform that isn't linux/x86 is a good solution for anything. Cheers, mwh -- 7. It is easier to write an incorrect program than understand a correct one. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html From skip at pobox.com Tue Dec 2 07:53:23 2003 From: skip at pobox.com (Skip Montanaro) Date: Tue Dec 2 07:53:24 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib site.py, 1.53.6.1, 1.53.6.2 In-Reply-To: References: Message-ID: <16332.35651.448681.927287@montanaro.dyndns.org> jhylton> Update of /cvsroot/python/python/dist/src/Lib jhylton> In directory sc8-pr-cvs1:/tmp/cvs-serv30842 jhylton> Modified Files: jhylton> Tag: release23-maint ^^^^^^^^^^^^^^^ jhylton> site.py jhylton> Log Message: jhylton> A hack: Disable import of re, because sre doesn't compile right. jhylton> DON'T MERGE THIS CHANGE TO THE TRUNK. Jeremy, Should this be on the release23-maint branch? Skip From mmclay at comcast.net Tue Dec 2 08:27:46 2003 From: mmclay at comcast.net (Michael McLay) Date: Tue Dec 2 08:24:37 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312020110.hB21APb32451@c-24-5-183-134.client.comcast.net> References: <200311240434.hAO4Y4L06979@c-24-5-183-134.client.comcast.net> <20031202010023.GA4439@panix.com> <200312020110.hB21APb32451@c-24-5-183-134.client.comcast.net> Message-ID: <200312020827.46280.mmclay@comcast.net> On Monday 01 December 2003 08:10 pm, Guido van Rossum wrote: This pretty much makes PEP 237 > finished except for the complete and utter eradication of the trailing > 'L' (and probably of the 'long' type altogether) in Python 3.0. Would there still be an int type and a long type in Python 3.0, or would the notion of a long be be dropped. If it were dropped, then the int representation would be transparently represented as a long if the size of the number could not fit in an int. If long is dropped then the long function could be added to the list of builtins that will disappear in Python 3.0. From pycon at python.org Tue Dec 2 09:10:15 2003 From: pycon at python.org (Steve Holden) Date: Tue Dec 2 09:10:17 2003 Subject: [Python-Dev] PyCon DC 2004 - Submissions Deadline Extended Message-ID: Dear Python User: We have received many enquiries asking "When and how will I be able to submit my paper to PyCon DC 2004?". This is encouraging news - from the correspondence to date it seems that there will be some interesting and exciting presentations next March. Please be patient as the system to handle submissions goes through to completion. When it is ready, look for a further announcement in the same place as this one. Just a reminder, while I have your attention. Submissions should ideally be in a single file, containing HTML or the reStructured Text format. At a pinch we will be prepared to accept PDF, and either tar or zip files containing sets of coordinated documents (such as an HTML master and the graphics to which it refers). Ulitmately we would like to publish all accepted papers on the web, and these rules should make it easier to do so. If your paper is accepted and you prepare an electronic presentation (in PDF, PythonPoint or PowerPoint) we will also happily publish that on the web site when PyCon is over. So, have at it! Anything describing the uses of Python, or better ways of teaching it, neat tricks for developers and suggestions for improvement to the language or its community is fair game. It's likely we will have a larger proportion of students attending in 2004, so we are particularly encouraging presentations with an educational spin, including material of a more introductory nature. Don't forget, too, that even if you don't want to make a formal presentation, the Open Space sessions and other informal activities will give everyone plenty of chance to contribute what they know and discuss what they need. PyCon should be *the* place to go to learn more about Python. Please help to ensure this is so. Sincerely Steve Holden Chairman PyCon DC 2004 From jeremy at alum.mit.edu Tue Dec 2 09:54:26 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Tue Dec 2 09:55:56 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib site.py, 1.53.6.1, 1.53.6.2 In-Reply-To: <2m1xrnjv7n.fsf@starship.python.net> References: <2m1xrnjv7n.fsf@starship.python.net> Message-ID: <1070376865.2609.0.camel@localhost.localdomain> On Tue, 2003-12-02 at 06:49, Michael Hudson wrote: > jhylton@users.sourceforge.net writes: > > > Update of /cvsroot/python/python/dist/src/Lib > > In directory sc8-pr-cvs1:/tmp/cvs-serv30842 > > > > Modified Files: > > Tag: release23-maint > > site.py > > Log Message: > > A hack: Disable import of re, because sre doesn't compile right. > > > > DON'T MERGE THIS CHANGE TO THE TRUNK. > > Was this meant to be on the ast-branch? If not, what's going on? Yikes. It was meant just for the ast-branch. Don't know how my release23-maint checkout got mixed up with ast-branch checkout. Jeremy From guido at python.org Tue Dec 2 10:10:55 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 2 10:10:59 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: Your message of "Tue, 02 Dec 2003 08:27:46 EST." <200312020827.46280.mmclay@comcast.net> References: <200311240434.hAO4Y4L06979@c-24-5-183-134.client.comcast.net> <20031202010023.GA4439@panix.com> <200312020110.hB21APb32451@c-24-5-183-134.client.comcast.net> <200312020827.46280.mmclay@comcast.net> Message-ID: <200312021510.hB2FAtX01085@c-24-5-183-134.client.comcast.net> > Would there still be an int type and a long type in Python 3.0, or > would the notion of a long be be dropped. If it were dropped, then > the int representation would be transparently represented as a long > if the size of the number could not fit in an int. If long is > dropped then the long function could be added to the list of > builtins that will disappear in Python 3.0. I'm not sure about that yet. I'd *like* to find a hack that lets the int type change representations, but the fact is that it's much easier to use different types to indicate different representations. But you're right, even if there are two types, there's probably no reason to expose the 'long' type as a builtin. So long should go on the list as *likely* to disappear in 3.0. --Guido van Rossum (home page: http://www.python.org/~guido/) From pf_moore at yahoo.co.uk Tue Dec 2 11:02:21 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Tue Dec 2 11:02:13 2003 Subject: [Python-Dev] Re: "groupby" iterator References: <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <002701c3b606$c61304a0$e841fea9@oemcomputer> <20031130073109.GA1560@hishome.net> <200311302157.49205.aleaxit@yahoo.com> Message-ID: <8ylvmcmq.fsf@yahoo.co.uk> Alex Martelli writes: > Since these functions or types are going to be in operator, I think > we can afford to "spend" two names to distinguish functionality > (even though attgetter and itemgetter look nowhere as neat as > extract -- I don't have better suggestions offhand). operator.getattr and operator.getitem? Paul. -- This signature intentionally left blank From pf_moore at yahoo.co.uk Tue Dec 2 11:31:11 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Tue Dec 2 11:31:01 2003 Subject: [Python-Dev] Re: "groupby" iterator References: <20031130073109.GA1560@hishome.net> <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <002701c3b606$c61304a0$e841fea9@oemcomputer> <20031130073109.GA1560@hishome.net> <5.1.0.14.0.20031130184217.02e3c1d0@mail.telecommunity.com> <200312010018.hB10IbS29532@c-24-5-183-134.client.comcast.ne t> <5.1.0.14.0.20031130210357.02f62d80@mail.telecommunity.com> Message-ID: <4qwjmbao.fsf@yahoo.co.uk> [Phillip J. Eby] > Yes. Really the whole extract thing isn't that useful, except to get > extra speed over using 'lambda x: x.foo' or whatever, which is what > I'd probably use in any code that wasn't composing functions or > compiling an OO query language. :) [Thomas Heller] > Hm, couldn't "lambda ob: ob.foo.bar" return exactly the same thing > as > "extract(extract(attr='foo'), attr='bar')" > ? In other words: return specialized C implemented functions for > simple lambda expressions? I agree with Thomas - rather than adding yet more specialised functions, it would seem more sensible to optimize lambda - probably via special cases like this. Paul. -- This signature intentionally left blank From theller at python.net Tue Dec 2 12:31:21 2003 From: theller at python.net (Thomas Heller) Date: Tue Dec 2 12:31:34 2003 Subject: [Python-Dev] 2.3.3 cycle In-Reply-To: <200312011422.hB1EMa9d023801@maxim.off.ekorp.com> (Anthony Baxter's message of "Tue, 02 Dec 2003 01:22:36 +1100") References: <200312011422.hB1EMa9d023801@maxim.off.ekorp.com> Message-ID: Anthony Baxter writes: > Ok, my email mailbox is still on tape at the moment, so I'm going > from memory here. My current plan (assuming this meets with everyone's > approval) is to shoot for a 2.3.3rc1 either Friday this week or else > on this weekend, and a final 2.3.3 release around the 17th. How's this > fit in with other people's plans? In particular Thomas or Tim, whoever's > going to need to do the Windows release. That's fine with me. Thomas From walter at livinglogic.de Tue Dec 2 13:42:02 2003 From: walter at livinglogic.de (=?ISO-8859-15?Q?Walter_D=F6rwald?=) Date: Tue Dec 2 13:42:08 2003 Subject: [Python-Dev] pprint and list/tuple/dict subclasses Message-ID: <3FCCDCFA.6070800@livinglogic.de> Currently pprint refuses to prettyprint subclasses of list, tuple and dict. Does it make sense to change that for 2.4? Of course once __repr__() is overwritten by a subclass, pprint has to use that instead of its own formatting. Bye, Walter D?rwald From fdrake at acm.org Tue Dec 2 13:45:41 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue Dec 2 13:45:52 2003 Subject: [Python-Dev] pprint and list/tuple/dict subclasses In-Reply-To: <3FCCDCFA.6070800@livinglogic.de> References: <3FCCDCFA.6070800@livinglogic.de> Message-ID: <16332.56789.356676.91289@grendel.fdrake.net> Walter D?rwald writes: > Currently pprint refuses to prettyprint subclasses of list, > tuple and dict. Does it make sense to change that for 2.4? > > Of course once __repr__() is overwritten by a subclass, > pprint has to use that instead of its own formatting. There's a patch for this that's on my list of things to look at I think. The discussion associated with the patch had the right idea, but I've not had time to review the patch (which I think is embarrassingly short); if you'd like to do so, be my guest! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From guido at python.org Tue Dec 2 13:50:58 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 2 13:51:05 2003 Subject: [Python-Dev] pprint and list/tuple/dict subclasses In-Reply-To: Your message of "Tue, 02 Dec 2003 19:42:02 +0100." <3FCCDCFA.6070800@livinglogic.de> References: <3FCCDCFA.6070800@livinglogic.de> Message-ID: <200312021850.hB2IowQ01754@c-24-5-183-134.client.comcast.net> > Currently pprint refuses to prettyprint subclasses of list, > tuple and dict. Does it make sense to change that for 2.4? > > Of course once __repr__() is overwritten by a subclass, > pprint has to use that instead of its own formatting. Sounds like a good plan. Do you have a suggested patch? --Guido van Rossum (home page: http://www.python.org/~guido/) From theller at python.net Tue Dec 2 13:59:16 2003 From: theller at python.net (Thomas Heller) Date: Tue Dec 2 13:59:28 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <4qwjmbao.fsf@yahoo.co.uk> (Paul Moore's message of "Tue, 02 Dec 2003 16:31:11 +0000") References: <20031130073109.GA1560@hishome.net> <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <002701c3b606$c61304a0$e841fea9@oemcomputer> <20031130073109.GA1560@hishome.net> <5.1.0.14.0.20031130184217.02e3c1d0@mail.telecommunity.com> <200312010018.hB10IbS29532@c-24-5-183-134.client.comcast.ne t> <5.1.0.14.0.20031130210357.02f62d80@mail.telecommunity.com> <4qwjmbao.fsf@yahoo.co.uk> Message-ID: Paul Moore writes: > [Phillip J. Eby] >> Yes. Really the whole extract thing isn't that useful, except to get >> extra speed over using 'lambda x: x.foo' or whatever, which is what >> I'd probably use in any code that wasn't composing functions or >> compiling an OO query language. :) > > [Thomas Heller] >> Hm, couldn't "lambda ob: ob.foo.bar" return exactly the same thing >> as > >> "extract(extract(attr='foo'), attr='bar')" > >> ? In other words: return specialized C implemented functions for >> simple lambda expressions? > > I agree with Thomas - rather than adding yet more specialised > functions, it would seem more sensible to optimize lambda - probably > via special cases like this. One question that remains is: do a handful of these specialized functions make it possible to replace the remaining uses lambda completely? Looking at parts of my codebase nearly all uses of lambda are 'lambda self: self.someattr'. The remaining occurences have not yet been ported to the idioms of newer Pythons. Thomas From hfastjava at yahoo.com Tue Dec 2 14:06:20 2003 From: hfastjava at yahoo.com (Hunter Peress) Date: Tue Dec 2 14:06:25 2003 Subject: [Python-Dev] pprint and list/tuple/dict subclasses Message-ID: <20031202190620.58700.qmail@web41302.mail.yahoo.com> I see the arguments,but the following code works fine. so what exactly is the issue? #!/usr/bin/python import sys,commands,os,re,string from pprint import pprint as ppr class t(list): pass inst = t() inst.append(1) inst.append({2:3}) inst.append([4,5,6,[7,8]]) ppr(inst) __________________________________ Do you Yahoo!? Free Pop-Up Blocker - Get it now http://companion.yahoo.com/ From fdrake at acm.org Tue Dec 2 14:18:54 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue Dec 2 14:19:05 2003 Subject: [Python-Dev] pprint and list/tuple/dict subclasses In-Reply-To: <20031202190620.58700.qmail@web41302.mail.yahoo.com> References: <20031202190620.58700.qmail@web41302.mail.yahoo.com> Message-ID: <16332.58782.958432.60132@grendel.fdrake.net> Hunter Peress writes: > I see the arguments,but the following code works fine. so what > exactly is the issue? The code doesn't work when the repr of the subclass instance should be wrapped: > #!/usr/bin/python > import sys,commands,os,re,string > from pprint import pprint as ppr > > class t(list): > pass > > inst = t() > inst.append(1) > inst.append({2:3}) > inst.append([4,5,6,[7,8]]) Change the last line to: for i in range(10): inst.append(range(10)) and what you get is: [1, {2: 3}, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]] (wrapped by pasting into my mail client), but what's expected is: [1, {2: 3}, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]] -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From guido at python.org Tue Dec 2 14:18:56 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 2 14:19:08 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: Your message of "Tue, 02 Dec 2003 19:59:16 +0100." References: <20031130073109.GA1560@hishome.net> <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <002701c3b606$c61304a0$e841fea9@oemcomputer> <20031130073109.GA1560@hishome.net> <5.1.0.14.0.20031130184217.02e3c1d0@mail.telecommunity.com> <200312010018.hB10IbS29532@c-24-5-183-134.client.comcast.ne t> <5.1.0.14.0.20031130210357.02f62d80@mail.telecommunity.com> <4qwjmbao.fsf@yahoo.co.uk> Message-ID: <200312021918.hB2JIuQ01834@c-24-5-183-134.client.comcast.net> > > I agree with Thomas - rather than adding yet more specialised > > functions, it would seem more sensible to optimize lambda - probably > > via special cases like this. > > One question that remains is: do a handful of these specialized > functions make it possible to replace the remaining uses lambda > completely? > > Looking at parts of my codebase nearly all uses of lambda are > 'lambda self: self.someattr'. Aha. Very interesting ideas both! In the past, we had a similar issue with exec/eval. We looked at the most frequent uses of these, and found that getting an attribute with a computed name was the most common, so we added getattr (and setattr and delattr). Importing a module with a computed name was also quite common, and now we have __import__. So now exec/eval are typically only used when we *really* want to run code provided by an end user. (Exception: I often use eval() to parse literals when I know it is a literal but it can have several types, e.g. string, int, float. Maybe there's a restricted form of eval that could be used for this too?) So again, here we have a mechanism that's rather generic (lambda) which is frequently used in a few stylized patterns (to extract an attribute or field). So Raymond's new functions attrgetter and itemgetter (whose names I cannot seem to remember :-) take care of these. But, at least for attrgetter, I am slightly unhappy with the outcome, because the attribute name is now expressed as a string literal rather than using attribute notation. This makes it harder to write automated tools that check or optimize code. (For itemgetter it doesn't really matter, since the index is a literal either way.) So, while I'm not particularly keen on lambda, I'm not that keen on attrgetter either. But what could be better? All I can think of are slightly shorter but even more crippled forms of lambda; for example, we could invent a new keyword XXX so that the expression (XXX.foo) is equivalent to (lambda self: self.foo). This isn't very attractive. Maybe the idea of recognizing some special forms of lambda and implementing them more efficiently indeed makes more sense! Hm, I see no end to this rambling, but I've got to go, so I'll just stop now... --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh at python.net Tue Dec 2 14:23:35 2003 From: mwh at python.net (Michael Hudson) Date: Tue Dec 2 14:23:38 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <200312021918.hB2JIuQ01834@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Tue, 02 Dec 2003 11:18:56 -0800") References: <20031130073109.GA1560@hishome.net> <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <002701c3b606$c61304a0$e841fea9@oemcomputer> <20031130073109.GA1560@hishome.net> <5.1.0.14.0.20031130184217.02e3c1d0@mail.telecommunity.com> <200312010018.hB10IbS29532@c-24-5-183-134.client.comcast.ne t> <5.1.0.14.0.20031130210357.02f62d80@mail.telecommunity.com> <4qwjmbao.fsf@yahoo.co.uk> <200312021918.hB2JIuQ01834@c-24-5-183-134.client.comcast.net> Message-ID: <2msmk3hvm0.fsf@starship.python.net> Guido van Rossum writes: > So again, here we have a mechanism that's rather generic (lambda) > which is frequently used in a few stylized patterns (to extract an > attribute or field). So Raymond's new functions attrgetter and > itemgetter (whose names I cannot seem to remember :-) take care of > these. > > But, at least for attrgetter, I am slightly unhappy with the outcome, > because the attribute name is now expressed as a string literal rather > than using attribute notation. This makes it harder to write > automated tools that check or optimize code. (For itemgetter it > doesn't really matter, since the index is a literal either way.) > > So, while I'm not particularly keen on lambda, I'm not that keen on > attrgetter either. But what could be better? All I can think of are > slightly shorter but even more crippled forms of lambda; for example, > we could invent a new keyword XXX so that the expression (XXX.foo) is > equivalent to (lambda self: self.foo). This isn't very attractive. Doesn't have to be a keyword... I implemented something like this years ago and then ditched it when list comps appeared. It would let you do things like >>> map(X + 1, range(2)) [1, 2, 3] too, IIRC. Cheers, mwh -- > With Python you can start a thread, but you can't stop it. Sorry. > You'll have to wait until reaches the end of execution. So, just the same as c.l.py, then? -- Cliff Wells & Steve Holden, comp.lang.python From guido at python.org Tue Dec 2 14:26:40 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 2 14:26:47 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: Your message of "Tue, 02 Dec 2003 19:23:35 GMT." <2msmk3hvm0.fsf@starship.python.net> References: <20031130073109.GA1560@hishome.net> <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <002701c3b606$c61304a0$e841fea9@oemcomputer> <20031130073109.GA1560@hishome.net> <5.1.0.14.0.20031130184217.02e3c1d0@mail.telecommunity.com> <200312010018.hB10IbS29532@c-24-5-183-134.client.comcast.ne t> <5.1.0.14.0.20031130210357.02f62d80@mail.telecommunity.com> <4qwjmbao.fsf@yahoo.co.uk> <200312021918.hB2JIuQ01834@c-24-5-183-134.client.comcast.net> <2msmk3hvm0.fsf@starship.python.net> Message-ID: <200312021926.hB2JQeP01889@c-24-5-183-134.client.comcast.net> > > So, while I'm not particularly keen on lambda, I'm not that keen on > > attrgetter either. But what could be better? All I can think of are > > slightly shorter but even more crippled forms of lambda; for example, > > we could invent a new keyword XXX so that the expression (XXX.foo) is > > equivalent to (lambda self: self.foo). This isn't very attractive. > > Doesn't have to be a keyword... I implemented something like this > years ago and then ditched it when list comps appeared. > > It would let you do things like > > >>> map(X + 1, range(2)) > [1, 2, 3] What was your notation like? Did you actually use 'X'? --Guido van Rossum (home page: http://www.python.org/~guido/) From theller at python.net Tue Dec 2 14:34:40 2003 From: theller at python.net (Thomas Heller) Date: Tue Dec 2 14:34:53 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <2msmk3hvm0.fsf@starship.python.net> (Michael Hudson's message of "Tue, 02 Dec 2003 19:23:35 +0000") References: <20031130073109.GA1560@hishome.net> <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <002701c3b606$c61304a0$e841fea9@oemcomputer> <20031130073109.GA1560@hishome.net> <5.1.0.14.0.20031130184217.02e3c1d0@mail.telecommunity.com> <200312010018.hB10IbS29532@c-24-5-183-134.client.comcast.ne t> <5.1.0.14.0.20031130210357.02f62d80@mail.telecommunity.com> <4qwjmbao.fsf@yahoo.co.uk> <200312021918.hB2JIuQ01834@c-24-5-183-134.client.comcast.net> <2msmk3hvm0.fsf@starship.python.net> Message-ID: Michael Hudson writes: > Guido van Rossum writes: > >> So again, here we have a mechanism that's rather generic (lambda) >> which is frequently used in a few stylized patterns (to extract an >> attribute or field). So Raymond's new functions attrgetter and >> itemgetter (whose names I cannot seem to remember :-) take care of >> these. >> >> But, at least for attrgetter, I am slightly unhappy with the outcome, >> because the attribute name is now expressed as a string literal rather >> than using attribute notation. This makes it harder to write >> automated tools that check or optimize code. (For itemgetter it >> doesn't really matter, since the index is a literal either way.) >> >> So, while I'm not particularly keen on lambda, I'm not that keen on >> attrgetter either. But what could be better? All I can think of are >> slightly shorter but even more crippled forms of lambda; for example, >> we could invent a new keyword XXX so that the expression (XXX.foo) is >> equivalent to (lambda self: self.foo). This isn't very attractive. > > Doesn't have to be a keyword... I implemented something like this > years ago and then ditched it when list comps appeared. > > It would let you do things like > >>>> map(X + 1, range(2)) Something like this? class Adder: def __init__(self, number): self._number = number def __call__(self, arg): return arg + self._number class X: def __add__(self, number): return Adder(number) X = X() print map(X + 1, range(2)) > [1, 2, 3] > (Although the above only prints [1, 2] ;-) From walter at livinglogic.de Tue Dec 2 14:39:15 2003 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Tue Dec 2 14:39:22 2003 Subject: [Python-Dev] pprint and list/tuple/dict subclasses In-Reply-To: <20031202190620.58700.qmail@web41302.mail.yahoo.com> References: <20031202190620.58700.qmail@web41302.mail.yahoo.com> Message-ID: <3FCCEA63.2070503@livinglogic.de> Hunter Peress wrote: > I see the arguments,but the following code works fine. so what exactly is the issue? > > #!/usr/bin/python > import sys,commands,os,re,string > from pprint import pprint as ppr > > class t(list): > pass > > inst = t() > inst.append(1) > inst.append({2:3}) > inst.append([4,5,6,[7,8]]) > > ppr(inst) This uses list.__repr__(), but you won't see any difference because the string is too short. Bye, Walter D?rwald From guido at python.org Tue Dec 2 15:02:08 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 2 15:02:15 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: Your message of "Tue, 02 Dec 2003 20:34:40 +0100." References: <20031130073109.GA1560@hishome.net> <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <002701c3b606$c61304a0$e841fea9@oemcomputer> <20031130073109.GA1560@hishome.net> <5.1.0.14.0.20031130184217.02e3c1d0@mail.telecommunity.com> <200312010018.hB10IbS29532@c-24-5-183-134.client.comcast.ne t> <5.1.0.14.0.20031130210357.02f62d80@mail.telecommunity.com> <4qwjmbao.fsf@yahoo.co.uk> <200312021918.hB2JIuQ01834@c-24-5-183-134.client.comcast.net> <2msmk3hvm0.fsf@starship.python.net> Message-ID: <200312022002.hB2K28301985@c-24-5-183-134.client.comcast.net> > > It would let you do things like > > > >>>> map(X + 1, range(2)) > > Something like this? > > class Adder: > def __init__(self, number): > self._number = number > def __call__(self, arg): > return arg + self._number > > class X: > def __add__(self, number): > return Adder(number) > > X = X() > > print map(X + 1, range(2)) Ah, of course. Nice. This can be extended to __getattr__ and __getitem__; unfortunately __call__ would be ambiguous. It could probably be made quite fast with a C implementation. Now the question remains, would it be better to hide this and simply use it under the hood as an alternative way of generating code for lambda, or should it be some sort of standard library module, to be invoked explicitly? In favor of the latter pleads that this would solve the semantic differences with lambda when free variables are involved: obviously X+q would evaluate q only once, while (lamda X: X+q) evaluates q on each invocation. Remember that for generator expressions we've made the decision that (X+q for X in seq) should evaluate q only once. --Guido van Rossum (home page: http://www.python.org/~guido/) From fumanchu at amor.org Tue Dec 2 15:06:03 2003 From: fumanchu at amor.org (Robert Brewer) Date: Tue Dec 2 15:11:12 2003 Subject: [Python-Dev] Re: "groupby" iterator Message-ID: After: > > > I agree with Thomas - rather than adding yet more specialised > > > functions, it would seem more sensible to optimize lambda > - probably > > > via special cases like this. > > > > One question that remains is: do a handful of these specialized > > functions make it possible to replace the remaining uses lambda > > completely? > > > > Looking at parts of my codebase nearly all uses of lambda are > > 'lambda self: self.someattr'. > Guido van Rossum wrote: > Aha. Very interesting ideas both! > > So again, here we have a mechanism that's rather generic (lambda) > which is frequently used in a few stylized patterns (to extract an > attribute or field). So Raymond's new functions attrgetter and > itemgetter (whose names I cannot seem to remember :-) take care of > these. > > But, at least for attrgetter, I am slightly unhappy with the outcome, > because the attribute name is now expressed as a string literal rather > than using attribute notation. This makes it harder to write > automated tools that check or optimize code. (For itemgetter it > doesn't really matter, since the index is a literal either way.) > > So, while I'm not particularly keen on lambda, I'm not that keen on > attrgetter either. But what could be better? All I can think of are > slightly shorter but even more crippled forms of lambda; for example, > we could invent a new keyword XXX so that the expression (XXX.foo) is > equivalent to (lambda self: self.foo). This isn't very attractive. > Maybe the idea of recognizing some special forms of lambda and > implementing them more efficiently indeed makes more sense! I swore I wouldn't post here without lurking for a couple months, but I can't help myself. I mentioned on c.l.p. that extract() looked like a classic case for currying to me, using getattr() and currying the attribute name. So that instead of (lambda self: self.foo), you'd end up with (rightcurry(getattr, 'foo')), which might give some of the speed increase desired. If one could assure that each object in an iteration had that 'foo' attribute at the same mem location (for example, instances of a class with a common function def), you could optimize further. > Now the question remains, would it be better to hide this and simply > use it under the hood as an alternative way of generating code for > lambda, or should it be some sort of standard library module, to be > invoked explicitly? In favor of the latter pleads that this would > solve the semantic differences with lambda when free variables are > involved: obviously X+q would evaluate q only once, while > (lamda X: X+q) evaluates q on each invocation. Remember that for > generator expressions we've made the decision that > (X+q for X in seq) > should evaluate q only once. Which currying would do, no? Instead of (lambda X: X+q) you get (rightcurry(operator.add, q)). The reason I bring it up is because I see curry applying in many more and more powerful situations than just extract(). Kill 16 birds with one stone. Obviously, there's been some work on curry already in PEP 309, which included the community response of "Lambda is good enough." Perhaps it's not in this case. Criticism welcomed, constructive or destructive. :) Robert Brewer MIS Amor Ministries fumanchu@amor.org From skip at pobox.com Tue Dec 2 15:13:34 2003 From: skip at pobox.com (Skip Montanaro) Date: Tue Dec 2 15:13:49 2003 Subject: [Python-Dev] minor interruption to service. In-Reply-To: <200311290344.hAT3iHF5013034@maxim.off.ekorp.com> References: <200311290344.hAT3iHF5013034@maxim.off.ekorp.com> Message-ID: <16332.62062.305234.903783@montanaro.dyndns.org> Anthony> I'm going to be pretty much offline for a week or so - we got Anthony> burgled the other night while we were asleep and my laptop was Anthony> stolen. The data's backed up, but it'll be a few days til the Anthony> replacement laptop arrives. That's a pisser. Sorry to hear about it. Anthony> In the meantime, if someone wants to take on the "upgrade to Anthony> autoconf 2.59" task, I'd appreciate it very much. Do you mean 2.58? I looked here: http://ftp.gnu.org/gnu/autoconf/ and 2.58 appears to be the latest release. Skip From ark-mlist at att.net Tue Dec 2 15:14:01 2003 From: ark-mlist at att.net (Andrew Koenig) Date: Tue Dec 2 15:13:58 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312021510.hB2FAtX01085@c-24-5-183-134.client.comcast.net> Message-ID: <024201c3b910$d3e855a0$6402a8c0@arkdesktop> > I'm not sure about that yet. I'd *like* to find a hack that lets the > int type change representations, but the fact is that it's much easier > to use different types to indicate different representations. Aren't integers immutable? If so, I would think it doesn't make sense for them to change representation, as they don't change value. Anyway, if you want to use type to encode representation, I would think that the various integer types should be related by inheritance. As a long can always substitute for an int, at least in theory, I would think that long should be derived from int. Then isinstance(42L, int) would yield True. If integers are related this way, LSP implies that converting a long to a string should not put an L at the end. From walter at livinglogic.de Tue Dec 2 15:15:51 2003 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Tue Dec 2 15:15:55 2003 Subject: [Python-Dev] pprint and list/tuple/dict subclasses In-Reply-To: <16332.56789.356676.91289@grendel.fdrake.net> References: <3FCCDCFA.6070800@livinglogic.de> <16332.56789.356676.91289@grendel.fdrake.net> Message-ID: <3FCCF2F7.7090805@livinglogic.de> Fred L. Drake, Jr. wrote: > Walter D?rwald writes: > > Currently pprint refuses to prettyprint subclasses of list, > > tuple and dict. Does it make sense to change that for 2.4? > > > > Of course once __repr__() is overwritten by a subclass, > > pprint has to use that instead of its own formatting. > > There's a patch for this that's on my list of things to look at I > think. The discussion associated with the patch had the right idea, > but I've not had time to review the patch (which I think is > embarrassingly short); AFAICT it should only change four lines. > if you'd like to do so, be my guest! Is this patch on SF? What's the patch id? Bye, Walter D?rwald From fdrake at acm.org Tue Dec 2 15:18:23 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue Dec 2 15:18:43 2003 Subject: [Python-Dev] pprint and list/tuple/dict subclasses In-Reply-To: <3FCCF2F7.7090805@livinglogic.de> References: <3FCCDCFA.6070800@livinglogic.de> <16332.56789.356676.91289@grendel.fdrake.net> <3FCCF2F7.7090805@livinglogic.de> Message-ID: <16332.62351.960759.135367@grendel.fdrake.net> Walter D?rwald writes: > Is this patch on SF? What's the patch id? http://www.python.org/sf/750542 -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From guido at python.org Tue Dec 2 15:28:12 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 2 15:28:27 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: Your message of "Tue, 02 Dec 2003 15:14:01 EST." <024201c3b910$d3e855a0$6402a8c0@arkdesktop> References: <024201c3b910$d3e855a0$6402a8c0@arkdesktop> Message-ID: <200312022028.hB2KSC902115@c-24-5-183-134.client.comcast.net> > > I'm not sure about that yet. I'd *like* to find a hack that lets the > > int type change representations, but the fact is that it's much easier > > to use different types to indicate different representations. > > Aren't integers immutable? If so, I would think it doesn't make sense for > them to change representation, as they don't change value. I was using shorthand-speak meaning that different instances of the same class would use a different representation (which the class can somehow recognize by looking at the instance, of course). > Anyway, if you want to use type to encode representation, I would > think that the various integer types should be related by > inheritance. As a long can always substitute for an int, at least > in theory, I would think that long should be derived from int. Then > isinstance(42L, int) would yield True. Or should int be a subclass of long? I believe that OO theorists consider the base class the set with the largest number of elements (since it is the least constrained). Now, all ints are longs, but all longs are not ints, since ints can only represent values in [-sys.maxint-1, sys.maxint]. According to this reasoning, long should be the base class. But the naming suggests different: 'int' suggests no particular size (except to C/C++ users :-) so should be the more general class, and that pleads for your version. I don't particularly like either approach, because the 'long' type is an implementation detail over which the user has no control (this is a change of heart since Python's original design!). I guess that means I have to work harder and make the single int type support both representations. I'm sure it can be done. > If integers are related this way, LSP implies that converting a long > to a string should not put an L at the end. Well, they aren't in Python 2.x, which is why the L stays until 3.0. --Guido van Rossum (home page: http://www.python.org/~guido/) From walter at livinglogic.de Tue Dec 2 15:40:05 2003 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Tue Dec 2 15:40:19 2003 Subject: [Python-Dev] pprint and list/tuple/dict subclasses In-Reply-To: <200312021850.hB2IowQ01754@c-24-5-183-134.client.comcast.net> References: <3FCCDCFA.6070800@livinglogic.de> <200312021850.hB2IowQ01754@c-24-5-183-134.client.comcast.net> Message-ID: <3FCCF8A5.1000309@livinglogic.de> Guido van Rossum wrote: >>Currently pprint refuses to prettyprint subclasses of list, >>tuple and dict. Does it make sense to change that for 2.4? >> >>Of course once __repr__() is overwritten by a subclass, >>pprint has to use that instead of its own formatting. > > > Sounds like a good plan. Do you have a suggested patch? Not yet, but Fred seems to have one. Bye, Walter D?rwald From walter at livinglogic.de Tue Dec 2 15:40:47 2003 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Tue Dec 2 15:40:57 2003 Subject: [Python-Dev] pprint and list/tuple/dict subclasses In-Reply-To: <16332.62351.960759.135367@grendel.fdrake.net> References: <3FCCDCFA.6070800@livinglogic.de> <16332.56789.356676.91289@grendel.fdrake.net> <3FCCF2F7.7090805@livinglogic.de> <16332.62351.960759.135367@grendel.fdrake.net> Message-ID: <3FCCF8CF.8080009@livinglogic.de> Fred L. Drake, Jr. wrote: > Walter D?rwald writes: > > Is this patch on SF? What's the patch id? > > http://www.python.org/sf/750542 OK, I'll take a look at it. Bye, Walter D?rwald From dan at sidhe.org Tue Dec 2 15:46:44 2003 From: dan at sidhe.org (Dan Sugalski) Date: Tue Dec 2 15:46:53 2003 Subject: [Python-Dev] Merry December Message-ID: And a short reminder that it'd be a good idea to put together The Great Pie-thon Benchmark Program. I'd hate to think that the reason I got to splat Guido with a pie is because of a default... :) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From fincher.8 at osu.edu Tue Dec 2 16:54:34 2003 From: fincher.8 at osu.edu (Jeremy Fincher) Date: Tue Dec 2 15:56:40 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <8ylvmcmq.fsf@yahoo.co.uk> References: <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <200311302157.49205.aleaxit@yahoo.com> <8ylvmcmq.fsf@yahoo.co.uk> Message-ID: <200312021654.34386.fincher.8@osu.edu> On Tuesday 02 December 2003 11:02 am, Paul Moore wrote: > Alex Martelli writes: > > Since these functions or types are going to be in operator, I think > > we can afford to "spend" two names to distinguish functionality > > (even though attgetter and itemgetter look nowhere as neat as > > extract -- I don't have better suggestions offhand). > > operator.getattr and operator.getitem? I think the functions they're talking about are actually right-curried versions of those functions, like this: def attrgetter(attr): return lambda x: getattr(x, attr) Jeremy From theller at python.net Tue Dec 2 15:56:41 2003 From: theller at python.net (Thomas Heller) Date: Tue Dec 2 15:56:56 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <200312022002.hB2K28301985@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Tue, 02 Dec 2003 12:02:08 -0800") References: <20031130073109.GA1560@hishome.net> <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <002701c3b606$c61304a0$e841fea9@oemcomputer> <20031130073109.GA1560@hishome.net> <5.1.0.14.0.20031130184217.02e3c1d0@mail.telecommunity.com> <200312010018.hB10IbS29532@c-24-5-183-134.client.comcast.ne t> <5.1.0.14.0.20031130210357.02f62d80@mail.telecommunity.com> <4qwjmbao.fsf@yahoo.co.uk> <200312021918.hB2JIuQ01834@c-24-5-183-134.client.comcast.net> <2msmk3hvm0.fsf@starship.python.net> <200312022002.hB2K28301985@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: >> > It would let you do things like >> > >> >>>> map(X + 1, range(2)) >> >> Something like this? >> >> class Adder: >> def __init__(self, number): >> self._number = number >> def __call__(self, arg): >> return arg + self._number >> >> class X: >> def __add__(self, number): >> return Adder(number) >> >> X = X() >> >> print map(X + 1, range(2)) > > Ah, of course. Nice. This can be extended to __getattr__ and > __getitem__; unfortunately __call__ would be ambiguous. It could > probably be made quite fast with a C implementation. > > Now the question remains, would it be better to hide this and simply > use it under the hood as an alternative way of generating code for > lambda, or should it be some sort of standard library module, to be > invoked explicitly? In favor of the latter pleads that this would > solve the semantic differences with lambda when free variables are > involved: obviously X+q would evaluate q only once, while > (lamda X: X+q) evaluates q on each invocation. Remember that for > generator expressions we've made the decision that > (X+q for X in seq) > should evaluate q only once. The latter has another advantage (or is this a disadvantage of the former?): You can invoke lambda x: x.something with a keyword arg, which would not be possible with a C implemented function, I assume. lambda expressions are often used to implement gui callbacks, and they are sometimes invoked this way. So the former would introduce incompatibilities. Thomas From ark-mlist at att.net Tue Dec 2 16:25:41 2003 From: ark-mlist at att.net (Andrew Koenig) Date: Tue Dec 2 16:25:42 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312022028.hB2KSC902115@c-24-5-183-134.client.comcast.net> Message-ID: <001801c3b91a$d694cc70$6402a8c0@arkdesktop> > > Aren't integers immutable? If so, I would think it doesn't make sense > for them to change representation, as they don't change value. > I was using shorthand-speak meaning that different instances of the > same class would use a different representation (which the class can > somehow recognize by looking at the instance, of course). Got it. > > Anyway, if you want to use type to encode representation, I would > > think that the various integer types should be related by > > inheritance. As a long can always substitute for an int, at least > > in theory, I would think that long should be derived from int. Then > > isinstance(42L, int) would yield True. > Or should int be a subclass of long? I believe that OO theorists > consider the base class the set with the largest number of elements > (since it is the least constrained). Now, all ints are longs, but all > longs are not ints, since ints can only represent values in > [-sys.maxint-1, sys.maxint]. According to this reasoning, long should > be the base class. I think that int should be the base class, because I can imagine long supporting operations that int does not support, but not vice versa. From walter at livinglogic.de Tue Dec 2 18:03:36 2003 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Tue Dec 2 18:03:45 2003 Subject: [Python-Dev] pprint and list/tuple/dict subclasses In-Reply-To: <16332.62351.960759.135367@grendel.fdrake.net> References: <3FCCDCFA.6070800@livinglogic.de> <16332.56789.356676.91289@grendel.fdrake.net> <3FCCF2F7.7090805@livinglogic.de> <16332.62351.960759.135367@grendel.fdrake.net> Message-ID: <3FCD1A48.5000202@livinglogic.de> Fred L. Drake, Jr. wrote: > Walter D?rwald writes: > > Is this patch on SF? What's the patch id? > > http://www.python.org/sf/750542 OK, I've updated the patch. Two other ideas for pprint: The arguments for the PrettyPrinter constructor could be exposed in pprint() and pformat(): """ import pprint print.pprint(foo, indent=3, width=40) """ The indent argument could be a string, which is used for indenting, so indent=" " would be the same as indent=1, indent=" " would be like indent=3, and we could use something like indent="\t" or even indent="\033[1;30m...\033[0m". Bye, Walter D?rwald From greg at cosc.canterbury.ac.nz Tue Dec 2 18:32:17 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Dec 2 18:32:25 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <001801c3b91a$d694cc70$6402a8c0@arkdesktop> Message-ID: <200312022332.hB2NWHc05139@oma.cosc.canterbury.ac.nz> Andrew Koenig > I think that int should be the base class, because I can imagine long > supporting operations that int does not support, but not vice versa. Perhaps 'int' should be an abstract class, with concrete subclasses 'long' and 'short'. And 'bool' would fit into this family as a sort of "extremely short int". Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Tue Dec 2 18:42:11 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Dec 2 18:42:20 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: Message-ID: <200312022342.hB2NgBH05169@oma.cosc.canterbury.ac.nz> Thomas Heller : > The latter has another advantage (or is this a disadvantage of the > former?): You can invoke > > lambda x: x.something > > with a keyword arg, which would not be possible with a C implemented > function, I assume. If you're asking whether a C function can have a keyword argument whose name is determined at runtime, I think that could be arranged -- it would just be a matter of concocting the appropriate arguments to PyArg_ParseTupleAndKeywords dynamically. As to a user-friendly syntax for invoking it, maybe something like arg.x.something Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Tue Dec 2 19:16:30 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Dec 2 19:16:36 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: Message-ID: <200312030016.hB30GU205334@oma.cosc.canterbury.ac.nz> Robert Brewer : > So that instead of (lambda self: self.foo), you'd end up with > (rightcurry(getattr, 'foo')) At the expense of a large drop in readability. We seem to be talking about two different things in this thread, speed and readability. The nice thing about the "attrgetter.x" etc. idea is that it has the potential to provide both at once. The nasty thing about it is that it smells a bit too much like a clever trick. It's really abusing the syntax to make it mean something different from what it usually means. I think I like the idea of optimising lambda, but that doesn't do anything for the readability. So, how about a nicer syntax for lambda? Maybe along the lines of x -> x.something A bonus of introducing a new lambda syntax is that it would provide the opportunity to give it early-binding semantics for free variables, like generator expressions. The old lambda would have to be kept around for a while for programs relying on the old semantics, but it could be deprecated, and removed in 3.0. Jeepers, *another* PEP to get around to writing one day... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From zack at codesourcery.com Tue Dec 2 19:51:26 2003 From: zack at codesourcery.com (Zack Weinberg) Date: Tue Dec 2 19:51:31 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <200312030016.hB30GU205334@oma.cosc.canterbury.ac.nz> (Greg Ewing's message of "Wed, 03 Dec 2003 13:16:30 +1300 (NZDT)") References: <200312030016.hB30GU205334@oma.cosc.canterbury.ac.nz> Message-ID: <87y8turaep.fsf@egil.codesourcery.com> Greg Ewing writes: > I think I like the idea of optimising lambda, but that doesn't do > anything for the readability. So, how about a nicer syntax for lambda? > Maybe along the lines of > > x -> x.something > > A bonus of introducing a new lambda syntax is that it would provide > the opportunity to give it early-binding semantics for free variables, > like generator expressions. The old lambda would have to be kept > around for a while for programs relying on the old semantics, but it > could be deprecated, and removed in 3.0. If you're going to mess with lambda at all, please try to come up with a way for it not to be limited to a single expression. I tried to do this last summer and couldn't get rid of grammar ambiguities. zw From python at rcn.com Wed Dec 3 01:56:20 2003 From: python at rcn.com (Raymond Hettinger) Date: Wed Dec 3 01:57:05 2003 Subject: [Python-Dev] Tutorial: Brief Introduction to the Standard Libary In-Reply-To: <200311271113.05578.aleax@aleax.it> Message-ID: <001701c3b96a$8f1cb100$e841fea9@oemcomputer> Thank you everyone for the ideas on what to include and exclude from the new tutorial section. Attached is a revised draft. Comments, suggestions, nitpicks, complaints, and accolades are welcome. Raymond Hettinger -------------- next part -------------- Brief Tour of some Standard Library Modules Operating System Interface The os module provides many functions for interacting with the operating system: >>> import os >>> os.system('copy /data/mydata.fil /backup/mydata.fil') 0 >>> os.getcwd() 'C:\\Python24' >>> os.chdir('/server/accesslogs') Be sure to use the "import os" style instead of "from os import *". This will keep os.open() from shadowing __builtin__.open() which operates much differently. File Wildcards The glob module provides a function for making file lists from directory wildcard searches: >>> glob.glob('*.py') ['primes.py', 'random.py', 'quote.py'] Command Line Arguments Common utility scripts often invoke processing command line arguments. These arguments are stored in the sys module's argv attribute as a list. For instance the following output results from running "python demo.py one two three" at the command line: >>> import sys >>> print sys.argv[] ['demo.py', 'one', 'two', 'three'] The getopt module processes sys.argv using the conventions of the Unix getopt() function: >>> import getopt >>> # sys.argv is ['myprog.py', '-a', '-b', '-cfoo', '-d', 'bar', 'a1', 'a2'] >>> optlist, args = getopt.getopt(sys.argv[1:], 'abc:d:') >>> optlist [('-a', ''), ('-b', ''), ('-c', 'foo'), ('-d', 'bar')] >>> args ['a1', 'a2'] More powerful and flexible command line processing is provided by the optparse module. Error Output Redirection and Program Termination The sys module also has attributes for stdin, stdout, and stderr. The latter is useful for emitting warnings and error messages to make them visible even when stdout has been redirected: >>> sys.stderr.write('Warning, log file not found starting a new one') Warning, log file not found starting a new one The most direct way to terminate a script is to use sys.exit(). String Pattern Matching The re module provides regular expression tools for advanced string processing. When only simple capabilities are needed, string methods are preferred because they are easier to read and debug. For more sophisticated applications, regular expressions can provide succinct, optimized solutions: >>> import re >>> re.findall(r'\bf[a-z]*', 'which foot or hand fell fastest') ['foot', 'fell', 'fastest'] >>> re.sub(r'(\b[a-z]+) \1', r'\1', 'cat in the the hat') 'cat in the hat' Math Library The math module gives access to the underlying C library functions for floating point math: >>> import math >>> math.cos(math.pi / 4.0) 0.70710678118654757 >>> math.log(1024, 2) 10.0 The random module provides tools for making random selections: >>> import random >>> random.choice(['apple', 'pear', 'banana']) 'apple' >>> random.sample(xrange(100), 10) # sampling without replacement [30, 83, 16, 4, 8, 81, 41, 50, 18, 33] >>> random.random() # random float 0.17970987693706186 >>> random.randrange(6) # random integer chosen from range(6) 4 Internet Access There are a number of modules for accessing the internet and processing internet protocols. Two of the simplest are urllib2 for retrieving data from urls and smtplib for sending mail: >>> import urllib >>> for line in urllib2.urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'): ... if 'EST' in line: # look for Eastern Standard Time ... print line
Nov. 25, 09:43:32 PM EST >>> import smtplib >>> server = smtplib.SMTP('localhost') >>> server.sendmail('soothsayer@tmp.org', 'jceasar@tmp.org', """To: jceasar@tmp.org From: soothsayer@tmp.org Beware the Ides of March. """) >>> server.quit() Dates and Times The datetime module supplies classes for manipulating dates and times in both simple and complex ways. While date and time arithmetic is supported, the focus of the implementation is on efficient member extraction for output formatting and manipulation. The module also supports objects that are time zone aware. # dates are easily constructed and formatted >>> from datetime import date >>> now = date.today() >>> now datetime.date(2003, 12, 2) >>> now.strftime("%m-%d-%y or %d%b %Y is a %A on the %d day of %B") '12-02-03 or 02Dec 2003 is a Tuesday on the 02 day of December' # dates support calendar arithmetic >>> birthday = date(1964, 7, 31) >>> age = now - birthday >>> age.days 14368 Data Compression Common data archiving and compression formats are directly supported by modules including : zlib, gzip, bz2, zipfile, and tar. >>> import zlib >>> s = 'witch which has which witches wrist watch' >>> len(s) 41 >>> t = zlib.compress(s) >>> len(t) 37 >>> zlib.decompress(t) 'witch which has which witches wrist watch' >>> zlib.crc32(t) -1438085031 Performance Measurement Some Python users develop a deep interest in knowing the relative performance between different approaches to the same problem. Python provides a measurement tool that answers those questions immediately. For example, it may be tempting to use the tuple packing and unpacking feature instead of the traditional approach to swapping arguments. The timeit module quickly demonstrates that the traditional approach is faster: >>> from timeit import Timer >>> dir(Timer) >>> Timer('t=a; a=b; b=t', 'a=1; b=1').timeit() 0.60864915603680925 >>> Timer('a,b = b,a', 'a=1; b=1').timeit() 0.8625194857439773 In contrast to timeit's fine level of granularity, the profile and pstats modules provide tools for identifying time critical sections of larger blocks of code. Quality Control One approach for developing high quality software is to write tests for each function as it is developed and to run those tests frequently during the development process. The doctest module provides a tool for scanning a module and validating tests embedded in a program's docstrings. Test construction is as simple as cutting-and-pasting a typical call along with its results into the docstring. This improves the documentation by providing the user with an example and it allows the doctest module to make sure the code remains true to the documentation: def average(values): """Computes the arithmetic mean of a list of numbers. >>> print average([20, 30, 70]) 40.0 """ return sum(values, 0.0) / len(values) import doctest doctest.testmod() # automatically validate the embedded tests The unittest module is not as effortless as the doctest module, but it allows a more comprehensive set of tests to be maintained in a separate file: import unittest class TestStatisticalFunctions(unittest.TestCase): def test_average(self): self.assertEqual(average([20, 30, 70]), 40.0) self.assertEqual(round(average([1, 5, 7]), 1), 4.3) self.assertRaises(ZeroDivisionError, average, []) self.assertRaises(TypeError, average, 20, 30, 70) unittest.main() # Calling from the command line invokes all tests Batteries Included Python has a "batteries included" philosophy. The is best seen through the sophisticated and robust capabilites of its larger packages: * The xmlrpclib and SimpleXMLRPCServer modules make implementing remote procedure calls into an almost trivial task. Despite the names, no direct knowledge or handling of XML is needed. * The email package is a library for managing email messages, including MIME and other RFC 2822-based message documents. Unlike smtplib and poplib which actually send and receive messages, the email package has a complete toolset for building or decoding complex message structures (including attachments) and for implementing internet encoding and header protocols. * The xml.dom and xml.sax packages provide robust support for parsing this popular data interchange format. Likewise, the csv module supports direct reads and writes in a common database format. Together, these modules and packages greatly simplify data interchange between python applications and other tools. * Internationalization is supported by a number of modules including gettext and the codecs package. From raymond.hettinger at verizon.net Wed Dec 3 04:21:32 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Wed Dec 3 04:22:45 2003 Subject: [Python-Dev] unittest output Message-ID: <000a01c3b97e$e8f69d80$e841fea9@oemcomputer> When a test fails, it is accompanied by tracebacks that include all the processing within the unittest module itself. This makes it darned difficult to see at a glance what went wrong. If like me, you use unittest as part of the development process, then you run it hundreds of times each week and this eyesore gets to be a real PITA. I don't recall when or where, but someone has proposed a fix. But, like many great ideas, it got stymied because someone, somewhere has a use for the rest of the traceback. Also, a developer raised the (IMO red herring) issue -- what if there is a bug in the unittest module, how would you track it. So, I would like to re-raise the issue. Can we get this fixed? One other nit: When unittest.main() is run from the new IDLE it fails on a sys.exit() attempt. Can we trap the exit, and just return? Raymond Hettinger From theller at python.net Wed Dec 3 04:34:41 2003 From: theller at python.net (Thomas Heller) Date: Wed Dec 3 04:34:53 2003 Subject: [Python-Dev] unittest output In-Reply-To: <000a01c3b97e$e8f69d80$e841fea9@oemcomputer> (Raymond Hettinger's message of "Wed, 3 Dec 2003 04:21:32 -0500") References: <000a01c3b97e$e8f69d80$e841fea9@oemcomputer> Message-ID: "Raymond Hettinger" writes: > When a test fails, it is accompanied by tracebacks that include > all the processing within the unittest module itself. This makes > it darned difficult to see at a glance what went wrong. If like > me, you use unittest as part of the development process, then you > run it hundreds of times each week and this eyesore gets to be a > real PITA. > > I don't recall when or where, but someone has proposed a fix. > But, like many great ideas, it got stymied because someone, > somewhere has a use for the rest of the traceback. Also, a > developer raised the (IMO red herring) issue -- what if there > is a bug in the unittest module, how would you track it. http://www.python.org/sf/722638 > > So, I would like to re-raise the issue. > Can we get this fixed? Yes, please. Thomas From theller at python.net Wed Dec 3 04:36:59 2003 From: theller at python.net (Thomas Heller) Date: Wed Dec 3 04:37:09 2003 Subject: [Python-Dev] unittest output In-Reply-To: <000a01c3b97e$e8f69d80$e841fea9@oemcomputer> (Raymond Hettinger's message of "Wed, 3 Dec 2003 04:21:32 -0500") References: <000a01c3b97e$e8f69d80$e841fea9@oemcomputer> Message-ID: <8ylu44zo.fsf@python.net> "Raymond Hettinger" writes: > When a test fails, it is accompanied by tracebacks that include > all the processing within the unittest module itself. This makes > it darned difficult to see at a glance what went wrong. If like > me, you use unittest as part of the development process, then you > run it hundreds of times each week and this eyesore gets to be a > real PITA. > > I don't recall when or where, but someone has proposed a fix. > But, like many great ideas, it got stymied because someone, > somewhere has a use for the rest of the traceback. Also, a > developer raised the (IMO red herring) issue -- what if there > is a bug in the unittest module, how would you track it. Well, distutils being mainly a user tool tries it's best to hide tracebacks from the user, and only prints a single 'error: ...' line. But it pays attention to a DISTUTILS_DEBUG env var, which will enable showing the full tracebacks. IIRC. Thomas From python at rcn.com Wed Dec 3 04:56:23 2003 From: python at rcn.com (Raymond Hettinger) Date: Wed Dec 3 04:57:05 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <200312021918.hB2JIuQ01834@c-24-5-183-134.client.comcast.net> Message-ID: <000e01c3b983$b5ca6360$e841fea9@oemcomputer> [Thomas Heller] > > Looking at parts of my codebase nearly all uses of lambda are > > 'lambda self: self.someattr'. Yes, they are everywhere. Getting rid of those lambdas was part of the attraction for attrgetter(). > But, at least for attrgetter, I am slightly unhappy with the outcome, > because the attribute name is now expressed as a string literal rather > than using attribute notation. This makes it harder to write > automated tools that check or optimize code. (For itemgetter it > doesn't really matter, since the index is a literal either way.) > > So, while I'm not particularly keen on lambda, I'm not that keen on > attrgetter either. But what could be better? I don't know if you like this, but there is a way to change the interface to attrgetter() so that the dot notation can be used instead of a string. It produces the same result and is neater, but I find it somewhat harder to explain: import operator class NewAttrGetter(object): def __getattribute__(self, name): return operator.attrgetter(name) newattrgetter = NewAttrGetter() class A: pass a = A() a.score = 10 getscore = operator.attrgetter('score') print getscore(a) getscore = newattrgetter.score print getscore(a) new-style-classes-rock-ly yours, Raymond Hettinger From python at rcn.com Wed Dec 3 05:02:03 2003 From: python at rcn.com (Raymond Hettinger) Date: Wed Dec 3 05:02:46 2003 Subject: [Python-Dev] unittest output In-Reply-To: Message-ID: <000f01c3b984$80bb8360$e841fea9@oemcomputer> > > When a test fails, it is accompanied by tracebacks that include > > all the processing within the unittest module itself. This makes > > it darned difficult to see at a glance what went wrong. If like > > me, you use unittest as part of the development process, then you > > run it hundreds of times each week and this eyesore gets to be a > > real PITA. > > > > I don't recall when or where, but someone has proposed a fix. > > But, like many great ideas, it got stymied because someone, > > somewhere has a use for the rest of the traceback. Also, a > > developer raised the (IMO red herring) issue -- what if there > > is a bug in the unittest module, how would you track it. > > http://www.python.org/sf/722638 Arghh. I had forgotten that Steve Purcell weighed in on this. It's his module. So unless he can be persuaded, the point is moot :-( > > So, I would like to re-raise the issue. > > Can we get this fixed? > > Yes, please. Raymond Hettinger From python at rcn.com Wed Dec 3 05:30:59 2003 From: python at rcn.com (Raymond Hettinger) Date: Wed Dec 3 05:31:41 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <000e01c3b983$b5ca6360$e841fea9@oemcomputer> Message-ID: <001001c3b988$8b1054e0$e841fea9@oemcomputer> [Guido] > > But, at least for attrgetter, I am slightly unhappy with the outcome, > > because the attribute name is now expressed as a string literal rather > > than using attribute notation. This makes it harder to write > > automated tools that check or optimize code. (For itemgetter it > > doesn't really matter, since the index is a literal either way.) > > > > So, while I'm not particularly keen on lambda, I'm not that keen on > > attrgetter either. But what could be better? [Me, with mildly wacky idea] > I don't know if you like this, but there is a way to change the > interface to attrgetter() so that the dot notation can be used > instead of a string. It produces the same result and is neater, > but I find it somewhat harder to explains Carried to the limit, the idea turns into something that is either sublime or severely bonkers. The good points are that Guido gets his dotted access and I get to trade in the two ugly names and for a single beautiful "extract". And there's no performance cost, the inner loop is the same. The downside is I still don't know how to explain it (AFAICT, super() is the closest thing to it): import operator class ExtractorClass(object): def __getattribute__(self, attr): return operator.attrgetter(attr) def __getitem__(self, key): return operator.itemgetter(key) extract = ExtractorClass() class A: pass a = A() a.score = 10 getscore = extract.score # Houston, we have dotted access print getscore(a) b = [10, 20, 30, 40] getsecond = extract[1] # and, we have the bracketed lookups print getsecond(b) animal_weights = [('cat', 10), ('dog', 90), ('human', 150), ('goldfish', 0.1), ('unladen_sparrow', 3)] list.sorted(animal_weights, key=extract[1]) list.sorted(students, key=extract.score) So, now we have a weird bird that is faster and better looking than lambda. Raymond From mwh at python.net Wed Dec 3 05:39:05 2003 From: mwh at python.net (Michael Hudson) Date: Wed Dec 3 05:39:08 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <200312021926.hB2JQeP01889@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Tue, 02 Dec 2003 11:26:40 -0800") References: <20031130073109.GA1560@hishome.net> <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <002701c3b606$c61304a0$e841fea9@oemcomputer> <20031130073109.GA1560@hishome.net> <5.1.0.14.0.20031130184217.02e3c1d0@mail.telecommunity.com> <200312010018.hB10IbS29532@c-24-5-183-134.client.comcast.ne t> <5.1.0.14.0.20031130210357.02f62d80@mail.telecommunity.com> <4qwjmbao.fsf@yahoo.co.uk> <200312021918.hB2JIuQ01834@c-24-5-183-134.client.comcast.net> <2msmk3hvm0.fsf@starship.python.net> <200312021926.hB2JQeP01889@c-24-5-183-134.client.comcast.net> Message-ID: <2m7k1eyyly.fsf@starship.python.net> Guido van Rossum writes: >> Doesn't have to be a keyword... I implemented something like this >> years ago and then ditched it when list comps appeared. >> >> It would let you do things like >> >> >>> map(X + 1, range(2)) >> [1, 2, 3] > > What was your notation like? Did you actually use 'X'? Um, I think so. I defined it in my $PYTHONSTARTUP file as an instance of a class _X, or something like that. Cheers, mwh -- In many ways, it's a dull language, borrowing solid old concepts from many other languages & styles: boring syntax, unsurprising semantics, few automatic coercions, etc etc. But that's one of the things I like about it. -- Tim Peters, 16 Sep 93 From mwh at python.net Wed Dec 3 05:42:44 2003 From: mwh at python.net (Michael Hudson) Date: Wed Dec 3 05:42:46 2003 Subject: [Python-Dev] minor interruption to service. In-Reply-To: <16332.62062.305234.903783@montanaro.dyndns.org> (Skip Montanaro's message of "Tue, 2 Dec 2003 14:13:34 -0600") References: <200311290344.hAT3iHF5013034@maxim.off.ekorp.com> <16332.62062.305234.903783@montanaro.dyndns.org> Message-ID: <2m3cc2yyfv.fsf@starship.python.net> Skip Montanaro writes: > Anthony> In the meantime, if someone wants to take on the "upgrade to > Anthony> autoconf 2.59" task, I'd appreciate it very much. > > Do you mean 2.58? No: http://article.gmane.org/gmane.comp.sysutils.autotools.announce/27 > I looked here: > > http://ftp.gnu.org/gnu/autoconf/ > > and 2.58 appears to be the latest release. Don't know whats going on there. Cheers, mwh -- > say-hi-to-the-flying-pink-elephants-for-me-ly y'rs, No way, the flying pink elephants are carrying MACHINE GUNS! Aiiee!! Time for a kinder, gentler hallucinogen... -- Barry Warsaw & Greg Ward, python-dev From mwh at python.net Wed Dec 3 05:52:38 2003 From: mwh at python.net (Michael Hudson) Date: Wed Dec 3 05:52:41 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <200312022002.hB2K28301985@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Tue, 02 Dec 2003 12:02:08 -0800") References: <20031130073109.GA1560@hishome.net> <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <002701c3b606$c61304a0$e841fea9@oemcomputer> <20031130073109.GA1560@hishome.net> <5.1.0.14.0.20031130184217.02e3c1d0@mail.telecommunity.com> <200312010018.hB10IbS29532@c-24-5-183-134.client.comcast.ne t> <5.1.0.14.0.20031130210357.02f62d80@mail.telecommunity.com> <4qwjmbao.fsf@yahoo.co.uk> <200312021918.hB2JIuQ01834@c-24-5-183-134.client.comcast.net> <2msmk3hvm0.fsf@starship.python.net> <200312022002.hB2K28301985@c-24-5-183-134.client.comcast.net> Message-ID: <2my8tuxjex.fsf@starship.python.net> Guido van Rossum writes: >> > It would let you do things like >> > >> >>>> map(X + 1, range(2)) >> >> Something like this? >> >> class Adder: >> def __init__(self, number): >> self._number = number >> def __call__(self, arg): >> return arg + self._number >> >> class X: >> def __add__(self, number): >> return Adder(number) >> >> X = X() >> >> print map(X + 1, range(2)) > > Ah, of course. Nice. This can be extended to __getattr__ and > __getitem__; unfortunately __call__ would be ambiguous. Yes. I think I used .c() for that. IIRC correctly, I also complicated things to the extent that there was a special object, say N, and you could write >>> map(X.split(N), ['a a', 'b b\nb'], [None, '\n']) [['a', 'a'], ['b b', 'b']] This might be going a bit far... > It could probably be made quite fast with a C implementation. Would be tedious to write though :-) > Now the question remains, would it be better to hide this and simply > use it under the hood as an alternative way of generating code for > lambda, or should it be some sort of standard library module, to be > invoked explicitly? In favor of the latter pleads that this would > solve the semantic differences with lambda when free variables are > involved: obviously X+q would evaluate q only once, while > (lamda X: X+q) evaluates q on each invocation. Remember that for > generator expressions we've made the decision that > (X+q for X in seq) > should evaluate q only once. I am not a fan of the idea of compiling certain kinds of lambdas differently. Cheers, mwh -- While preceding your entrance with a grenade is a good tactic in Quake, it can lead to problems if attempted at work. -- C Hacking -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html From mwh at python.net Wed Dec 3 05:54:59 2003 From: mwh at python.net (Michael Hudson) Date: Wed Dec 3 05:55:02 2003 Subject: [Python-Dev] pprint and list/tuple/dict subclasses In-Reply-To: <3FCD1A48.5000202@livinglogic.de> ( =?iso-8859-1?q?Walter_D=F6rwald's_message_of?= "Wed, 03 Dec 2003 00:03:36 +0100") References: <3FCCDCFA.6070800@livinglogic.de> <16332.56789.356676.91289@grendel.fdrake.net> <3FCCF2F7.7090805@livinglogic.de> <16332.62351.960759.135367@grendel.fdrake.net> <3FCD1A48.5000202@livinglogic.de> Message-ID: <2mu14ixjb0.fsf@starship.python.net> Walter D?rwald writes: > The arguments for the PrettyPrinter constructor could > be exposed in pprint() and pformat(): > """ > import pprint > print.pprint(foo, indent=3, width=40) > """ +1e6! I've wanted this for so long I forgot that I could do something +about it :-) Cheers, mwh -- at any rate, I'm satisfied that not only do they know which end of the pointy thing to hold, but where to poke it for maximum effect. -- Eric The Read, asr, on google.com From mwh at python.net Wed Dec 3 05:56:35 2003 From: mwh at python.net (Michael Hudson) Date: Wed Dec 3 05:56:38 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312022332.hB2NWHc05139@oma.cosc.canterbury.ac.nz> (Greg Ewing's message of "Wed, 03 Dec 2003 12:32:17 +1300 (NZDT)") References: <200312022332.hB2NWHc05139@oma.cosc.canterbury.ac.nz> Message-ID: <2mptf6xj8c.fsf@starship.python.net> Greg Ewing writes: > Andrew Koenig > >> I think that int should be the base class, because I can imagine long >> supporting operations that int does not support, but not vice versa. > > Perhaps 'int' should be an abstract class, with concrete > subclasses 'long' and 'short'. That's quite a nice idea -- a bit like an Objective C class cluster. Cheers, mwh -- ZAPHOD: You know what I'm thinking? FORD: No. ZAPHOD: Neither do I. Frightening isn't it? -- The Hitch-Hikers Guide to the Galaxy, Episode 11 From ncoghlan at iinet.net.au Wed Dec 3 06:18:32 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Wed Dec 3 06:18:38 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312022028.hB2KSC902115@c-24-5-183-134.client.comcast.net> References: <024201c3b910$d3e855a0$6402a8c0@arkdesktop> <200312022028.hB2KSC902115@c-24-5-183-134.client.comcast.net> Message-ID: <3FCDC688.4080001@iinet.net.au> Guido van Rossum wrote: > I don't particularly like either approach, because the 'long' type is > an implementation detail over which the user has no control (this is a > change of heart since Python's original design!). I guess that means > I have to work harder and make the single int type support both > representations. I'm sure it can be done. Isn't the 'long' format the more general representation, with the 'int' format then being a performance hack to take advantage of the speed of C integer arithmetic? I'm just wondering if a different way of thinking about it might help with figuring out how to handle a combined implementation. Cheers, Nick. From ncoghlan at iinet.net.au Wed Dec 3 06:48:49 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Wed Dec 3 06:48:59 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <001001c3b988$8b1054e0$e841fea9@oemcomputer> References: <001001c3b988$8b1054e0$e841fea9@oemcomputer> Message-ID: <3FCDCDA1.2040402@iinet.net.au> Raymond Hettinger wrote: > Carried to the limit, the idea turns into something that is > either sublime or severely bonkers. The good points are that > Guido gets his dotted access and I get to trade in the two ugly > names and for a single beautiful "extract". And there's no performance > cost, the inner loop is the same. The downside is I still don't > know how to explain it (AFAICT, super() is the closest thing to it): What you end up with is still a unary operator, so it could still live in the operator module, to. And I think what you posted would work for strings as dictionary keys, too - answering another of the objections to the original operator.extract Which leaves figuring out a concise explanation for what the hell it does (without using lambda in the examples, since I assume part of the aim here is to avoid explaining lambda to people who don't need it). . . "operator.extract provides an interim target for an attribute or item access where the real target of the access is to be determined later. The access is made normally (i.e. dotted notation or indexing), with 'operator.extract' substituted where the target would normally be written. The actual access is carried out by calling the result returned by the expression operator.extract is part of with the real target as the first argument. E.g. y = operator.extract.foo is equivalent to def y(x): return x.foo y = operator.extract[2] is equivalent to def y(x): return x[2] " If you meant a concise explanation of _how_ it does what it does, then I'm not sure that can be done :) And I still don't know if I should be admiring this or running away screaming! Cheers, Nick. From ajsiegel at optonline.net Tue Dec 2 23:21:42 2003 From: ajsiegel at optonline.net (Arthur) Date: Wed Dec 3 07:01:49 2003 Subject: [Python-Dev] Tutorial: Brief Introduction to the Standard Libary Message-ID: <0HPB006FGILSWO@mta2.srv.hcvlny.cv.net> No mention of the copy module. Is this open for discussion? Raymond had written: >My first draft included: > copy, glob, shelve, pickle, os, re, math/cmath, urllib, smtplib >Guido's thoughts: >- copy tends to be overused by beginners While Guido offers no evidence, I have no evidence to dispute him. Let's accept it as true. [Snip] >While I think of copy() and deepcopy() as builtins that got tucked away >in module, Guido is right about their rarity in well-crafted code. Accepting Guido is right and you are right in acknowledging that, why is it that you, nonetheless, think of copy()/deepcopy() as displaced built-ins. I would argue because you acknowledge they are conceptually core - though (we are accepting) not functionally core. OTOH, functionally, haven't some built-ins been growing copy methods? Could be wrong about that - seem to get that idea somewhere. Is it unreasonable to argue that hiding - isn't that is what is being advocated, in essence - something that is arguably conceptually important, cannot properly be considered to be a service to beginners? Is it totally out-of-left field to argue that confronting "copy" forces a confrontation with the fundamentals of assignment - perhaps the most subtle concept faced in approaching Python. Particularly difficult because the beginner does not know to expect subtleties. I would argue that these subtleties are *always* surprising - particularly and importantly because of an important missing clue. That clue, specifically, the existence of copy. I had always assumed that the absence of copy()/deepcopy() from built-ins was for 'big boy' reasons. And accepted it as that - feeling, in general, that the big boy issues should override the 'good for beginners' issue. And only take up the cause of copy() to the extent that - on better evidence - it appears to be being handled as it is for paternalistic reasons. I am closer to being a beginner than most. And nonetheless have in all debates concluded with some confidence that what makes sense for Python, independent of any focus on the needs of beginners, is in fact what makes sense for beginners. I believe that - at core - that is a position respectful of Python and its BDFL, though it seems to put me consistently in conflict with it and him. Art From tzot at sil-tec.gr Wed Dec 3 07:17:44 2003 From: tzot at sil-tec.gr (Christos Georgiou) Date: Wed Dec 3 07:18:07 2003 Subject: [Python-Dev] Re: "groupby" iterator Message-ID: I don't know if this will be useful to anyone, but what the heck... This post does not propose a complete substitute for lambda, but covers most of its use cases using current python capabilities and syntax. It's a very similar idea to what Michael Hudson already has mentioned. And it doesn't apply to lambdas with more than one argument. (There is also a module of mine, at http://www.sil-tec.gr/~tzot/predicates.py that implements this idea in a somehow harsh way to implement all and any predicates on iterables --check at the end of the module for example usage.) The whole idea is a C-implemented (after all, extract et al are about speed) class, say Evaluator or Eval for short, which has an implementation for all __*__ special methods; calling (directly or indirectly) any special method of an instance of Eval returns a callable that does exactly that operation to its argument. Precedence etc are taken care of by python. So, this way, >>> myf= lambda x: x**2 + 1 is equivalent to >>> myf= Eval()**2 + 1 attrgetter("attribute_name") becomes Eval().attribute_name and indexgetter(index) becomes Eval()[index] However, I have no obvious way to replace stuff as lambda x: x[:2] + x[-2:] or lambda x: divmod(x, 6) although in my mentioned module above I have a special .do() method for the latter case, but I presume most of you would find that ugly. So, this idea covers the attrgetter and indexgetter use cases and then some more, but it is not a complete lambda replacement. Just my two eurocents :-| -- Christos Georgiou, Customer Support Engineer Silicon Technologies S.A. 41-45 Kifissias Ave, Maroussi 151 23 Greece Tel +30 (21) 06152600 (ext. 605) Fax +30 (21) 06198140 "Dave always exaggerated." --HAL From mwh at python.net Wed Dec 3 07:40:05 2003 From: mwh at python.net (Michael Hudson) Date: Wed Dec 3 07:40:32 2003 Subject: [Python-Dev] Tutorial: Brief Introduction to the Standard Libary In-Reply-To: <0HPB006FGILSWO@mta2.srv.hcvlny.cv.net> (Arthur's message of "Tue, 02 Dec 2003 23:21:42 -0500") References: <0HPB006FGILSWO@mta2.srv.hcvlny.cv.net> Message-ID: <2mllpuxefu.fsf@starship.python.net> I don't want to be rude -- is English your first language? -- but I found this article to be written in particularly baroque prose. I'm not quite sure what your central point is. Arthur writes: > No mention of the copy module. Is this open for discussion? Well, I dunno. If it was my decision, there's no way it would be mentioned in the builtins. > OTOH, functionally, haven't some built-ins been growing copy methods? Could > be wrong about that - seem to get that idea somewhere. No changes here in a long while. > Is it unreasonable to argue that hiding - isn't that is what is being > advocated, in essence - something that is arguably conceptually important, > cannot properly be considered to be a service to beginners? I don't think it's unreasonable at all. To me the copy module is something you only need when you really need it, then you go looking for it. > Is it totally out-of-left field to argue that confronting "copy" forces a > confrontation with the fundamentals of assignment - perhaps the most subtle > concept faced in approaching Python. Certainly, the failure to understand assignment and erroneous beliefs that the copy module is necessary in a given situation go hand in hand. Assignment in Python *isn't* particularly subtle, though. My impression is that it's people from e.g. C who get tripped up here, not total newbies. > Particularly difficult because the beginner does not know to expect > subtleties. I would argue that these subtleties are *always* > surprising - particularly and importantly because of an important > missing clue. That clue, specifically, the existence of copy. > > I had always assumed that the absence of copy()/deepcopy() from built-ins > was for 'big boy' reasons. Another reason might just be that copy() is implemented in Python... > And accepted it as that - feeling, in general, that the big boy > issues should override the 'good for beginners' issue. And only > take up the cause of copy() to the extent that - on better evidence > - it appears to be being handled as it is for paternalistic > reasons. I am closer to being a beginner than most. Have you ever used the copy module? I am *not* a beginner, and have used it *once* (and I can't remember what for, either). Cheers, mwh -- if-you-need-your-own-xxx.py-you-know-where-to-shove-it-ly y'rs - tim -- Tim Peters dishes out versioning advice on python-dev From guido at python.org Wed Dec 3 08:50:55 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 3 08:51:05 2003 Subject: [Python-Dev] Tutorial: Brief Introduction to the Standard Libary In-Reply-To: Your message of "Wed, 03 Dec 2003 01:56:20 EST." <001701c3b96a$8f1cb100$e841fea9@oemcomputer> References: <001701c3b96a$8f1cb100$e841fea9@oemcomputer> Message-ID: <200312031350.hB3DotY03405@c-24-5-183-134.client.comcast.net> > Thank you everyone for the ideas on what to include and exclude from > the new tutorial section. > > Attached is a revised draft. Comments, suggestions, nitpicks, > complaints, and accolades are welcome. This version looks great! --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Dec 3 09:09:58 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 3 09:10:12 2003 Subject: [Python-Dev] unittest output In-Reply-To: Your message of "Wed, 03 Dec 2003 05:02:03 EST." <000f01c3b984$80bb8360$e841fea9@oemcomputer> References: <000f01c3b984$80bb8360$e841fea9@oemcomputer> Message-ID: <200312031409.hB3E9wt03471@c-24-5-183-134.client.comcast.net> > > > When a test fails, it is accompanied by tracebacks that include > > > all the processing within the unittest module itself. This makes > > > it darned difficult to see at a glance what went wrong. If like > > > me, you use unittest as part of the development process, then you > > > run it hundreds of times each week and this eyesore gets to be a > > > real PITA. > > > > > > I don't recall when or where, but someone has proposed a fix. > > > But, like many great ideas, it got stymied because someone, > > > somewhere has a use for the rest of the traceback. Also, a > > > developer raised the (IMO red herring) issue -- what if there > > > is a bug in the unittest module, how would you track it. > > > > http://www.python.org/sf/722638 > > Arghh. I had forgotten that Steve Purcell weighed in on this. > It's his module. So unless he can be persuaded, the point is moot :-( Not so fast. The patch claims to only suppress the traceback if the test *failed* (e.g. called self.fail(), or one of the assertXXX variations failed). If the test raised an unexpected exception (unittest calls this an *error*) the traceback is printed normally. This seems right to me. A refinement could be to make the output *look* like a (short) traceback as printed by the traceback module even in the failure case; this would address Steve's issue tht IDEs find lines in the code via the traceback. And there should be a command line switch and an environment variable to show the full traceback in all cases. I'd also plead for a switch to let unittest *not* catch exceptions at all -- there are situations where you really want to invoke a debugger. The IDLE issue is separate; I agree that IDLE should treat SystemExit differently (simply go back to the >>> prompt without printing a traceback, printing the status only if it is true). You should add this to the idlefork SF tracker so Kurt can address it. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Dec 3 09:16:14 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 3 09:16:25 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: Your message of "Wed, 03 Dec 2003 10:56:35 GMT." <2mptf6xj8c.fsf@starship.python.net> References: <200312022332.hB2NWHc05139@oma.cosc.canterbury.ac.nz> <2mptf6xj8c.fsf@starship.python.net> Message-ID: <200312031416.hB3EGEn03502@c-24-5-183-134.client.comcast.net> > > Andrew Koenig > >> I think that int should be the base class, because I can imagine long > >> supporting operations that int does not support, but not vice versa. > Greg Ewing writes: > > Perhaps 'int' should be an abstract class, with concrete > > subclasses 'long' and 'short'. MWH: > That's quite a nice idea -- a bit like an Objective C class cluster. I've been pondering this one too. The only downside I can think of is that code that inherits from class int to add some method (e.g. a class hexint whose repr() and str() call hex() instead) will no longer inherit any implementation, and thus won't be very useful. Inheriting from short or long doesn't quite solve the problem either. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Dec 3 09:17:21 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 3 09:17:32 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: Your message of "Wed, 03 Dec 2003 21:18:32 +1000." <3FCDC688.4080001@iinet.net.au> References: <024201c3b910$d3e855a0$6402a8c0@arkdesktop> <200312022028.hB2KSC902115@c-24-5-183-134.client.comcast.net> <3FCDC688.4080001@iinet.net.au> Message-ID: <200312031417.hB3EHLi03515@c-24-5-183-134.client.comcast.net> > Guido van Rossum wrote: > > I don't particularly like either approach, because the 'long' type is > > an implementation detail over which the user has no control (this is a > > change of heart since Python's original design!). I guess that means > > I have to work harder and make the single int type support both > > representations. I'm sure it can be done. Nick Coghlan: > Isn't the 'long' format the more general representation, with the 'int' > format then being a performance hack to take advantage of the speed of C > integer arithmetic? > > I'm just wondering if a different way of thinking about it might help > with figuring out how to handle a combined implementation. I proposed that too, but Andrew Koenig didn't like it. --Guido van Rossum (home page: http://www.python.org/~guido/) From duncan at rcp.co.uk Wed Dec 3 09:25:25 2003 From: duncan at rcp.co.uk (Duncan Booth) Date: Wed Dec 3 09:25:35 2003 Subject: [Python-Dev] Re: "groupby" iterator References: <001001c3b988$8b1054e0$e841fea9@oemcomputer> <3FCDCDA1.2040402@iinet.net.au> Message-ID: Nick Coghlan wrote in news:3FCDCDA1.2040402@iinet.net.au: > Raymond Hettinger wrote: >> Carried to the limit, the idea turns into something that is >> either sublime or severely bonkers. The good points are that >> Guido gets his dotted access and I get to trade in the two ugly >> names and for a single beautiful "extract". And there's no >> performance cost, the inner loop is the same. The downside is I >> still don't know how to explain it (AFAICT, super() is the closest >> thing to it): > > What you end up with is still a unary operator, so it could still live > in the operator module, to. > > And I think what you posted would work for strings as dictionary keys, > too - answering another of the objections to the original > operator.extract > You could even consider reinstating the ability to chain extract operations. (Although this may be even harder to explain). The version below allows attribute and item access to be chained and mixed freely: class ExtractorClass(object): def __init__(self, fn=None, arg=None, parent=None): if fn is None: extract = [] else: extract = [(fn,arg)] if parent is not None: extract = parent._accessors + extract self._accessors = extract def __getattribute__(self, attr): if attr == '_accessors': return object.__getattribute__(self, attr) return ExtractorClass(getattr, attr, self) def __getitem__(self, key): return ExtractorClass(operator.getitem, key, self) def __call__(self, obj): for fn, arg in self._accessors: obj=fn(obj, arg) return obj extract = ExtractorClass() class A: pass a = A() a.score = 10 b = A() a.b = b b.score = 42 getscore = extract.score # Houston, we have dotted access print getscore(a) getsubscore = extract.b.score # Chaining print getsubscore(a) b = [10, 20, 30, 40] getsecond = extract[1] # and, we have the bracketed lookups print getsecond(b) a.b = b getbsecond = extract.b[1] # Chain a mix of attributes and indexes. print getbsecond(a) -- Duncan Booth duncan@rcp.co.uk int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3" "\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure? From theller at python.net Wed Dec 3 09:40:42 2003 From: theller at python.net (Thomas Heller) Date: Wed Dec 3 09:40:55 2003 Subject: [Python-Dev] unittest output In-Reply-To: <200312031409.hB3E9wt03471@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Wed, 03 Dec 2003 06:09:58 -0800") References: <000f01c3b984$80bb8360$e841fea9@oemcomputer> <200312031409.hB3E9wt03471@c-24-5-183-134.client.comcast.net> Message-ID: <3cc22cd1.fsf@python.net> Guido van Rossum writes: >> > > When a test fails, it is accompanied by tracebacks that include >> > > all the processing within the unittest module itself. This makes >> > > it darned difficult to see at a glance what went wrong. If like >> > > me, you use unittest as part of the development process, then you >> > > run it hundreds of times each week and this eyesore gets to be a >> > > real PITA. >> > > >> > > I don't recall when or where, but someone has proposed a fix. >> > > But, like many great ideas, it got stymied because someone, >> > > somewhere has a use for the rest of the traceback. Also, a >> > > developer raised the (IMO red herring) issue -- what if there >> > > is a bug in the unittest module, how would you track it. >> > >> > http://www.python.org/sf/722638 >> >> Arghh. I had forgotten that Steve Purcell weighed in on this. >> It's his module. So unless he can be persuaded, the point is moot :-( > > Not so fast. > > The patch claims to only suppress the traceback if the test *failed* > (e.g. called self.fail(), or one of the assertXXX variations failed). > If the test raised an unexpected exception (unittest calls this an > *error*) the traceback is printed normally. This seems right to me. > > A refinement could be to make the output *look* like a (short) > traceback as printed by the traceback module even in the failure case; > this would address Steve's issue tht IDEs find lines in the code via > the traceback. And there should be a command line switch and an > environment variable to show the full traceback in all cases. The patch does this, also. My original usecase was that I wanted Xemacs to automatically go to the line of the failed test, and not inside the unittest source line where the exception was raised. And the patch *only* suppresses the deepest frame (that inside the unittest source code), if the traceback is n levels deep, n-1 levels are printed. At least this was my intention, if the patch has a chance to make it in, I'll try it again, and bring it up to date if needed. (Another question is this: is raising an exception the right thing to do when a test fails? I'm not so sure, although changing this would be a much bigger change...) Thomas From guido at python.org Wed Dec 3 10:00:04 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 3 10:00:16 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: Your message of "Wed, 03 Dec 2003 13:16:30 +1300." <200312030016.hB30GU205334@oma.cosc.canterbury.ac.nz> References: <200312030016.hB30GU205334@oma.cosc.canterbury.ac.nz> Message-ID: <200312031500.hB3F04b03619@c-24-5-183-134.client.comcast.net> (This thread has nothing to do with the groupby iterator any more, but I'm loathe to change the subject since so many messages are already under this one.) (I've read quite a few messages posted after Greg's post, but Greg still summarizes the issue best for me, *and* it has an alternative idea that needs a response.) [Greg Ewing] > We seem to be talking about two different things in this thread, > speed and readability. The nice thing about the "attrgetter.x" > etc. idea is that it has the potential to provide both at once. > > The nasty thing about it is that it smells a bit too much like a > clever trick. It's really abusing the syntax to make it mean > something different from what it usually means. It is also somewhat weak in that it only addresses lambdas with one argument, and only allows a single reference to that argument in the resulting expression, and can't really be made to handle method calls without more gross notational hacks -- even though it *can* be made to handle arbitrary binary and unary operators. Yet, it captures 90% of the use cases quite well. I also wonder if the simple trick of requiring to call a "constructor" on each use might not make it more palatable. I.e., instead of writing map(Voodoo.address[0], database) you'd write map(Voodoo().address[0], database) where you can replace Voodoo with a name of your choice, perhaps operator.extract -- although I think this is too different to belong in the operator module. Nick Goghlan showed that a pretty readable brief explanation *can* be written. On the other hand... > I think I like the idea of optimising lambda, but that doesn't do > anything for the readability. It's also been shown by now to be a bad idea -- the semantic differences are too subtle (e.g. keyword args). > So, how about a nicer syntax for lambda? Maybe along the lines of > > x -> x.something > > A bonus of introducing a new lambda syntax is that it would provide > the opportunity to give it early-binding semantics for free > variables, like generator expressions. This is what everyone seems to expect and want of lambda anyway... > The old lambda would have to be kept around for a while for programs > relying on the old semantics, but it could be deprecated, and > removed in 3.0. I'm not sure that the -> notation is more understandable than lambda; it would surely confuse C/C++ programmers who are new to Python. Scary thought: how about simply introducing early-binding semantics for lambda in 3.0? Another radical idea would be to use an anonymous-block notation like Smalltalk and Ruby. We could use some kind of funky brackets like [|...|]. A lambda would require an argument notation too. I believe Ruby uses [|x| x+1] where we would write lambda x: x+1, maybe we could use [|x: x+1|]. (I like structure with an explicit close more than open ones like lambda.) Yet another far-out thought: I'd hoped to have gotten rid of most use cases for lambda with list comprehensions, recently generalized into generator expressions. But we keep inventing things (like list.sort(key=), and now groupby(key=)) that aren't expressible using generator expressions. Perhaps we should try harder to find a generalization that covers these cases too, or to define APIs that *can* be used with generator expressions? For groupby, the best I can think of would be to change its API to take an iterable of (key, value) pairs, so you could write: groupby((x.key, x) for x in sequence) instead of groupby(sequence, lambda x: x.key) but that doesn't work for list.sort(), where the sequence already exists, and the whole point is to avoid having to make the explicit decorate-sort-undecorate step. (Well, the groupby does the decorate part sort-of explicit and avoids the undecorate, so it gets there at least halfway.) I guess the most radical idea would be to have the scope of a generator expression extend to other arguments of the same call, so you could write groupby(x for x in sequence, x.key) but that looks too subtle, not to mention ambiguous, and perhaps unimplementable -- what if it was instead groupby(x.value for x in sequence, x.key) ??? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Dec 3 10:05:16 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 3 10:05:27 2003 Subject: [Python-Dev] unittest output In-Reply-To: Your message of "Wed, 03 Dec 2003 15:40:42 +0100." <3cc22cd1.fsf@python.net> References: <000f01c3b984$80bb8360$e841fea9@oemcomputer> <200312031409.hB3E9wt03471@c-24-5-183-134.client.comcast.net> <3cc22cd1.fsf@python.net> Message-ID: <200312031505.hB3F5Gj03643@c-24-5-183-134.client.comcast.net> > http://www.python.org/sf/722638 > > A refinement could be to make the output *look* like a (short) > > traceback as printed by the traceback module even in the failure case; > > this would address Steve's issue tht IDEs find lines in the code via > > the traceback. And there should be a command line switch and an > > environment variable to show the full traceback in all cases. > > The patch does this, also. My original usecase was that I wanted Xemacs > to automatically go to the line of the failed test, and not inside the > unittest source line where the exception was raised. Right. I have the same use case. > And the patch *only* suppresses the deepest frame (that inside the > unittest source code), if the traceback is n levels deep, n-1 levels > are printed. Ah. I didn't look at the patch, but in your description you also suppress the line Traceback (most recent call last): *and* you only show a one-item traceback, so that wasn't very clear. > At least this was my intention, if the patch has a chance to make it in, > I'll try it again, and bring it up to date if needed. Please do -- feel free to add a pointer to this thread to the SF tracker. > (Another question is this: is raising an exception the right thing to do > when a test fails? I'm not so sure, although changing this would be a > much bigger change...) Well, we're stuck with that because everybody writes tests that contain many assertEqual() (of failIf() etc.) calls in a row and expects the control flow to end if one such call fails. Without raising an exception the rest of the test function would be executed regardless, and that would probably raise the probability of unexpected exceptions dramatically. So don't bring that issue up in the same SF item... --Guido van Rossum (home page: http://www.python.org/~guido/) From python at rcn.com Wed Dec 3 10:27:49 2003 From: python at rcn.com (Raymond Hettinger) Date: Wed Dec 3 10:28:32 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <200312031500.hB3F04b03619@c-24-5-183-134.client.comcast.net> Message-ID: <003a01c3b9b2$02c3a7c0$e841fea9@oemcomputer> > (This thread has nothing to do with the groupby iterator any more, but > I'm loathe to change the subject since so many messages are already > under this one.) > > (I've read quite a few messages posted after Greg's post, but Greg > still summarizes the issue best for me, *and* it has an alternative > idea that needs a response.) I'm am still curious to hear your thoughts on the either sublime or crazy idea for a generalized extractor object: list.sorted(students, key=extract.score) list.sorted(animal_weights, key=extract[1]) groupby(students, key=extract.homeroom) groupby(recordtuples, key=extract[0]) Raymond Hettinger P.S. The implementation code is very simple: class ExtractorClass(object): def __getattribute__(self, attr): return operator.attrgetter(attr) def __getitem__(self, key): return operator.itemgetter(key) extract = ExtractorClass() From guido at python.org Wed Dec 3 10:31:33 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 3 10:31:49 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: Your message of "Wed, 03 Dec 2003 10:27:49 EST." <003a01c3b9b2$02c3a7c0$e841fea9@oemcomputer> References: <003a01c3b9b2$02c3a7c0$e841fea9@oemcomputer> Message-ID: <200312031531.hB3FVXQ03914@c-24-5-183-134.client.comcast.net> > I'm am still curious to hear your thoughts on the either sublime or > crazy idea for a generalized extractor object: > > list.sorted(students, key=extract.score) > list.sorted(animal_weights, key=extract[1]) > groupby(students, key=extract.homeroom) > groupby(recordtuples, key=extract[0]) This is the same as the "Voodoo" idea; my comments on that are repeated here: """ It is also somewhat weak in that it only addresses lambdas with one argument, and only allows a single reference to that argument in the resulting expression, and can't really be made to handle method calls without more gross notational hacks -- even though it *can* be made to handle arbitrary binary and unary operators. Yet, it captures 90% of the use cases quite well. I also wonder if the simple trick of requiring to call a "constructor" on each use might not make it more palatable. I.e., instead of writing map(Voodoo.address[0], database) you'd write map(Voodoo().address[0], database) where you can replace Voodoo with a name of your choice, perhaps operator.extract -- although I think this is too different to belong in the operator module. Nick Goghlan showed that a pretty readable brief explanation *can* be written. """ --Guido van Rossum (home page: http://www.python.org/~guido/) From ark-mlist at att.net Wed Dec 3 10:41:43 2003 From: ark-mlist at att.net (Andrew Koenig) Date: Wed Dec 3 10:41:38 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312031417.hB3EHLi03515@c-24-5-183-134.client.comcast.net> Message-ID: <011201c3b9b3$f42b7880$6402a8c0@arkdesktop> > > Isn't the 'long' format the more general representation, with the 'int' > > format then being a performance hack to take advantage of the speed of C > > integer arithmetic? > > > > I'm just wondering if a different way of thinking about it might help > > with figuring out how to handle a combined implementation. > > I proposed that too, but Andrew Koenig didn't like it. It's not me, it's Barbara Liskov at MIT. The general notion is that if Y is derived from X, then you should be able to use a Y wherever you might otherwise want to use an X. In other words, Y should support every operation that X supports, but Y can add operations of its own. From guido at python.org Wed Dec 3 11:10:00 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 3 11:10:14 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: Your message of "Wed, 03 Dec 2003 10:41:43 EST." <011201c3b9b3$f42b7880$6402a8c0@arkdesktop> References: <011201c3b9b3$f42b7880$6402a8c0@arkdesktop> Message-ID: <200312031610.hB3GA1204029@c-24-5-183-134.client.comcast.net> > It's not me, it's Barbara Liskov at MIT. :-) (For non-OO wizards, this is called "Liskov substitutability".) > The general notion is that if Y is derived from X, then you should > be able to use a Y wherever you might otherwise want to use an X. > In other words, Y should support every operation that X supports, > but Y can add operations of its own. So the question is, does long have operations that int doesn't have? And if so, why can't those operations be added to int? And if there's a reason, is it good enough? If the sets of operations are identical, is there a way to break the tie? --Guido van Rossum (home page: http://www.python.org/~guido/) From edcjones at erols.com Wed Dec 3 11:06:48 2003 From: edcjones at erols.com (Edward C. Jones) Date: Wed Dec 3 11:10:31 2003 Subject: [Python-Dev] Python suggestions mostly from C99 Message-ID: <3FCE0A18.7080108@erols.com> Here are a few suggestions for Python: Extend hex() and %x to work on longs, floats, and strings. Add the C99 %a and %A formats. Add a module that wraps the C99 macros and functions that work with special floating point values and floating point exceptions. Since most C compilers are not C99 compliant, some of these ideas might need to be written in C rather than just wrapped. From fincher.8 at osu.edu Wed Dec 3 12:15:17 2003 From: fincher.8 at osu.edu (Jeremy Fincher) Date: Wed Dec 3 11:17:23 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312031610.hB3GA1204029@c-24-5-183-134.client.comcast.net> References: <011201c3b9b3$f42b7880$6402a8c0@arkdesktop> <200312031610.hB3GA1204029@c-24-5-183-134.client.comcast.net> Message-ID: <200312031215.17933.fincher.8@osu.edu> On Wednesday 03 December 2003 11:10 am, Guido van Rossum wrote: > So the question is, does long have operations that int doesn't have? > And if so, why can't those operations be added to int? And if there's > a reason, is it good enough? Taking into account their difference in representation, a long can support 1<<32, but an int can't. I'm not saying that the "proper" (long) behavior can't be unified into a single type, just giving an example of an operation long supports but int doesn't. Jeremy From gball at cfa.harvard.edu Wed Dec 3 11:18:21 2003 From: gball at cfa.harvard.edu (Greg Ball) Date: Wed Dec 3 11:18:25 2003 Subject: [Python-Dev] Re: "groupby" iterator Message-ID: Would it work to generalise generator expressions in the following way: (x[1] for x) == lambda x:x[1] (x.score for x) == lambda x: x.score (x+y for x,y) == lambda x,y: x+y (len(x) for x) == lambda x,len=len: len(x) # roughly equivalent i.e. an anonymous block notation, with early binding semantics. It could work as a lambda replacement. But no-argument lambdas would require some extra ugliness. (1 for) == lambda:1 # yuck! (1 for ()) == lambda:1 # better? And should this be allowed? ((a,b) for a in l for b,l) == lambda b,l: ((a,b) for a in l) -- Greg Ball From tim at zope.com Wed Dec 3 11:26:08 2003 From: tim at zope.com (Tim Peters) Date: Wed Dec 3 11:26:35 2003 Subject: [Python-Dev] test_unicode_file failing on trunk, Win98SE Message-ID: C:\Code\python\PCbuild>python ../lib/test/test_unicode_file.py test_directories (__main__.TestUnicodeFiles) ... ERROR test_equivalent_files (__main__.TestUnicodeFiles) ... ok test_single_files (__main__.TestUnicodeFiles) ... ERROR ====================================================================== ERROR: test_directories (__main__.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_unicode_file.py", line 161, in test_directories os.getcwdu) File "../lib/test/test_unicode_file.py", line 98, in _do_directory os.mkdir(make_name) OSError: [Errno 2] No such file or directory: '@test-?????.dir' Don't know about 2.3 maint, & can't test that today: C:\Code\python\PCbuild>python ../lib/test/test_unicode_file.py test_directories (__main__.TestUnicodeFiles) ... ERROR test_equivalent_files (__main__.TestUnicodeFiles) ... ok test_single_files (__main__.TestUnicodeFiles) ... ERROR *====================================================================== ERROR: test_single_files (__main__.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_unicode_file.py", line 142, in test_single_files self._test_single(TESTFN_UNICODE) File "../lib/test/test_unicode_file.py", line 116, in _test_single self._do_single(filename) File "../lib/test/test_unicode_file.py", line 32, in _do_single os.utime(filename, None) UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-7: ordinal not in range(128) ---------------------------------------------------------------------- Ran 3 tests in 0.170s FAILED (errors=2) Traceback (most recent call last): File "../lib/test/test_unicode_file.py", line 169, in ? test_main() File "../lib/test/test_unicode_file.py", line 166, in test_main run_suite(suite) File "C:\CODE\PYTHON\lib\test\test_support.py", line 267, in run_suite raise TestFailed(msg) test.test_support.TestFailed: errors occurred; run in verbose mode for details C:\Code\python\PCbuild> From pje at telecommunity.com Wed Dec 3 11:30:35 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Dec 3 11:28:37 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: Message-ID: <5.1.0.14.0.20031203112646.03c0a8f0@mail.telecommunity.com> At 11:18 AM 12/3/03 -0500, Greg Ball wrote: >Would it work to generalise generator expressions in the following way: > >(x[1] for x) == lambda x:x[1] >(x.score for x) == lambda x: x.score >(x+y for x,y) == lambda x,y: x+y >(len(x) for x) == lambda x,len=len: len(x) # roughly equivalent -1. It looks like a generator expression, but the semantics are way too different. Unless you intended that it should be invoked with .next() rather than __call__, but then there's no way to pass the arguments. (FWIW, my first reaction was, "Whoa! Cool!", until I thought about how you'd actually have to use it.) From pje at telecommunity.com Wed Dec 3 11:38:51 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Dec 3 11:36:53 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312031215.17933.fincher.8@osu.edu> References: <200312031610.hB3GA1204029@c-24-5-183-134.client.comcast.net> <011201c3b9b3$f42b7880$6402a8c0@arkdesktop> <200312031610.hB3GA1204029@c-24-5-183-134.client.comcast.net> Message-ID: <5.1.0.14.0.20031203113051.03c05e20@mail.telecommunity.com> At 12:15 PM 12/3/03 -0500, Jeremy Fincher wrote: >On Wednesday 03 December 2003 11:10 am, Guido van Rossum wrote: > > So the question is, does long have operations that int doesn't have? > > And if so, why can't those operations be added to int? And if there's > > a reason, is it good enough? > >Taking into account their difference in representation, a long can support >1<<32, but an int can't. Not so. Both '1' and '32' can be represented by int, so only the operation *result* needs to be a long. Further, if the idea here is that 'int' will be a subclass of 'long', then it's perfectly valid to return an int from any operation declared as returning a long. Further, since it's acceptable to *pass* an int for any argument declared 'long', it should suffice to use 'long' for all integer inputs and outputs of methods on 'long'. Anyway, if the idea is that 'long' will be the base class, IMO the name 'long' is confusing. It should probably be called 'integer', with the subclass being either 'int' or 'short'. From pje at telecommunity.com Wed Dec 3 11:44:33 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Dec 3 11:42:36 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <200312031500.hB3F04b03619@c-24-5-183-134.client.comcast.ne t> References: <200312030016.hB30GU205334@oma.cosc.canterbury.ac.nz> Message-ID: <5.1.0.14.0.20031203100859.0247ddd0@mail.telecommunity.com> At 07:00 AM 12/3/03 -0800, Guido van Rossum wrote: >It is also somewhat weak in that it only addresses lambdas with one >argument, and only allows a single reference to that argument in the >resulting expression, and can't really be made to handle method calls >without more gross notational hacks -- even though it *can* be made to >handle arbitrary binary and unary operators. > >Yet, it captures 90% of the use cases quite well. I also wonder if >the simple trick of requiring to call a "constructor" on each use >might not make it more palatable. I.e., instead of writing > > map(Voodoo.address[0], database) > >you'd write > > map(Voodoo().address[0], database) > >where you can replace Voodoo with a name of your choice, perhaps >operator.extract -- although I think this is too different to belong >in the operator module. Nick Goghlan showed that a pretty readable >brief explanation *can* be written. What if it was possible to use, e.g: arg(0).foo + arg('somekeyword').bar That is, have the constructor take an argument position number or kwarg name (perhaps defaulting to 0 for convenience)? Then, you could create expressions that replaced multi-argument lambdas. Then, the only things missing are ways to construct structure expressions like [arg(0).foo,arg(1).bar]. For that, you'd also need an 'args', and to invoke other functions you'd need an argapply. Yuck. I guess that limiting these "argument expressions" to what one might find in an SQL 'select' or 'where' clause (minus nested selects, of course) might not be a bad thing. There might also be a problem with supporting all binary operators... using 'not in' would break, I think, since there isn't a __notcontains__ method. :) You'd have to use e.g. ~(arg(0) in [1,2,9]) instead of 'arg(0) not in [1,2,9]' or 'not arg(0) in [1,2,9]'. Anyway, the implementation of binary operators would also need to check if the 'other' argument was another "argument expression", and if so return a slightly different result than it would for a non-argument (i.e. constant) expression. Interestingly, all this is quite similar to Ian Bicking's SQLObject, which uses expressions of this sort to represent SQL as Python code. From tzot at sil-tec.gr Wed Dec 3 12:04:44 2003 From: tzot at sil-tec.gr (Christos Georgiou) Date: Wed Dec 3 12:04:52 2003 Subject: [Python-Dev] Re: "groupby" iterator Message-ID: <7j5ssvs27cb89qni7eggfo782lo6gdbahj@4ax.com> Guido van Rossum said: >> The nasty thing about it is that it smells a bit too much like a >> clever trick. It's really abusing the syntax to make it mean >> something different from what it usually means. > >It is also somewhat weak in that it only addresses lambdas with one >argument, and only allows a single reference to that argument in the >resulting expression, and can't really be made to handle method calls >without more gross notational hacks -- even though it *can* be made to >handle arbitrary binary and unary operators. A minor correction: the Voodoo approach handles fine method calls by defining a __call__ method in the Voodoo class; it can't handle *function* calls in an elegant way: lambda x: x.startswith("text") This works fine as: Voodoo().startswith("text") lambda x: math.sin(x) This needs gross syntax, the following does not work: math.sin(Voodoo()) -- TZOTZIOY, I speak England very best, Ils sont fous ces Redmontains! --Harddix From guido at python.org Wed Dec 3 12:26:06 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 3 12:26:17 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: Your message of "Wed, 03 Dec 2003 12:15:17 EST." <200312031215.17933.fincher.8@osu.edu> References: <011201c3b9b3$f42b7880$6402a8c0@arkdesktop> <200312031610.hB3GA1204029@c-24-5-183-134.client.comcast.net> <200312031215.17933.fincher.8@osu.edu> Message-ID: <200312031726.hB3HQ6504171@c-24-5-183-134.client.comcast.net> > > So the question is, does long have operations that int doesn't have? > > And if so, why can't those operations be added to int? And if there's > > a reason, is it good enough? > > Taking into account their difference in representation, a long can support > 1<<32, but an int can't. We're talking about a hypothetical int here where that operation returns 4294967296L. (Not so hypothetical, it's implemented in Python 2.4 in CVS.) > I'm not saying that the "proper" (long) behavior can't be unified into a > single type, just giving an example of an operation long supports but int > doesn't. Sorry, your counterexample is rejected. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Wed Dec 3 12:29:46 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Dec 3 12:27:49 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <7j5ssvs27cb89qni7eggfo782lo6gdbahj@4ax.com> Message-ID: <5.1.0.14.0.20031203121737.04321020@mail.telecommunity.com> At 07:04 PM 12/3/03 +0200, Christos Georgiou wrote: >A minor correction: the Voodoo approach handles fine method calls by >defining a __call__ method in the Voodoo class; __call__ is needed for the resulting object to be callable. >it can't handle >*function* calls in an elegant way: > >lambda x: x.startswith("text") >This works fine as: >Voodoo().startswith("text") Voodoo().startswith("text") is equivalent to (lambda x: x.startswith)("text"), which is not the same thing. >lambda x: math.sin(x) >This needs gross syntax, the following does not work: >math.sin(Voodoo()) That too, but also method calls. Basically, once you get past really simple expressions, the whole thing collapses under its own weight. Ironically, it would be easy to do this with generic functions, since one would simply define a methods for the case 'math.sin(Voodoo)' that returns lambda x: math.sin(x). (Hm. This gives me an entirely unrelated idea about how to unify object methods and generic functions in Python using "curried generics". Interesting. And the Voodoo() aka 'arg()' concept could be applied to predicate dispatching as well.) From edloper at gradient.cis.upenn.edu Wed Dec 3 13:30:25 2003 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Wed Dec 3 12:28:53 2003 Subject: [Python-Dev] Tutorial: Brief Introduction to the Standard Libary In-Reply-To: References: Message-ID: <3FCE2BC1.10304@gradient.cis.upenn.edu> > Attached is a revised draft. Comments, suggestions, nitpicks, > complaints, and accolades are welcome. General comments: - Overall, it looks very good. :) - Since your descriptions are so brief, you should probably provide links to the reference documentation for each module/function that you reference. - You might also want to mention pydoc. Not only is it a very useful module for beginners in its own right, but it would give them a way to learn more about the other modules that you describe. > >>>import os > >>>os.system('copy /data/mydata.fil /backup/mydata.fil') > 0 > > >>>os.getcwd() > 'C:\\Python24' > > >>>os.chdir('/server/accesslogs') You might want to add short "#" comments for these, saying what they do. (Some novices won't recognize what system() is doing; and not everyone immediately associates cwd with current working directory.) Nothing fancy, just something like: >>> os.getcwd() # Return the current working directory > Be sure to use the "import os" style instead of "from os import *". > This will keep os.open() from shadowing __builtin__.open() which > operates much differently. This sentence would be more accessible to beginners if you said "from shadowing the builtin open() function, which operates much differently." >>>>optlist, args = getopt.getopt(sys.argv[1:], 'abc:d:') It might be worth including an option that doesn't get used, just to show that they're optional. > because they are easier to read and debug. For more sophisticated applications, I would replace s/For/But for/ > >>>import urllib That should be "import urllib2" > For example, it may be tempting to use the tuple packing and unpacking feature > instead of the traditional approach to swapping arguments. The timeit module > quickly demonstrates that the traditional approach is faster: I think that you should add a caveat here. Something along the lines of "But the tuple method is much easier to read, and not much faster, so unless it's in a really tight loop where speed is critical, you should use the tuple method." > In contrast to timeit's fine level of granularity, the profile and pstats modules > provide tools for identifying time critical sections of larger blocks of code. I think it's worth noting that profile & pstats are generally much more useful for optimizing code than timeit is. (That fact might not be obvious to beginners.) > Batteries Included > > Python has a "batteries included" philosophy. The is best seen through the > sophisticated and robust capabilites of its larger packages: s/packages:/packages, including:/ Also, I'm not sure if the meaning of "batteries included" will be obvious to people who haven't seen the term before. Perhaps you could steal some text from PEP 206; so something like: Python has a "batteries included" philosophy -- it includes a rich and versatile standard library which is immediately available, without making the user download separate packages. The is best seen through the sophisticated and robust capabilites of its larger packages, including: -Edward From mfb at lotusland.dyndns.org Wed Dec 3 12:34:33 2003 From: mfb at lotusland.dyndns.org (Matthew F. Barnes) Date: Wed Dec 3 12:34:50 2003 Subject: [Python-Dev] Python suggestions mostly from C99 In-Reply-To: <3FCE0A18.7080108@erols.com> References: <3FCE0A18.7080108@erols.com> Message-ID: <10469.130.76.64.14.1070472873.squirrel@server.lotusland.dyndns.org> Edward C. Jones said: > Extend hex() and %x to work on longs, floats, and strings. Just FYI - Extending hex() (and oct()) to work with strings has been discussed in the past: Matthew Barnes From guido at python.org Wed Dec 3 12:48:05 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 3 12:48:21 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: Your message of "Wed, 03 Dec 2003 19:04:44 +0200." <7j5ssvs27cb89qni7eggfo782lo6gdbahj@4ax.com> References: <7j5ssvs27cb89qni7eggfo782lo6gdbahj@4ax.com> Message-ID: <200312031748.hB3Hm5504233@c-24-5-183-134.client.comcast.net> > >It is also somewhat weak in that it only addresses lambdas with one > >argument, and only allows a single reference to that argument in the > >resulting expression, and can't really be made to handle method calls > >without more gross notational hacks -- even though it *can* be made to > >handle arbitrary binary and unary operators. > > A minor correction: the Voodoo approach handles fine method calls by > defining a __call__ method in the Voodoo class; [...] > > lambda x: x.startswith("text") > This works fine as: > Voodoo().startswith("text") Hm. But the whole point of Voodoo is that the resulting object has a __call__ method that evaluates the expression for a given argument. How do you do that? (I suppose you can do it if you don't do chaining, but I like the chaining.) --Guido van Rossum (home page: http://www.python.org/~guido/) From tzot at sil-tec.gr Wed Dec 3 13:28:47 2003 From: tzot at sil-tec.gr (Christos Georgiou) Date: Wed Dec 3 13:28:52 2003 Subject: [Python-Dev] Re: "groupby" iterator Message-ID: Guido van Rossum said: [Christos] >> A minor correction: the Voodoo approach handles fine method calls by >> defining a __call__ method in the Voodoo class; [...] >> >> lambda x: x.startswith("text") >> This works fine as: >> Voodoo().startswith("text") > >Hm. But the whole point of Voodoo is that the resulting object has a >__call__ method that evaluates the expression for a given argument. >How do you do that? (I suppose you can do it if you don't do >chaining, but I like the chaining.) (replying to Phillip Eby too) Yes, you are correct (I was thinking in terms of my All and Any classes, which don't need to be used as callables). There is a solution involving a Voodoo and an (unexported) VoodooAttribute class which *can* solve this (Voodoo.__getattr__ returns an instance of VoodooAttribute (subclass of Voodoo overloading __call__), whose .__call__ method returns an instance of Voodoo), *but* then you can't use: Voodoo().startswith # this is a VoodooAttribute instance as an argument to groupby, unless you use: Voodoo(Voodoo().startswith) # this is a Voodoo instance I can't think at the moment a case in the groupby context where the latter would be needed (a VoodooAttribute behaves the same for attributes that are not methods), but I'm sure many others will, and even if they don't, it sure smells 'breakage'. Unless the Voodoo class does what it does best (and I'll rename it to Argument for clarity below), and an instance of it is the only argument to the *explicit* initialisation of the FuncMaker class (the final Voodoo :): lambda x: x.startswith("text") equals FuncMaker(Argument().startswith("text")) Too unpythonic perhaps? PS attrgetter and itemgetter sound fine btw, I'm just trying to help this brainstorming for more general solutions --if any. -- TZOTZIOY, I speak England very best, Ils sont fous ces Redmontains! --Harddix From pf_moore at yahoo.co.uk Wed Dec 3 13:31:52 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Wed Dec 3 13:31:41 2003 Subject: [Python-Dev] Re: "groupby" iterator References: <200312030016.hB30GU205334@oma.cosc.canterbury.ac.nz> <200312031500.hB3F04b03619@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: > [Greg Ewing] >> We seem to be talking about two different things in this thread, >> speed and readability. The nice thing about the "attrgetter.x" >> etc. idea is that it has the potential to provide both at once. Yes. Frankly, readability (anybody's flavour) is easy to handle - all of the various suggestions can be implemented in user code, without any real problem. While a standard library solution may be useful for consistency across applications, it'll only get consistently used if it gives some other significant benefit (ie, speed). So speed is the crucial point here, with readability a poor second. But speed is very rarely *really* crucial, no matter what people might think. So we need something that is fast and nice-looking, to avoid the "attractive nuisance" of something slower, but more "obvious" or "readable" (ie, lambda). Personally, I find the name operator.attrgetter (with semantics as currently implemented) ugly enough that I'll probably never use it (or only rarely, where speed is crucial). It's definitely not "the one obvious way" for me. But naming is highly personal, so that's not a good argument by itself. >> The nasty thing about it is that it smells a bit too much like a >> clever trick. It's really abusing the syntax to make it mean >> something different from what it usually means. While it still uses the name "attrgetter", I agree. However, with the name "extract", I like the look of it enough that I'd overlook the syntax abuse. But Guido's suggestion of requiring a constructor call would swing me back the other way again (actually, I'd cheat and do extract = extract() once and be done with it, but I'd feel unclean and avoid the construct because of this...) All personal opinion again, of course. > It is also somewhat weak in that it only addresses lambdas with one > argument, and only allows a single reference to that argument in the > resulting expression, and can't really be made to handle method calls > without more gross notational hacks -- even though it *can* be made to > handle arbitrary binary and unary operators. > > Yet, it captures 90% of the use cases quite well. I think this is the key thing here. Lambda in its full form is very flexible and wide ranging - but 90% of the time, this flexibility isn't needed. And everything pays a cost for that flexibility, even when it's not used. Something that did that 90%, fast and efficiently, while leaving lambda for the few remaining cases where its flexibility is needed (possibly with an intent to deprecate it in the future) would probably be the best option. Paul. -- This signature intentionally left blank From pf_moore at yahoo.co.uk Wed Dec 3 13:33:35 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Wed Dec 3 13:40:52 2003 Subject: [Python-Dev] Re: "groupby" iterator References: <200311282146.hASLkr317367@c-24-5-183-134.client.comcast.net> <200311302157.49205.aleaxit@yahoo.com> <8ylvmcmq.fsf@yahoo.co.uk> <200312021654.34386.fincher.8@osu.edu> Message-ID: Jeremy Fincher writes: > On Tuesday 02 December 2003 11:02 am, Paul Moore wrote: >> Alex Martelli writes: >> > Since these functions or types are going to be in operator, I think >> > we can afford to "spend" two names to distinguish functionality >> > (even though attgetter and itemgetter look nowhere as neat as >> > extract -- I don't have better suggestions offhand). >> >> operator.getattr and operator.getitem? > > I think the functions they're talking about are actually right-curried > versions of those functions, like this: I know - I was hoping that the fact that these particular versions were in the operator module might justify using the same name for two mildly-related things. But it's probably more confusing than helpful. Paul -- This signature intentionally left blank From eppstein at ics.uci.edu Wed Dec 3 13:50:03 2003 From: eppstein at ics.uci.edu (David Eppstein) Date: Wed Dec 3 13:49:58 2003 Subject: [Python-Dev] Re: "groupby" iterator References: <7j5ssvs27cb89qni7eggfo782lo6gdbahj@4ax.com> <200312031748.hB3Hm5504233@c-24-5-183-134.client.comcast.net> Message-ID: In article <200312031748.hB3Hm5504233@c-24-5-183-134.client.comcast.net>, Guido van Rossum wrote: > > A minor correction: the Voodoo approach handles fine method calls by > > defining a __call__ method in the Voodoo class; [...] > > > > lambda x: x.startswith("text") > > This works fine as: > > Voodoo().startswith("text") > > Hm. But the whole point of Voodoo is that the resulting object has a > __call__ method that evaluates the expression for a given argument. > How do you do that? (I suppose you can do it if you don't do > chaining, but I like the chaining.) I think you would need two kinds of voodoo. extract.attr could expand to lambda x: x.attr (or something similar via __call__ methods) but method.startswith could expand to lambda *args: lambda x: x.startswith(*args) Then you could do sort(stringlist, key=method.count("kibo")) or whatever. Obviously this doesn't allow you to easily express e.g. methods of attributes, but beyond a certain level of complexity you should be using separate defs anyway. -- David Eppstein http://www.ics.uci.edu/~eppstein/ Univ. of California, Irvine, School of Information & Computer Science From pedronis at bluewin.ch Wed Dec 3 14:14:57 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Wed Dec 3 14:11:52 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <200312031500.hB3F04b03619@c-24-5-183-134.client.comcast.ne t> References: <200312030016.hB30GU205334@oma.cosc.canterbury.ac.nz> Message-ID: <5.2.1.1.0.20031203200451.02860ed0@pop.bluewin.ch> At 07:00 03.12.2003 -0800, Guido van Rossum wrote: > > The old lambda would have to be kept around for a while for programs > > relying on the old semantics, but it could be deprecated, and > > removed in 3.0. > >I'm not sure that the -> notation is more understandable than lambda; >it would surely confuse C/C++ programmers who are new to Python. > >Scary thought: how about simply introducing early-binding semantics >for lambda in 3.0? that would confuse schemers >Another radical idea would be to use an anonymous-block notation like >Smalltalk and Ruby. We could use some kind of funky brackets like >[|...|]. A lambda would require an argument notation too. I believe >Ruby uses [|x| x+1] where we would write lambda x: x+1, maybe we could >use [|x: x+1|]. (I like structure with an explicit close more than >open ones like lambda.) I would expect blocks to be blocks, accepting also statements and sharing the scope with the surrounding code, not having early-binding semantics. Honestly, given the introduction of generator exprs, a substitute expression for lambda with early-binding semantics makes sense. Personally I can see it also as meaningful wrt the statemenent/expression dichotomy in python, and we would have early-binding as a general rule for "expressions". But again I think that a block-like syntax should be used, if at all, for real blocks. regards. From guido at python.org Wed Dec 3 14:14:40 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 3 14:14:52 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: Your message of "Wed, 03 Dec 2003 18:31:52 GMT." References: <200312030016.hB30GU205334@oma.cosc.canterbury.ac.nz> <200312031500.hB3F04b03619@c-24-5-183-134.client.comcast.net> Message-ID: <200312031914.hB3JEeW04395@c-24-5-183-134.client.comcast.net> > I think this is the key thing here. Lambda in its full form is very > flexible and wide ranging - but 90% of the time, this flexibility > isn't needed. And everything pays a cost for that flexibility, even > when it's not used. > > Something that did that 90%, fast and efficiently, while leaving > lambda for the few remaining cases where its flexibility is needed > (possibly with an intent to deprecate it in the future) would probably > be the best option. For the remaining 10%, you could just use 'def' if lambda didn't exist at all. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Dec 3 14:19:52 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 3 14:20:05 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: Your message of "Wed, 03 Dec 2003 20:14:57 +0100." <5.2.1.1.0.20031203200451.02860ed0@pop.bluewin.ch> References: <200312030016.hB30GU205334@oma.cosc.canterbury.ac.nz> <5.2.1.1.0.20031203200451.02860ed0@pop.bluewin.ch> Message-ID: <200312031919.hB3JJqg04439@c-24-5-183-134.client.comcast.net> [Samuele] > I would expect blocks to be blocks, accepting also statements and > sharing the scope with the surrounding code, not having > early-binding semantics. > > Honestly, given the introduction of generator exprs, a substitute > expression for lambda with early-binding semantics makes > sense. Personally I can see it also as meaningful wrt the > statemenent/expression dichotomy in python, and we would have > early-binding as a general rule for "expressions". > > But again I think that a block-like syntax should be used, if at > all, for real blocks. Thanks -- that all makes sense. --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Wed Dec 3 14:27:25 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Dec 3 14:25:29 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <200312031914.hB3JEeW04395@c-24-5-183-134.client.comcast.ne t> References: <200312030016.hB30GU205334@oma.cosc.canterbury.ac.nz> <200312031500.hB3F04b03619@c-24-5-183-134.client.comcast.net> Message-ID: <5.1.0.14.0.20031203142518.030a3660@mail.telecommunity.com> At 11:14 AM 12/3/03 -0800, Guido van Rossum wrote: > > I think this is the key thing here. Lambda in its full form is very > > flexible and wide ranging - but 90% of the time, this flexibility > > isn't needed. And everything pays a cost for that flexibility, even > > when it's not used. > > > > Something that did that 90%, fast and efficiently, while leaving > > lambda for the few remaining cases where its flexibility is needed > > (possibly with an intent to deprecate it in the future) would probably > > be the best option. > >For the remaining 10%, you could just use 'def' if lambda didn't exist >at all. Some use cases could get quite inconvenient unless function decorators were also available, but if we're talking about 3.0, then I suppose they would be. :) From walter at livinglogic.de Wed Dec 3 14:37:11 2003 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed Dec 3 14:37:16 2003 Subject: [Python-Dev] pprint and list/tuple/dict subclasses In-Reply-To: <2mu14ixjb0.fsf@starship.python.net> References: <3FCCDCFA.6070800@livinglogic.de> <16332.56789.356676.91289@grendel.fdrake.net> <3FCCF2F7.7090805@livinglogic.de> <16332.62351.960759.135367@grendel.fdrake.net> <3FCD1A48.5000202@livinglogic.de> <2mu14ixjb0.fsf@starship.python.net> Message-ID: <3FCE3B67.604@livinglogic.de> Michael Hudson wrote: > Walter D?rwald writes: > >>The arguments for the PrettyPrinter constructor could >>be exposed in pprint() and pformat(): >>""" >>import pprint >>print.pprint(foo, indent=3, width=40) >>""" > > +1e6! I've wanted this for so long I forgot that I could do something > +about it :-) OK, then I'll go ahead and implement that. Bye, Walter D?rwald From walter at livinglogic.de Wed Dec 3 15:27:46 2003 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed Dec 3 15:27:51 2003 Subject: [Python-Dev] pprint and list/tuple/dict subclasses In-Reply-To: <2mu14ixjb0.fsf@starship.python.net> References: <3FCCDCFA.6070800@livinglogic.de> <16332.56789.356676.91289@grendel.fdrake.net> <3FCCF2F7.7090805@livinglogic.de> <16332.62351.960759.135367@grendel.fdrake.net> <3FCD1A48.5000202@livinglogic.de> <2mu14ixjb0.fsf@starship.python.net> Message-ID: <3FCE4742.9040104@livinglogic.de> Michael Hudson wrote: > Walter D?rwald writes: > > >>The arguments for the PrettyPrinter constructor could >>be exposed in pprint() and pformat(): >>""" >>import pprint >>print.pprint(foo, indent=3, width=40) >>""" > > > +1e6! I've wanted this for so long I forgot that I could do something > +about it :-) Done. (And patch #750542 too) Bye, Walter D?rwald From ark-mlist at att.net Wed Dec 3 15:33:23 2003 From: ark-mlist at att.net (Andrew Koenig) Date: Wed Dec 3 15:33:18 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312031610.hB3GA1204029@c-24-5-183-134.client.comcast.net> Message-ID: <003f01c3b9dc$b2bfe510$6402a8c0@arkdesktop> > So the question is, does long have operations that int doesn't have? > And if so, why can't those operations be added to int? And if there's > a reason, is it good enough? If every operation that might conceivably yield an int quietly yields a long when the result doesn't fit in an int, then it is possible for int and long to have the same operations. In which case, why do they have different types? Just as an implementation detail? From guido at python.org Wed Dec 3 15:42:29 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 3 15:42:40 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: Your message of "Wed, 03 Dec 2003 15:33:23 EST." <003f01c3b9dc$b2bfe510$6402a8c0@arkdesktop> References: <003f01c3b9dc$b2bfe510$6402a8c0@arkdesktop> Message-ID: <200312032042.hB3KgTu04629@c-24-5-183-134.client.comcast.net> > > So the question is, does long have operations that int doesn't > > have? And if so, why can't those operations be added to int? And > > if there's a reason, is it good enough? > > If every operation that might conceivably yield an int quietly > yields a long when the result doesn't fit in an int, then it is > possible for int and long to have the same operations. > > In which case, why do they have different types? Just as an > implementation detail? Right. The implementation uses the type as the vtable of a C++ class, and the functions in the vtable "know" the representation, which is very different for short and long ints. This is all nice and efficient, except that the identity of the vtable is returned as the type of the object, so two different vtables means two different types. The alternative implementation technique is to have a single type/vtable, embed a flag in each instance that tells the implementation which representation is used, and have an "if" at the top of each implementation function switching between the two. The downslide should be obvious: not only an extra test+branch for each operation, but also extra space in the object (the short representation has no spare bits). --Guido van Rossum (home page: http://www.python.org/~guido/) From mhammond at skippinet.com.au Wed Dec 3 17:22:12 2003 From: mhammond at skippinet.com.au (Mark Hammond) Date: Wed Dec 3 17:22:28 2003 Subject: [Python-Dev] RE: test_unicode_file failing on trunk, Win98SE In-Reply-To: Message-ID: <045401c3b9eb$e6d70450$2c00a8c0@eden> > C:\Code\python\PCbuild>python ../lib/test/test_unicode_file.py > test_directories (__main__.TestUnicodeFiles) ... ERROR > test_equivalent_files (__main__.TestUnicodeFiles) ... ok > test_single_files (__main__.TestUnicodeFiles) ... ERROR My apologies - I have fixed it (I hope! No Win98 SE here for me to test on) Note the error was in the test suite semantics only, not the core code. > OSError: [Errno 2] No such file or directory: '@test-?????.dir' Interestingly, this is yet another bug. The error you *should* have got is a UnicodeDecodeError. However, the Windows 'mbcs' encoding currently acts as if in errors=replace mode. See 850997. That is really neither here nor there tho - all it does is change the error you get when passing a Unicode filename. > Don't know about 2.3 maint, & can't test that today: The fix to posixmodule was checked into both head and 2.3-maint, but the test suite changes were only made to head - for exactly this reason . Thus, 2.3 should be fine. Please let me know if there are still errors. Thanks, Mark. From greg at cosc.canterbury.ac.nz Wed Dec 3 18:49:38 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 3 18:49:46 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <87y8turaep.fsf@egil.codesourcery.com> Message-ID: <200312032349.hB3Nncq13330@oma.cosc.canterbury.ac.nz> Zack Weinberg : > If you're going to mess with lambda at all, please try to come up with > a way for it not to be limited to a single expression. That's not the direction I want to go in. Lambdas are best for extremely small functions. For anything more complicated, I believe you're better off using a def. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From andrew-pythondev at puzzling.org Wed Dec 3 19:23:48 2003 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Wed Dec 3 19:23:55 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: References: Message-ID: <20031204002348.GA28006@frobozz> On Wed, Dec 03, 2003 at 08:28:47PM +0200, Christos Georgiou wrote: > > There is a solution involving a Voodoo and an (unexported) > VoodooAttribute class which *can* solve this (Voodoo.__getattr__ returns > an instance of VoodooAttribute (subclass of Voodoo overloading > __call__), whose .__call__ method returns an instance of Voodoo), *but* > then you can't use: [...] This discussion is starting to remind me of something that's been in the Twisted sandbox for quite a while: http://cvs.twistedmatrix.com/cvs/sandbox/cake.py?rev=1.4&content-type=text/vnd.viewcvs-markup The code isn't exactly clear, but the example in the if __name__ == '__main__' block at the bottom should give you some idea of what it can do... I can't escape the feeling that this discussion is slowly reinventing cake.py. I'm not sure I'd really like to read code that uses some of the more outlandish proposals in this thread... it looks like a good way to write convoluted code. -Andrew. From mike.thompson at day8.com.au Wed Dec 3 20:46:36 2003 From: mike.thompson at day8.com.au (Mike Thompson) Date: Wed Dec 3 21:10:37 2003 Subject: [Python-Dev] Re: Int FutureWarnings and other 2.4 TODOs References: <011201c3b9b3$f42b7880$6402a8c0@arkdesktop> <200312031610.hB3GA1204029@c-24-5-183-134.client.comcast.net> Message-ID: "Guido van Rossum" wrote in message news:200312031610.hB3GA1204029@c-24-5-183-134.client.comcast.net... > > It's not me, it's Barbara Liskov at MIT. > > :-) > > (For non-OO wizards, this is called "Liskov substitutability".) > > > The general notion is that if Y is derived from X, then you should > > be able to use a Y wherever you might otherwise want to use an X. > > In other words, Y should support every operation that X supports, > > but Y can add operations of its own. > > So the question is, does long have operations that int doesn't have? > And if so, why can't those operations be added to int? And if there's > a reason, is it good enough? > > If the sets of operations are identical, is there a way to break the > tie? > In this case, one could argue that no tie breaker is needed. If one accepts that int and long have the same operations but offer different implementations, then the obvious solution is: an abstract base class (Int) and two sub-classes (Short and Long?). Of course, this obvious solution has a problem: a programmer can't easily sub-class Int. They must choose one a sub-class from Short or Long (unless they want to provide a full implementation themselves) and that's probably not a decision they want to make. Perhaps, because of this, the GOF "Bridge Pattern" might be suitable here. (This pattern can be framed as: "adapt multiple implementations to an interface using delegation" -- which, after all, is pretty much what the vtable in previous solution gives you.) If the existence of Short and Long is an implementation detail best hidden from the python programmer, then a Bridge Pattern implementation has a lot going for it. Use of the Bridge pattern might even allow for three different implementations: PreDefined, Short, Long ... where PreDefined is one of the numbers between 0 and 256 which I'm given to understand are pre-allocated by the runtime environment. -- Mike From greg at cosc.canterbury.ac.nz Wed Dec 3 21:36:54 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 3 21:37:04 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <5.1.0.14.0.20031203121737.04321020@mail.telecommunity.com> Message-ID: <200312040236.hB42asw13987@oma.cosc.canterbury.ac.nz> "Phillip J. Eby" : > Ironically, it would be easy to do this with generic functions, since one > would simply define a methods for the case 'math.sin(Voodoo)' that returns > lambda x: math.sin(x). > (Hm. This gives me an entirely unrelated idea about how to unify object > methods and generic functions in Python using "curried > generics". Interesting. And the Voodoo() aka 'arg()' concept could be > applied to predicate dispatching as well.) If we go much further along this road, we're going to end up re-implementing the whole Python interpreter using threaded code! Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From ajsiegel at optonline.net Wed Dec 3 14:08:13 2003 From: ajsiegel at optonline.net (Arthur) Date: Wed Dec 3 21:45:16 2003 Subject: [Python-Dev] re: Tutorial: Brief Introduction to the Standard Libary Message-ID: <0HPC003JRNNFLF@mta7.srv.hcvlny.cv.net> Michael writes - >I don't want to be rude -- is English your first language? -- but I >found this article to be written in particularly baroque prose. English first and only. Same with Python. But if I have your demographics correct, I am of a different era. Baroque? I prefer Dylan, actually, >I'm not quite sure what your central point is. I believe you. >Well, I dunno. If it was my decision, there's no way it would be >mentioned in the built-ins Or in the tutorial accompanying Python? Which is what is under discussion. >No changes here in a long while. I believe you again. >I don't think it's unreasonable at all. To me the copy module is >something you only need when you really need it, then you go looking >for it. If you understand a) when you need it b) it exists. Tutorial discussion. In the negative, if you like. It exists, it does this, and if you think you need it in a particular case are probably wrong, because...would serve my purpose. Perfectly. >Certainly, the failure to understand assignment and erroneous beliefs >that the copy module is necessary in a given situation go hand in hand. I don't think that view is inconsistent with anything I am asserting. >Assignment in Python *isn't* particularly subtle, though. My >impression is that it's people from e.g. C who get tripped up here, >not total newbies. That assertion is, IMO, an overly convenient way to put the matter to bed. It is an assertion with which I disagree. And I don't know who can claim particular insight or expertise on this kind of issue, and on what basis. Which is a problem. I can promise you that a background slanted by C was not at work in my case. It would be nice if it were. >Another reason might just be that copy() is implemented in Python... I prefer those kinds of reasons, than the newbies ones. Sound weighty and beyond my ken. The newbies ones don't. Sorry. >Have you ever used the copy module? I am *not* a beginner, and have >used it *once* (and I can't remember what for, either). In my view, there are usually more convenient ways to accomplish what copy is about, because you don't have to import copy. And we are just in a circular argument. And its use is discouraged, as a matter of style. More circles. But isn't it a harmless alternative, and a bit more explicit, in a good deal of circumstances - arguably. But I truly do understand that I may be missing something fundamental here. Performance issues? Who knows what else. I'm afraid to find out, actually. [OT] """ EuroPython: What do you like best about Python? MH: I think, probably, the single biggest plus of Python is that it blurs the distinction between using and programming a computer. """ One of my favorite Python quotes of recent vintage, BTW. Art From greg at cosc.canterbury.ac.nz Wed Dec 3 21:54:21 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 3 21:54:27 2003 Subject: [Python-Dev] Re: Int FutureWarnings and other 2.4 TODOs In-Reply-To: Message-ID: <200312040254.hB42sLI14069@oma.cosc.canterbury.ac.nz> Mike Thompson : > Of course, this obvious solution has a problem: a programmer can't easily > sub-class Int. I can't see that subclassing int is a particularly useful thing to do, even as matters stand today. As soon as you do any operation on your int subclass, you get a result which is not an instance of your subclass any more, which makes using it rather fragile. Does anyone have a real-life use case for subclassing int? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Wed Dec 3 22:16:25 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 3 22:16:30 2003 Subject: [Python-Dev] re: Tutorial: Brief Introduction to the Standard Libary In-Reply-To: <0HPC003JRNNFLF@mta7.srv.hcvlny.cv.net> Message-ID: <200312040316.hB43GPM14271@oma.cosc.canterbury.ac.nz> Arthur : > In my view, there are usually more convenient ways to accomplish what copy > is about, because you don't have to import copy. And we are just in a > circular argument. And its use is discouraged, as a matter of style. More > circles. But isn't it a harmless alternative, and a bit more explicit, in a > good deal of circumstances - arguably. There isn't any "good deal of circumstances" in the first place. I'm not sure why it is, but in my experience, and in other people's too, it seems, it's very rare that you ever *need* to copy anything in Python. There's nothing circular here -- it would still be true even if copy() were a built-in. And on the rare occasions where you *do* need to copy something, you need to know a fair bit about the kind of thing you're copying, and for what purpose you're copying it, so that you know exactly how much of it to copy. It's a delicate operation that I'd be reluctant to trust to any general-purpose copying function. The result of all this is that I don't think I've *ever* used copy() or deepcopy(). Whenever I've need to copy something I've always used something type-specific such as list[:] or dict.copy(), or written my own copying function. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Wed Dec 3 22:22:49 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 3 22:22:59 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: Message-ID: <200312040322.hB43Mn814298@oma.cosc.canterbury.ac.nz> Greg Ball : > (x[1] for x) == lambda x:x[1] > (x.score for x) == lambda x: x.score > (x+y for x,y) == lambda x,y: x+y > (len(x) for x) == lambda x,len=len: len(x) # roughly equivalent Interesting idea, but it reads a bit strangely. Maybe this would work better with a different keyword... x[1] given x x.score given x x+y given x,y len(x) given x stuff = groupby((x for x in seq), key = (x.score given x)) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From anthony at ekit-inc.com Wed Dec 3 22:42:05 2003 From: anthony at ekit-inc.com (Anthony Baxter) Date: Wed Dec 3 22:42:28 2003 Subject: [Python-Dev] minor interruption to service. In-Reply-To: Message from Skip Montanaro of "Tue, 02 Dec 2003 14:13:34 MDT." <16332.62062.305234.903783@montanaro.dyndns.org> Message-ID: <200312040342.hB43g6Co009702@maxim.off.ekorp.com> >>> Skip Montanaro wrote > Anthony> In the meantime, if someone wants to take on the "upgrade to > Anthony> autoconf 2.59" task, I'd appreciate it very much. > > Do you mean 2.58? I looked here: > > http://ftp.gnu.org/gnu/autoconf/ > > and 2.58 appears to be the latest release. There's a 2.59 release announced on the autoconf list http://mail.gnu.org/archive/html/autoconf-maintainers/2003-11/msg00001.html but it's never appeared on the GNU ftp servers. Based on this bit of uncertainty, as well as a sheer lack of round tuits on my part, I've held off on the autoconf upgrade. 2.3.4 can have it. Anthony From ajsiegel at optonline.net Wed Dec 3 23:05:55 2003 From: ajsiegel at optonline.net (Arthur) Date: Wed Dec 3 23:02:11 2003 Subject: [Python-Dev] re: Tutorial: Brief Introduction to the Standard Libary Message-ID: <0HPC00A6FR7INO@mta2.srv.hcvlny.cv.net> Greg writes: >The result of all this is that I don't think I've *ever* used >copy() or deepcopy(). Whenever I've need to copy something I've >always used something type-specific such as list[:] or dict.copy(), >or written my own copying function. What is fundamentally wrong with copy(list) as an alternative to list[:]? Guess I am feeling invincible. Worst case, I can be told that I need to read a tutorial ;), Art From alexis.layton at post.harvard.edu Wed Dec 3 23:05:41 2003 From: alexis.layton at post.harvard.edu (Alexis Layton) Date: Wed Dec 3 23:05:44 2003 Subject: [Python-Dev] Re: "groupby" iterator Message-ID: <20199DBD-260F-11D8-A36A-000A957D895E@post.harvard.edu> Greetings illustrious Python developers! I've been reading the python-dev archives for years now, but the current discussion of lambda alternatives has finally got me to post. It seems there is a tradeoff between more general (fully functional, if you will, pardon the groans) lambda-type constructs, and the simpler "90%" constructs that have been being batted around here. One simple style is to "infer" the lambda using positional arguments, for example: (&1 + &2.foo()) (this would use "&" as a unary operator which I believe is available?) Or one could use "$", which begins to look a little like shell programming: ($1['fred'].x) (Or \ if you want to risk being Haskelly....) This style of construct implies the lambda or function without really saying it, which may be too magical for Python. The other main problem with constructs like this are that some number of current uses of lambda need to ignore arguments; this happens particularly for GUI callbacks, I think. Also, the syntax doesn't really support no-argument lambdas. On the other side is the spelled-out form. Perhaps def can really be re-used: def(arglist: expr) as in def(a, b: a + b.foo()) #equivalent to first example def(a: a['fred'].x) #equivalent to second example of course, this is very close to lambda in syntax, although it does have the advantage of a close paren; it can clearly handle all forms of arguments (keyword, default values (although why would you need...), *, **, etc.) The hitch there is that the syntax looks different from regular def. That may be ok, considering the beasts are different.... I'd personally hate to see something equivalent to lambda completely disappear from the language. ---- Alexis Layton alexis dot layton at post dot harvard dot edu From tdelaney at avaya.com Wed Dec 3 23:32:25 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Wed Dec 3 23:32:31 2003 Subject: [Python-Dev] re: Tutorial: Brief Introduction to the Standard Libary Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEF5FC06@au3010avexu1.global.avaya.com> > From: Arthur > > What is fundamentally wrong with copy(list) as an alternative > to list[:]? Well, it's slower ... it involves a Python function call and python code, whereas list[:] is all C. Of course, I didn't say it's a *lot* slower ... the actual copying is likely to take longer. However, I have to agree that I've almost never used copy or deepcopy. Tim Delaney From guido at python.org Wed Dec 3 23:50:04 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 3 23:50:16 2003 Subject: [Python-Dev] re: Tutorial: Brief Introduction to the Standard Libary In-Reply-To: Your message of "Thu, 04 Dec 2003 15:32:25 +1100." <338366A6D2E2CA4C9DAEAE652E12A1DEF5FC06@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DEF5FC06@au3010avexu1.global.avaya.com> Message-ID: <200312040450.hB44o4x05333@c-24-5-183-134.client.comcast.net> Can we stop this thread on copy()? It's one of Arthur's pet subjects and you're not going to convince him, so it's pointless to argue. Arthur, why don't you blog on this so the world at large can read your views, rather than boring the developers here? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Dec 4 00:05:18 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 4 00:05:34 2003 Subject: [Python-Dev] Re: Int FutureWarnings and other 2.4 TODOs In-Reply-To: Your message of "Thu, 04 Dec 2003 12:46:36 +1100." References: <011201c3b9b3$f42b7880$6402a8c0@arkdesktop> <200312031610.hB3GA1204029@c-24-5-183-134.client.comcast.net> Message-ID: <200312040505.hB455IJ05377@c-24-5-183-134.client.comcast.net> > If one accepts that int and long have the same operations but offer > different implementations, then the obvious solution is: an abstract > base class (Int) and two sub-classes (Short and Long?). Right, this seems to be one of the preferred solutions. > Of course, this obvious solution has a problem: a programmer can't > easily sub-class Int. They must choose one a sub-class from Short > or Long (unless they want to provide a full implementation > themselves) and that's probably not a decision they want to make. (It has been pointed out that subclassing int isn't very useful, so maybe this is a moot point. Does anybody have a real use case?) > Perhaps, because of this, the GOF "Bridge Pattern" might be suitable > here. (This pattern can be framed as: "adapt multiple > implementations to an interface using delegation" -- which, after > all, is pretty much what the vtable in previous solution gives you.) > > If the existence of Short and Long is an implementation detail best > hidden from the python programmer, then a Bridge Pattern > implementation has a lot going for it. Hmm... I'm not very familiar with the Bridge pattern (and my GoF book is on one of 65 boxes still in the garage waiting until we move into a larger house :-). Can you give a little more detail about how this would be done? Is it something that a user wishing to subclass Int should do, or something that the Int class (or its subclasses) should provide? > Use of the Bridge pattern might even allow for three different > implementations: PreDefined, Short, Long ... where PreDefined is one > of the numbers between 0 and 256 which I'm given to understand are > pre-allocated by the runtime environment. I don't think so -- the beauty is that (most of) the implementation doesn't know that certain int values have eternal life; their representation is exactly the same as that of regular ints. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Thu Dec 4 00:26:07 2003 From: tim.one at comcast.net (Tim Peters) Date: Thu Dec 4 00:26:11 2003 Subject: [Python-Dev] RE: test_unicode_file failing on trunk, Win98SE In-Reply-To: <045401c3b9eb$e6d70450$2c00a8c0@eden> Message-ID: [Mark Hammond] > My apologies - I have fixed it (I hope! No Win98 SE here for me to > test on) Not yet, but we may be getting closer . First problem: test_support.py did TESTFN_UNICODE_UNENCODABLE = None on Win98SE, but test_unicode_filename.py tries to import TESTFN_UNICODE_UNENCODEABLE That's a clear spelling mistake in the former, so I'll fix that and check it in. Second problem: Then the test gets one failure: ====================================================================== ERROR: test_single_files (__main__.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_unicode_file.py", line 142, in test_single_files self._test_single(TESTFN_UNICODE) File "../lib/test/test_unicode_file.py", line 116, in _test_single self._do_single(filename) File "../lib/test/test_unicode_file.py", line 32, in _do_single os.utime(filename, None) UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-7: ordinal not in range(128) Note that the test doesn't skip itself on 98SE, because the try: TESTFN_ENCODED = TESTFN_UNICODE.encode(TESTFN_ENCODING) except (UnicodeError, TypeError): # Either the file system encoding is None, or the file name # cannot be encoded in the file system encoding. raise TestSkipped("No Unicode filesystem semantics on this platform.") doesn't trigger: >>> import sys >>> sys.getfilesystemencoding() 'mbcs' >>> u'@test-\xe0\xf2'.encode('mbcs') '@test-\xe0\xf2' >>> >> Don't know about 2.3 maint, & can't test that today: > The fix to posixmodule was checked into both head and 2.3-maint, but > the test suite changes were only made to head - for exactly this > reason . Thus, 2.3 should be fine. Yes, 2.3 maint *is* fine on 98SE. All the above was about the trunk. From mike.thompson at day8.com.au Thu Dec 4 05:53:55 2003 From: mike.thompson at day8.com.au (Mike Thompson) Date: Thu Dec 4 05:54:06 2003 Subject: [Python-Dev] Re: Re: Int FutureWarnings and other 2.4 TODOs References: <011201c3b9b3$f42b7880$6402a8c0@arkdesktop><200312031610.hB3GA1204029@c-24-5-183-134.client.comcast.net> <200312040505.hB455IJ05377@c-24-5-183-134.client.comcast.net> Message-ID: [Mike Thompson] > > Of course, this obvious solution has a problem: a programmer can't > > easily sub-class Int. They must choose one a sub-class from Short > > or Long (unless they want to provide a full implementation > > themselves) and that's probably not a decision they want to make. > [Guido van Rossum] > (It has been pointed out that subclassing int isn't very useful, so > maybe this is a moot point. Does anybody have a real use case?) I don't. It just rubs my intuition up the wrong way for it not to be subclassable, but that'll be because I'm not a fully reformed OO Bigot ;-) [Mike Thompson] > > Perhaps, because of this, the GOF "Bridge Pattern" might be suitable > > here. (This pattern can be framed as: "adapt multiple > > implementations to an interface using delegation" -- which, after > > all, is pretty much what the vtable in previous solution gives you.) > > > > If the existence of Short and Long is an implementation detail best > > hidden from the python programmer, then a Bridge Pattern > > implementation has a lot going for it. > [Guido van Rossum] > Hmm... I'm not very familiar with the Bridge pattern (and my GoF book > is on one of 65 boxes still in the garage waiting until we move into a > larger house :-). Can you give a little more detail about how this > would be done? To me, Bridge Pattern is most useful when either: 1. There is a need to sub-class the abstraction (Int) other than for implementational purposes. 2. The implementation of the abstraction (Int) needs to change at runtime. So, given you doubt the need for 1. and that 2. was never needed because of Int-immutability, I should withdraw my suggestion in favour of the more simple abstract-Int-base-with-Long-And-Short-Iimplementation-in-sub-classes approach previously mentioned. BTW, a summary of most patterns can be found on-line at http://patterndigest.com/ [Guido van Rossum] > Is it something that a user wishing to subclass Int > should do, or something that the Int class (or its subclasses) should > provide? Its something that the writer of the Int class would use iff they wanted to ensure that programmers were oblivious to the two or three possible implementations of Int AND one of the two conditions I mentioned above (needed subclassability or runtime-change in implementation) held true. [Mike Thompson] > > Use of the Bridge pattern might even allow for three different > > implementations: PreDefined, Short, Long ... where PreDefined is one > > of the numbers between 0 and 256 which I'm given to understand are > > pre-allocated by the runtime environment. > [Guido van Rossum] > I don't think so -- the beauty is that (most of) the implementation > doesn't know that certain int values have eternal life; their > representation is exactly the same as that of regular ints. > That beauty could remain. At the risk of lapsing completely into pattern vocabulary ... an AbstractFactory would create Ints and bind them to a subsequently hidden implementation. When the int involved was tiny (-1 to 256?) this AbstractFactory would use one of the pre-allocated, shared (see FlyWeight) implementations. Outside of this range distinct Short, Long or DamnHuge implementations would be created. -- Mike From pedronis at bluewin.ch Thu Dec 4 08:09:43 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Thu Dec 4 08:06:42 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <200312040322.hB43Mn814298@oma.cosc.canterbury.ac.nz> References: Message-ID: <5.2.1.1.0.20031204140010.0288fff8@pop.bluewin.ch> At 16:22 04.12.2003 +1300, Greg Ewing wrote: >Greg Ball : > > > (x[1] for x) == lambda x:x[1] > > (x.score for x) == lambda x: x.score > > (x+y for x,y) == lambda x,y: x+y > > (len(x) for x) == lambda x,len=len: len(x) # roughly equivalent > >Interesting idea, but it reads a bit strangely. > >Maybe this would work better with a different keyword... > > x[1] given x > x.score given x > x+y given x,y > len(x) given x > > stuff = groupby((x for x in seq), key = (x.score given x)) on the plus side it nicely parallels generator expressions and the keyword makes lookup in the docs easy. On the minus side is maybe verbose. On the other hand non-keyword notations risk to be too cryptic: some candidates: stuff = groupby((x for x in seq), key = (x.score\x)) # this would require the lexer to distinguish \ as line continuation vs this usage, # it would probably still be read as "x.score given x" stuff = groupby((x for x in seq), key = (x-->x.score)) # --> would be a single token From anthony at interlink.com.au Thu Dec 4 08:41:21 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Dec 4 08:41:36 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: Message from Anthony Baxter of "Tue, 02 Dec 2003 01:22:36 +1100." Message-ID: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> In the absence of any screams of "no, dear god, please don't", I am declaring the start of the 2.3.3 release process. I'll start cutting the rc1 tomorrow morning (in around 10 hours time). So, from now, until we get 2.3.3 final out the door (in about 2 weeks), please do NOT check in on the release23-maint branch unless your name is Anthony, Thomas, or Fred. If you see something that absolutely MUST be fixed, please check with me first. Remember - suprises on the release23-maint branch make for an unhappy release manager, and after the last couple of weeks, I'd _really_ like a nice, simple, smooth release. Thanks for your co-operation, Anthony From fdrake at acm.org Thu Dec 4 08:54:26 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Dec 4 08:54:32 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> References: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> Message-ID: <16335.15506.191582.612851@grendel.fdrake.net> Anthony Baxter writes: > So, from now, until we get 2.3.3 final out the door (in about > 2 weeks), please do NOT check in on the release23-maint branch > unless your name is Anthony, Thomas, or Fred. If you see something > that absolutely MUST be fixed, please check with me first. The release23-maint branch's "logging" package still uses apply(), which generates a PendingDeprecationWarning. I'd like to fix that (by removing the use of apply()). If that's ok, I can fix that now on the release23-maint branch and the trunk this morning. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From mcherm at mcherm.com Thu Dec 4 08:56:17 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Thu Dec 4 08:56:23 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs Message-ID: <1070546177.3fcf3d01e6f52@mcherm.com> [Guido writes:] > The implementation uses the type as the vtable of a C++ class, > and the functions in the vtable "know" the representation, which is > very different for short and long ints. This is all nice and > efficient, except that the identity of the vtable is returned as the > type of the object, so two different vtables means two different > types. ... [and clipping overaggressively:] > The alternative ... embed a flag ... downslide should be obvious > ... extra test+branch ... also extra space ... no spare bits. I have an idea, but don't know enough about the type mechanisms to know whether this is possible. So I'll throw it out there and let you shoot it down. To my mind, the *ideal* is that there's a single type, probably called "int" or "integer" which can take on an arbitrary size and precision BUT which, if it happens to be small enough to fit in a C int will take up less memory and run faster. The fact that it's stored differently should be an implementation detail not visible to the user. But which user? I would say that someone using the C api's to write extension modules probably DOES care about the particular representation of ints vs longs, and it's not unreasonable to expect them to deal with it. But for someone programming from Python it should "just work". If you agree with this premise, then it suggests a different approach. Use different implementations in C, but have type(x) in Python lie. type(x) would always return the type object which is now known as "long". That type object would be smart enough to realize that any time the object it is dealing with had a value in the range [-MAXINT-1, MAXINT] that it was ACTUALLY using the layout now known as "int". This would require some clever tricks in the implementation of the "long" type object, but that's not where we care about performance. Of course, it's possible that there are no tricks clever enough to handle this -- I certainly don't know how to do it, but I wouldn't be TOO surprised if someone else here did. -- Michael Chermside PS: If this approach were used, the sensible type names would be something like "shortint" (visible only from C) and "int" (today's long). From mcherm at mcherm.com Thu Dec 4 09:08:29 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Thu Dec 4 09:08:33 2003 Subject: [Python-Dev] Re: Int FutureWarnings and other 2.4 TODOs Message-ID: <1070546909.3fcf3fdd143e5@mcherm.com> Guido writes: > Hmm... I'm not very familiar with the Bridge pattern (and my GoF book > is on one of 65 boxes still in the garage waiting until we move into a > larger house :-). Can you give a little more detail about how this > would be done? If I understand it properly (and I DID check my copy of GOF which isn't packed ), it may be better recognized by its alternative name: "Handle/Body". The basic idea is to add a level of indirection... the GOF name the "front" object the "Abstraction" and the "back" object the "Implementor". And it seems like a poor choice for implementing ints, which by definition should be designed for the minimum possible overhead, unless that "front" object can be made VERY lightweight. (Hmm... to avoid cache effects we might want to store the "abstraction" adjacent in memory to the "implementor". And if we did THAT we might even be able to dispense with even a pointer in the "abstraction". But now I've talked myself out of the bridge pattern into something more like just having two separate types... really 3 now, since there's a third layout needed to handle user-designed int subclasses. I still don't like it.) -- Michael Chermside From ark-mlist at att.net Thu Dec 4 09:12:17 2003 From: ark-mlist at att.net (Andrew Koenig) Date: Thu Dec 4 09:12:09 2003 Subject: [Python-Dev] Re: Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312040505.hB455IJ05377@c-24-5-183-134.client.comcast.net> Message-ID: <002b01c3ba70$9fce5970$6402a8c0@arkdesktop> > (It has been pointed out that subclassing int isn't very useful, so > maybe this is a moot point. Does anybody have a real use case?) isinstance(x, int) That doesn't require an inheritance relationship between the integral types, but it does require that long inherit from int. So int could be a base class with long and short inheriting from it, or long could inherit from int. From ark-mlist at att.net Thu Dec 4 09:36:40 2003 From: ark-mlist at att.net (Andrew Koenig) Date: Thu Dec 4 09:36:35 2003 Subject: [Python-Dev] Re: Re: Int FutureWarnings and other 2.4 TODOs In-Reply-To: Message-ID: <003301c3ba74$0849c1d0$6402a8c0@arkdesktop> > At the risk of lapsing completely into pattern vocabulary ... an > AbstractFactory would create Ints and bind them to a subsequently hidden > implementation. When the int involved was tiny (-1 to 256?) this > AbstractFactory would use one of the pre-allocated, shared (see FlyWeight) > implementations. Outside of this range distinct Short, Long or DamnHuge > implementations would be created. Or, since we're talking implementation, we could use the following hack, which I borrowed from Standard ML of New Jersey: There is just one type, namely int. An int is a 31-bit integer, INCLUDING sign. One extra bit indicates whether the integer is really a number or, alternatively, a pointer to the rest of the representation. Now, you may object that implementing this strategy in C will require lots of shifting and masking. I would have thought so, too. However, every C implementation of which I am aware puts all but the most trivial data structures on boundaries of two or more bytes. This fact frees the LOW-order bit of the integer to indicate whether it is a pointer. So here's the strategy: If the low-order bit of an integer is *off*, it's really a pointer to the rest of the implementation. If the low-order bit is *on*, then it represents an integral value that can be obtained by doing a one-bit arithmetic right shift. Yes, it's sleazy. But I imagine it would be much faster than using inheritance. From fincher.8 at osu.edu Thu Dec 4 10:38:25 2003 From: fincher.8 at osu.edu (Jeremy Fincher) Date: Thu Dec 4 09:41:07 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: <16335.15506.191582.612851@grendel.fdrake.net> References: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> <16335.15506.191582.612851@grendel.fdrake.net> Message-ID: <200312041038.26937.fincher.8@osu.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday 04 December 2003 08:54 am, Fred L. Drake, Jr. wrote: > Anthony Baxter writes: > > So, from now, until we get 2.3.3 final out the door (in about > > 2 weeks), please do NOT check in on the release23-maint branch > > unless your name is Anthony, Thomas, or Fred. If you see something > > that absolutely MUST be fixed, please check with me first. > > The release23-maint branch's "logging" package still uses apply(), > which generates a PendingDeprecationWarning. I'd like to fix that (by > removing the use of apply()). I thought the logging module was supposed to remain 1.5.2-compatible. At least, that's the reason I saw the last time someone wanted to remove those apply calls. Jeremy -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (GNU/Linux) iD8DBQE/z1TxqkDiu+Bs+JIRAuAUAJ4601VMuG59mew2nKJvtH6w1w0xewCfY2ly KAy/iM77SCqEawJODJAfCZ8= =C9V5 -----END PGP SIGNATURE----- From fdrake at acm.org Thu Dec 4 09:51:46 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Dec 4 09:52:03 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: <200312041038.26937.fincher.8@osu.edu> References: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> <16335.15506.191582.612851@grendel.fdrake.net> <200312041038.26937.fincher.8@osu.edu> Message-ID: <16335.18946.557564.95591@grendel.fdrake.net> Jeremy Fincher writes: > I thought the logging module was supposed to remain 1.5.2-compatible. At > least, that's the reason I saw the last time someone wanted to remove those > apply calls. That sounds vaguely familiar. If we really want the copy distributed with Python to be compatible with Python's that old, we can use a different approach. Unfortunately, we can't simply add something like: if sys.version_info >= (2, 3): def apply(func, args, kw=None): if kw is None: return func(*args) else: return func(*args, **kw) This isn't permissible since it would cause a SyntaxError for Python 1.5.2; we'd have to enlist the "exec" statement to make this work. Do we still really want to support Python 1.5.2? That just seems quite extreme to me; even PyXML is no longer maintained for Python 1.5.2. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From mwh at python.net Thu Dec 4 09:52:24 2003 From: mwh at python.net (Michael Hudson) Date: Thu Dec 4 09:53:04 2003 Subject: [Python-Dev] are CObjects inherently unsafe? Message-ID: <78CB3A69-2669-11D8-9361-0003931DF95C@python.net> It seems to me that any time there are more than two CObjects around we have a good chance of causing mischief: >>> import cStringIO, _curses >>> cStringIO.cStringIO_CAPI = _curses._C_API >>> import cPickle >>> cPickle.dumps(range(10)) Illegal instruction (if you're trying this at home, starting python with -E might be necessary to make sure cPickle hasn't been imported already). Question 1: do we care? I think we do: "no core dumps" is a pretty strong Pythonic principle, even though this is certainly a "doctor it hurts when I do this" situation. OTOH, Armin's gc.get_referrers hack was deemed allowable. It seems to me that it would be more sensible to have a dict mapping names (maybe just module names...) to cobjects in the interpreter state. I guess there might be a way of getting ahold of the dict with the gc module (though I can't think of one off hand). This wouldn't be a difficult change, and if the CObjects are left in the module dicts, it shouldn't even do that much damage to binary compatibility. Thoughts? Cheers, mwh (who also notices that the cobject section in Doc/ext/ could do with being updated to use PyCObject_Import...) From ncoghlan at iinet.net.au Thu Dec 4 09:50:40 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu Dec 4 09:55:38 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312031726.hB3HQ6504171@c-24-5-183-134.client.comcast.net> References: <011201c3b9b3$f42b7880$6402a8c0@arkdesktop> <200312031610.hB3GA1204029@c-24-5-183-134.client.comcast.net> <200312031215.17933.fincher.8@osu.edu> <200312031726.hB3HQ6504171@c-24-5-183-134.client.comcast.net> Message-ID: <3FCF49C0.4010702@iinet.net.au> [Guido] >> I guess that means >> I have to work harder and make the single int type support both >> representations. I'm sure it can be done. Guido van Rossum wrote: >>>So the question is, does long have operations that int doesn't have? >>>And if so, why can't those operations be added to int? And if there's >>>a reason, is it good enough? >> >>Taking into account their difference in representation, a long can support >>1<<32, but an int can't. > > We're talking about a hypothetical int here where that operation > returns 4294967296L. (Not so hypothetical, it's implemented in Python > 2.4 in CVS.) The 'performance hack' point of view I was trying to suggest was along the lines of: "Python integers are capable of storing values of arbitrary magnitude, subject only to the memory capacity of the machine. As a matter of performance, Python will use native C integers (and native arithmetic) when the stored value is small enough to fit." That is, I was agreeing with Guido's first point above. From barry at python.org Thu Dec 4 10:02:54 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 4 10:03:01 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: <16335.18946.557564.95591@grendel.fdrake.net> References: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> <16335.15506.191582.612851@grendel.fdrake.net> <200312041038.26937.fincher.8@osu.edu> <16335.18946.557564.95591@grendel.fdrake.net> Message-ID: <1070550162.27289.5.camel@anthem> On Thu, 2003-12-04 at 09:51, Fred L. Drake, Jr. wrote: > Jeremy Fincher writes: > > I thought the logging module was supposed to remain 1.5.2-compatible. So says PEP 291. > That sounds vaguely familiar. If we really want the copy distributed > with Python to be compatible with Python's that old, we can use a > different approach. Unfortunately, we can't simply add something like: > > if sys.version_info >= (2, 3): > def apply(func, args, kw=None): > if kw is None: > return func(*args) > else: > return func(*args, **kw) > > This isn't permissible since it would cause a SyntaxError for Python > 1.5.2; we'd have to enlist the "exec" statement to make this work. The email package has a fairly elaborate mechanism to support compatibility with older Pythons. It segregates syntactically incompatible stuff into separate modules _compat21.py and _compat22.py and then imports the appropriate one based on the Python version. Yes, it's a hack. > Do we still really want to support Python 1.5.2? That just seems > quite extreme to me; even PyXML is no longer maintained for Python > 1.5.2. I think I'd like to elaborate PEP 291 a bit. It seems perfectly reasonable to say that Python 2.3's email package must remain compatible with Python 2.1, but that Python 2.4's email package needs no such backwards compatibility. Now, the email package has its own version number so in that sense, I could say that email 2.x is Python 2.1 compatible but email 3.x requires Python 2.3 or 2.4. I'm not sure how well that works with the other packages listed in PEP 291, but given that Python has multiple branches, I don't think we have to require the code on the trunk to keep backwards compatibility with ancient Pythons. -Barry From barry at python.org Thu Dec 4 10:14:50 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 4 10:14:55 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: <1070550162.27289.5.camel@anthem> References: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> <16335.15506.191582.612851@grendel.fdrake.net> <200312041038.26937.fincher.8@osu.edu> <16335.18946.557564.95591@grendel.fdrake.net> <1070550162.27289.5.camel@anthem> Message-ID: <1070550889.27289.8.camel@anthem> On Thu, 2003-12-04 at 10:02, Barry Warsaw wrote: > I think I'd like to elaborate PEP 291 a bit. Which I've just done. It's a little awkward to get all the information in there, but see if you like my 5 minute update. http://www.python.org/peps/pep-0291.html -Barry From pje at telecommunity.com Thu Dec 4 10:38:21 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Dec 4 10:36:23 2003 Subject: [Python-Dev] Re: Re: Int FutureWarnings and other 2.4 TODOs In-Reply-To: <003301c3ba74$0849c1d0$6402a8c0@arkdesktop> References: Message-ID: <5.1.0.14.0.20031204103428.034866f0@mail.telecommunity.com> At 09:36 AM 12/4/03 -0500, Andrew Koenig wrote: >So here's the strategy: If the low-order bit of an integer is *off*, it's >really a pointer to the rest of the implementation. If the low-order bit is >*on*, then it represents an integral value that can be obtained by doing a >one-bit arithmetic right shift. > >Yes, it's sleazy. But I imagine it would be much faster than using >inheritance. I imagine it wouldn't, because it'd add an extra test to not only every Py_INCREF and Py_DECREF, but every PyObject_something call. From mwh at python.net Thu Dec 4 10:45:11 2003 From: mwh at python.net (Michael Hudson) Date: Thu Dec 4 10:45:17 2003 Subject: [Python-Dev] Re: Re: Int FutureWarnings and other 2.4 TODOs In-Reply-To: <5.1.0.14.0.20031204103428.034866f0@mail.telecommunity.com> (Phillip J. Eby's message of "Thu, 04 Dec 2003 10:38:21 -0500") References: <5.1.0.14.0.20031204103428.034866f0@mail.telecommunity.com> Message-ID: <2moeuowprs.fsf@starship.python.net> "Phillip J. Eby" writes: > At 09:36 AM 12/4/03 -0500, Andrew Koenig wrote: >>So here's the strategy: If the low-order bit of an integer is *off*, it's >>really a pointer to the rest of the implementation. If the low-order bit is >>*on*, then it represents an integral value that can be obtained by doing a >>one-bit arithmetic right shift. >> >>Yes, it's sleazy. But I imagine it would be much faster than using >>inheritance. > > I imagine it wouldn't, because it'd add an extra test to not only > every Py_INCREF and Py_DECREF, but every PyObject_something call. It wouldn't have to be that bad if you put the pointer/int thingy in the ob_ival slot. Cheers, mwh -- The only problem with Microsoft is they just have no taste. -- Steve Jobs, (From _Triumph of the Nerds_ PBS special) and quoted by Aahz on comp.lang.python From fdrake at acm.org Thu Dec 4 11:30:19 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Dec 4 11:30:30 2003 Subject: [Python-Dev] Re: Re: Int FutureWarnings and other 2.4 TODOs In-Reply-To: <003301c3ba74$0849c1d0$6402a8c0@arkdesktop> References: <003301c3ba74$0849c1d0$6402a8c0@arkdesktop> Message-ID: <16335.24859.801823.731366@grendel.fdrake.net> Andrew Koenig writes: > An int is a 31-bit integer, INCLUDING sign. One extra bit indicates whether > the integer is really a number or, alternatively, a pointer to the rest of > the representation. > > Now, you may object that implementing this strategy in C will require lots > of shifting and masking. I would have thought so, too. However, every C > implementation of which I am aware puts all but the most trivial data > structures on boundaries of two or more bytes. This fact frees the > LOW-order bit of the integer to indicate whether it is a pointer. I recently mentioned this approach to coaxing an extra bit from an existing structure to Tim, and he pointed out (rightly!) that this only works for machines with byte-addressable memories. On word-addressable memories, there's no chance to coax the extra bit from a pointer at all, and Python still runs on at least one such platform (some Cray thing, IIRC). Otherwise I like the approach. ;-) -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tim.one at comcast.net Thu Dec 4 11:44:09 2003 From: tim.one at comcast.net (Tim Peters) Date: Thu Dec 4 11:44:11 2003 Subject: [Python-Dev] Re: Re: Int FutureWarnings and other 2.4 TODOs In-Reply-To: <2moeuowprs.fsf@starship.python.net> Message-ID: [Andrew Koenig] >>> So here's the strategy: If the low-order bit of an integer is >>> *off*, it's really a pointer to the rest of the implementation. If >>> the low-order bit is *on*, then it represents an integral value >>> that can be obtained by doing a one-bit arithmetic right shift. >>> >>> Yes, it's sleazy. But I imagine it would be much faster than using >>> inheritance. [Phillip J. Eby] >> I imagine it wouldn't, because it'd add an extra test to not only >> every Py_INCREF and Py_DECREF, but every PyObject_something call. [Michael Hudson] > It wouldn't have to be that bad if you put the pointer/int thingy in > the ob_ival slot. Not all HW Python runs on is byte-addressed, so the base idea that "the last bit" of a pointer-to-non-trivial-structure is always 0 doesn't get off the ground. C doesn't have an arithmetic right-shift operator either, but that one is easier to worm around (pyport.h already has a Py_ARITHMETIC_RIGHT_SHIFT macro to "do the right thing"). OTOH, the only current Python platform I know of that's word-addressed is the Cray T3E, which also happens to be the only one I know of where C's signed ">>" zero-fills (and is also the only one I know of that has no 16-bit integral type -- the T3E porters had an interesting time). From kbk at shore.net Thu Dec 4 12:35:05 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Thu Dec 4 12:35:14 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> (Anthony Baxter's message of "Fri, 05 Dec 2003 00:41:21 +1100") References: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> Message-ID: Anthony Baxter writes: > So, from now, until we get 2.3.3 final out the door (in about > 2 weeks), please do NOT check in on the release23-maint branch > unless your name is Anthony, Thomas, or Fred. If you see something > that absolutely MUST be fixed, please check with me first. idlelib: idlever.py needs to be updated to 1.0.2 The release date needs to be set in NEWS.txt I can do this if you like, or you can. Let me know. -- KBK From gjc at inescporto.pt Thu Dec 4 12:51:06 2003 From: gjc at inescporto.pt (Gustavo J. A. M. Carneiro) Date: Thu Dec 4 12:51:48 2003 Subject: [Python-Dev] LC_NUMERIC patch updated Message-ID: <1070560266.12807.25.camel@spectrum.inescn.pt> It is SF patch [ 774665 ] making Python LC_NUMERIC agnostic. With this new patch I have also removed the code hacks that keep LC_NUMERIC always set to 'C'. I hope there aren't serious objections, because this is actually the main point of adding locale-independent floating point conversion routines. Anything else I can do? Regards. -- Gustavo Jo?o Alves Marques Carneiro From gisle at ActiveState.com Thu Dec 4 13:40:16 2003 From: gisle at ActiveState.com (Gisle Aas) Date: Thu Dec 4 13:40:29 2003 Subject: [Python-Dev] Re: Re: Int FutureWarnings and other 2.4 TODOs In-Reply-To: <003301c3ba74$0849c1d0$6402a8c0@arkdesktop> References: <003301c3ba74$0849c1d0$6402a8c0@arkdesktop> Message-ID: "Andrew Koenig" writes: > An int is a 31-bit integer, INCLUDING sign. One extra bit indicates whether > the integer is really a number or, alternatively, a pointer to the rest of > the representation. This is exactly how it's done in Ruby as well. Search for FIXNUM in http://www.ruby-lang.org/cgi-bin/cvsweb.cgi/ruby/ruby.h?rev=1.95&content-type=text/x-cvsweb-markup Regards, Gisle From Juli at joelaw.cc Thu Dec 4 14:33:23 2003 From: Juli at joelaw.cc (Juli Longacre) Date: Thu Dec 4 14:33:34 2003 Subject: [Python-Dev] PEP draft - Guidelines to processing patches Message-ID: <008e01c3ba9d$7c1d9090$07fea8c0@joesonydesktop> I am trying to find a Michael Bartl, son of Theresa Irvin and Richard Bajenske for a legal matter. If you are that person, please contact me immediately. Thank you. Juli Longacre Paralegal to Joe B. Stephens The Stephens Law Firm The Lyric Center 440 Louisiana, Suite 2000 Houston, Texas 77002 713-224-0000 713-224-0055 fax juli@joelaw.cc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031204/9dbea920/attachment.html From theller at python.net Thu Dec 4 15:23:51 2003 From: theller at python.net (Thomas Heller) Date: Thu Dec 4 15:24:13 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: (Kurt B. Kaiser's message of "Thu, 04 Dec 2003 12:35:05 -0500") References: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> Message-ID: kbk@shore.net (Kurt B. Kaiser) writes: > Anthony Baxter writes: > >> So, from now, until we get 2.3.3 final out the door (in about >> 2 weeks), please do NOT check in on the release23-maint branch >> unless your name is Anthony, Thomas, or Fred. If you see something >> that absolutely MUST be fixed, please check with me first. > > idlelib: > > idlever.py needs to be updated to 1.0.2 > > The release date needs to be set in NEWS.txt > > I can do this if you like, or you can. Let me know. I'll do it. Can you update PEP 101 and 102 with this information, please? Thanks, Thomas From guido at python.org Thu Dec 4 15:35:17 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 4 15:35:28 2003 Subject: [Python-Dev] Re: Re: Int FutureWarnings and other 2.4 TODOs In-Reply-To: Your message of "Thu, 04 Dec 2003 09:36:40 EST." <003301c3ba74$0849c1d0$6402a8c0@arkdesktop> References: <003301c3ba74$0849c1d0$6402a8c0@arkdesktop> Message-ID: <200312042035.hB4KZHn06585@c-24-5-183-134.client.comcast.net> > Or, since we're talking implementation, we could use the following hack, > which I borrowed from Standard ML of New Jersey: [...] This is a well-known scheme. We used it in ABC 20 years ago. My experience with it was negative though: there were too many core dumps because some code was dereferencing a pointer without checking the low bit. So I'd rather not try this. I also somehow doubt that the speed gains are that significant. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Dec 4 15:40:08 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 4 15:40:17 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: Your message of "Thu, 04 Dec 2003 05:56:17 PST." <1070546177.3fcf3d01e6f52@mcherm.com> References: <1070546177.3fcf3d01e6f52@mcherm.com> Message-ID: <200312042040.hB4Ke8D06619@c-24-5-183-134.client.comcast.net> > If you agree with this premise, then it suggests a different > approach. Use different implementations in C, but have type(x) in > Python lie. type(x) would always return the type object which is > now known as "long". If this can be made to work, I like it. __class__ should also be hacked, and isinstance(); and who knows how many other places, but probably not too many. > That type object would be smart enough to realize that any time the > object it is dealing with had a value in the range [-MAXINT-1, > MAXINT] that it was ACTUALLY using the layout now known as "int". This part doesn't make sense -- the value cannot be known unless the layout is known. But it shouldn't matter -- the long methods should never be passed a short representation, because at the C level the short implementation would be invoked automatically through the vtable or type. Of course, the long code should be careful to downcase any *results* in the short range to the short representation before returning it. > This would require some clever tricks in the implementation of the > "long" type object, but that's not where we care about performance. > Of course, it's possible that there are no tricks clever enough to > handle this -- I certainly don't know how to do it, but I wouldn't > be TOO surprised if someone else here did. Someone ought to try to implement this on a branch or as a patch, to see how feasible it is. --Guido van Rossum (home page: http://www.python.org/~guido/) From python-kbutler at sabaydi.com Thu Dec 4 16:27:31 2003 From: python-kbutler at sabaydi.com (Kevin J. Butler) Date: Thu Dec 4 16:27:50 2003 Subject: [Python-Dev] re: Tutorial: Brief Introduction to the Standard Library In-Reply-To: References: Message-ID: <3FCFA6C3.90509@sabaydi.com> From: Arthur >What is fundamentally wrong with copy(list) as an alternative to list[:]? > > Well, copy(list) doesn't contain an emoticon of two beady little eyes peeking out of a box, for one. ;-) The opposite question, "What is fundamentally wrong with list[:]..." has an easy answer: It takes experience or explanation of slicing to know what it does. copy(list) is easy even for novices to understand. When I know my code will be read by people unfamiliar with Python, I tend to annotate use of slices something like: list[:] # copy the list Python usually makes it easier to avoid sacrificing clarity on the altar of micro-optimization, but since the slice is the idiomatically correct way to spell copy(list) in Python, that's the way to write it. kb From kbk at shore.net Thu Dec 4 16:52:32 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Thu Dec 4 16:52:35 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: (Thomas Heller's message of "Thu, 04 Dec 2003 21:23:51 +0100") References: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> Message-ID: Thomas Heller writes: > Can you update PEP 101 and 102 with this information, please? Yes, I'll update them to cover the IDLE details. -- KBK From martin at v.loewis.de Thu Dec 4 14:37:07 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 4 17:06:48 2003 Subject: [Python-Dev] are CObjects inherently unsafe? In-Reply-To: <78CB3A69-2669-11D8-9361-0003931DF95C@python.net> References: <78CB3A69-2669-11D8-9361-0003931DF95C@python.net> Message-ID: Michael Hudson writes: > It seems to me that it would be more sensible to have a dict mapping > names (maybe just module names...) to cobjects in the interpreter > state. I guess there might be a way of getting ahold of the dict with > the gc module (though I can't think of one off hand). This wouldn't > be a difficult change, and if the CObjects are left in the module > dicts, it shouldn't even do that much damage to binary compatibility. > > Thoughts? Good idea. Alternatively, we could make "APIs" a feature of modules: PyModule_GetAPI, PyModule_SetAPI. We would then define typedef struct { PyObject_HEAD PyObject *md_dict; PyObject *md_api; } PyModuleObject; We could restrict md_api to CObjects, which, means we would not need to change module_traverse. Regards, Martin From pje at telecommunity.com Thu Dec 4 18:27:28 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Dec 4 18:30:17 2003 Subject: [Python-Dev] re: Tutorial: Brief Introduction to the Standard Library In-Reply-To: <3FCFA6C3.90509@sabaydi.com> References: Message-ID: <5.1.1.6.0.20031204182545.02ae79c0@telecommunity.com> At 02:27 PM 12/4/03 -0700, Kevin J. Butler wrote: >The opposite question, "What is fundamentally wrong with list[:]..." has >an easy answer: > >It takes experience or explanation of slicing to know what it does. >copy(list) is easy even for novices to understand. > >When I know my code will be read by people unfamiliar with Python, I tend >to annotate use of >slices something like: > > list[:] # copy the list > >Python usually makes it easier to avoid sacrificing clarity on the altar >of micro-optimization, but since the slice is the idiomatically correct >way to spell copy(list) in Python, that's the way to write it. Well, dictionaries have a .copy() method; lists could also grow one. From jjl at pobox.com Thu Dec 4 18:36:05 2003 From: jjl at pobox.com (John J Lee) Date: Thu Dec 4 18:36:12 2003 Subject: [Python-Dev] Re: Int FutureWarnings and other 2.4 TODOs In-Reply-To: <002b01c3ba70$9fce5970$6402a8c0@arkdesktop> References: <002b01c3ba70$9fce5970$6402a8c0@arkdesktop> Message-ID: On Thu, 4 Dec 2003, Andrew Koenig wrote: > > (It has been pointed out that subclassing int isn't very useful, so > > maybe this is a moot point. Does anybody have a real use case?) > > isinstance(x, int) If that's really the only use case, is the following not sufficient? isinstance(x, types.IntegralTypes) (where IntegralTypes == (int, long), analogously to types.StringTypes) John From greg at cosc.canterbury.ac.nz Thu Dec 4 18:44:03 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 4 18:44:11 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: <20199DBD-260F-11D8-A36A-000A957D895E@post.harvard.edu> Message-ID: <200312042344.hB4Ni3K22308@oma.cosc.canterbury.ac.nz> Alexis Layton : > One simple style is to "infer" the lambda using positional > arguments, for example: > > (&1 + &2.foo()) > > This style of construct implies the lambda or function without really > saying it, which may be too magical for Python. I think it's ambiguous, too, since it's not clear how far the lambda extends. Is ((&1 + &2), (&1 - &2)) one lambda returning a tuple, or a tuple containing two lambdas? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From nas-python at python.ca Thu Dec 4 19:02:17 2003 From: nas-python at python.ca (Neil Schemenauer) Date: Thu Dec 4 18:55:25 2003 Subject: [Python-Dev] Re: Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312040254.hB42sLI14069@oma.cosc.canterbury.ac.nz> References: <200312040254.hB42sLI14069@oma.cosc.canterbury.ac.nz> Message-ID: <20031205000217.GA19503@mems-exchange.org> On Thu, Dec 04, 2003 at 03:54:21PM +1300, Greg Ewing wrote: > Does anyone have a real-life use case for subclassing int? Not me. Dylan does not allow it's concrete numeric classes to be subclassed. That allows efficient method dispatch given restrictive enough declarations. Here is the standard numberic classes: class number(object): # Open Abstract Class class complex(number): # Sealed Abstract Class class real(complex): # Sealed Abstract Class class float(real): # Sealed Abstract Class class single_float(float) # Sealed Class class double_float(float) # Sealed Class class rational(real): # Sealed Abstract Class class integer(rational): # Sealed Class complex is sealed which basically means that it cannot be subclassed by users. That allows Sealing is described here: http://www.gwydiondylan.org/drm/drm_70.htm. The language reference does not describe a "long" integer type but the implementation is free to provide one. Neil From ajsiegel at optonline.net Thu Dec 4 06:59:45 2003 From: ajsiegel at optonline.net (Arthur) Date: Thu Dec 4 18:56:10 2003 Subject: [Python-Dev] re: Tutorial: Brief Introduction to the Standard Libary Message-ID: <0HPD007N0D5XMT@mta4.srv.hcvlny.cv.net> Guido writes: >Can we stop this thread on copy()? It's one of Arthur's pet subjects >and you're not going to convince him, so it's pointless to argue. Convince me of what? If I have done nothing more than demonstrate that someone - from a substrata of the audience Python talks about reaching, and who has been using it for over 4 years - is confused and ignorant in this area.... Seems to me I have made my point. So will go away. Art From greg at cosc.canterbury.ac.nz Thu Dec 4 19:47:02 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 4 19:47:09 2003 Subject: [Python-Dev] Re: Int FutureWarnings and other 2.4 TODOs In-Reply-To: <002b01c3ba70$9fce5970$6402a8c0@arkdesktop> Message-ID: <200312050047.hB50l2K22493@oma.cosc.canterbury.ac.nz> Andrew Koenig : > > (It has been pointed out that subclassing int isn't very useful, so > > maybe this is a moot point. Does anybody have a real use case?) > > isinstance(x, int) > > That doesn't require an inheritance relationship between the integral types, > but it does require that long inherit from int. Having int be an abstract class doesn't prevent it from being subclassed - it just means the subclass needs to to more work in order to behave like other concrete int subclasses. Along with short and long, there could be a UserInt subclass which delegates all the operations to an implementation object, for people who really want an int subclass for some reason. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From edloper at gradient.cis.upenn.edu Thu Dec 4 20:52:35 2003 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Thu Dec 4 19:51:11 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: References: Message-ID: <3FCFE4E3.4000602@gradient.cis.upenn.edu> >>If you agree with this premise, then it suggests a different >>approach. Use different implementations in C, but have type(x) in >>Python lie. type(x) would always return the type object which is >>now known as "long". > > If this can be made to work, I like it. __class__ should also be > hacked, and isinstance(); and who knows how many other places, but > probably not too many. If we go this route, then how general should the mechanism for "merging" types in Python-space (but keeping them separate in c-space) be? I.e., should this be an integer-specific change, or a general mechanism that happens to be used by int? It seems like there might be other use cases for similar "merges" (string/unicode comes to mind). -Edward From greg at cosc.canterbury.ac.nz Thu Dec 4 19:56:43 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 4 19:56:50 2003 Subject: [Python-Dev] are CObjects inherently unsafe? In-Reply-To: <78CB3A69-2669-11D8-9361-0003931DF95C@python.net> Message-ID: <200312050056.hB50uh722511@oma.cosc.canterbury.ac.nz> Michael Hudson : > It seems to me that any time there are more than two CObjects around we > have a good chance of causing mischief: Indeed, CObjects seem fundamentally dangerous to me, unless the modules which create and use them are extremely careful to make sure they can tell whether they've got the right kind of CObject. And I suspect that many modules which make use of CObjects aren't that careful. CStringIO obviously isn't... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From kbk at shore.net Thu Dec 4 19:57:11 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Thu Dec 4 19:57:16 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: (Thomas Heller's message of "Thu, 04 Dec 2003 21:23:51 +0100") References: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> Message-ID: I notice there are a number of version updates you guys are making which aren't in the PEPs. Do you want me to work up a patch? Second, the PEPs talk about "rc" release id's but you are all using "c". Do you want that included in the patch? -- KBK From greg at cosc.canterbury.ac.nz Thu Dec 4 20:09:17 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 4 20:09:23 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312042040.hB4Ke8D06619@c-24-5-183-134.client.comcast.net> Message-ID: <200312050109.hB519Hw22644@oma.cosc.canterbury.ac.nz> Guido van Rossum : > > If you agree with this premise, then it suggests a different > > approach. Use different implementations in C, but have type(x) in > > Python lie. type(x) would always return the type object which is > > now known as "long". > > If this can be made to work, I like it. __class__ should also be > hacked, and isinstance(); and who knows how many other places, but > probably not too many. What would happen under this scheme if someone did try to subclass int, or long? What would isinstance() make of such subclasses? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From fdrake at acm.org Thu Dec 4 22:32:42 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Dec 4 22:32:56 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: References: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> Message-ID: <16335.64602.857668.389843@grendel.fdrake.net> Kurt B. Kaiser writes: > I notice there are a number of version updates you guys are making > which aren't in the PEPs. Do you want me to work up a patch? Which changes in particular? I think the version updates in Doc/ aren't in the PEP anymore because they're no longer needed on the trunk. I don't know about the rest of the changes. I'd be more interested in problems making a trunk release based on the PEPs than a branch release, but it's probably worth noting the differences in a separate section. > Second, the PEPs talk about "rc" release id's but you are all using > "c". Do you want that included in the patch? This would be good. The "c" is for "candidate", which is intended to sort before "final". -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From fumanchu at amor.org Fri Dec 5 00:18:08 2003 From: fumanchu at amor.org (Robert Brewer) Date: Fri Dec 5 00:23:20 2003 Subject: [Python-Dev] Re: "groupby" iterator Message-ID: How did this thread start again? ;) Guido van Rossum, a long time ago, wrote: > In the shower (really!) I was thinking about the old problem of going > through a list of items that are supposed to be grouped by some key, > and doing something extra at the end of each group... > So I realized this is easy to do with a generator, assuming we can > handle keeping a list of all items in a group. > for group in groupby(key, sequence): Greg Ewing's proposal of a "given" keyword (x.score given x) got me thinking. I figured I would play around a bit and try to come up with the most readable version of the original "groupby" idea (for which I could imagine *some* implementation): for group in sequence.groups(using item.score - item.penalty): ...do stuff with group Having written this down, it seems to me the most readable so far. The keyword "using" creates a new scope, within which "item" is bound to the arg (or *args?) passed in. I don't know about you all, but the thing I like least about lambda is having to mention 'x' twice: lambda x: x.score Why have the programmer bind a custom name to an object we're going to then use 'anonymously' anyway? I understand its historical necessity, but it's always struck me as more complex than the concept being implemented. Ideally, we should be able to reference the passed-in objects without having to invent names for them. Now, consider multi-arg lambdas such as: sequence.sort(lambda x, y: cmp(x[0], y[0])) In these cases, we wish to apply the same operation to each item (that is, we calculate x[0] and y[0]). If we bind "item" to each argument *in turn*, we save a lot of syntax. The above might then be written as: sequence.sort(using cmp(item[0])) # Hard to implement. or: sequence.sort(cmp(using item[0])) # Easier but ugly. Meh. or: sequence.sort(cmp using item[0]) # Oooh. Nice. :) or: # might we assume cmp(), since sort does...? sequence.sort(using item[0]) I like #3, since cmp is explicit but doesn't use cmp(), which looks too much like a call. Given (cmp using item[0]), the "using block" would look at the arguments supplied by sort(), call __getitem__[0] for each, and pass those values in order into cmp, returning the result. The "item" keyword functions similarly to Guido's Voodoo.foo() proposal, now that I think about it. There's no reason it couldn't grow some early binding, either, as suggested, although multiple operations would become unwieldy. How would you early-bind this? sequence.groups(using divmod(item, 4)[1]) ...except perhaps by using multiply-nested scopes to bind the "1" and then the "4"? Hmm. It would have to do some fancy dancing to get everything in the right order. Too much like reinventing Python to think about at the moment. :) The point is, passing the "item" instance through such a scheme should be the easy part. Robert Brewer MIS Amor Ministries fumanchu@amor.org From aahz at pythoncraft.com Fri Dec 5 00:57:33 2003 From: aahz at pythoncraft.com (Aahz) Date: Fri Dec 5 00:57:35 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <1070546177.3fcf3d01e6f52@mcherm.com> References: <1070546177.3fcf3d01e6f52@mcherm.com> Message-ID: <20031205055733.GA12283@panix.com> On Thu, Dec 04, 2003, Michael Chermside wrote: > > To my mind, the *ideal* is that there's a single type, probably > called "int" or "integer" which can take on an arbitrary size and > precision BUT which, if it happens to be small enough to fit in > a C int will take up less memory and run faster. The fact that > it's stored differently should be an implementation detail not > visible to the user. Hmmmm... How important is the "less memory" angle versus the "faster" angle? Why not just add a few bytes to the long type to store an int? If you're using the int, the pointer is NULL, and you're saving that memory, at least. The only harm is some extra memory in the type object. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From tim at multitalents.net Fri Dec 5 00:57:47 2003 From: tim at multitalents.net (Tim Rice) Date: Fri Dec 5 00:58:04 2003 Subject: [Python-Dev] release23-maint tests Message-ID: I didn't see a catagory for tests on source forge so I'm reporting here. I pulled the release23-maint branch this morning and built on UnixWare 7.1.1. ----------------- test_dl skipped -- Could not open any shared libraries It does have shared libraries, just doesn't have /usr/lib/libc.so. Adding /usr/lib/libc.so.1 to the list will fix it. ----------------- test_largefile skipped -- filesystem does not have largefile support UnixWare 7.0.0 and later does have largefile suport. The config.log shows configure:9741: checking whether to enable large file support configure:9751: result: yes 12G of free space to play with. Any tips on how to track this down would be apreciated. ----------------- test test_resource produced unexpected output: ********************************************************************** *** mismatch between line 2 of expected output and line 2 of actual output: - True + False ********************************************************************** Tips on tracking this down? test test_tarfile crashed -- exceptions.IOError: [Errno 30] Read-only file system: '/opt/src/utils/python/python-2.3/src/Lib/test/testtar.tar.gz' Some of us with multiple platforms build from a single read only source tree. ----------------- test_tempfile Exception exceptions.AttributeError: "mkstemped instance has no attribute 'fd'" in > ignored test test_tempfile failed -- Traceback (most recent call last): File "/opt/src/utils/python/python-2.3/src/Lib/test/test_tempfile.py", line 246, in test_basic_many File "/opt/src/utils/python/python-2.3/src/Lib/test/test_tempfile.py", line 229, in do_create File "/opt/src/utils/python/python-2.3/src/Lib/test/test_tempfile.py", line 45, in failOnException File "/opt/src/utils/python/python-2.3/src/Lib/unittest.py", line 270, in fail AssertionError: _mkstemp_inner raised exceptions.OSError: [Errno 24] Too many open files: '/tmp/aakOvEqF' limits.h shows 60 for OPEN_MAX. Lowering TEST_FILES to 55 in test_tempfile.py makes it pass. Maybe you just want to add openunix8 and unixware7 to if sys.platform == 'mac': -- Tim Rice Multitalents (707) 887-1469 tim@multitalents.net From Jack.Jansen at cwi.nl Fri Dec 5 05:13:35 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Fri Dec 5 05:13:38 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <3FCFE4E3.4000602@gradient.cis.upenn.edu> References: <3FCFE4E3.4000602@gradient.cis.upenn.edu> Message-ID: On 5-dec-03, at 2:52, Edward Loper wrote: >>> If you agree with this premise, then it suggests a different >>> approach. Use different implementations in C, but have type(x) in >>> Python lie. type(x) would always return the type object which is >>> now known as "long". >> If this can be made to work, I like it. __class__ should also be >> hacked, and isinstance(); and who knows how many other places, but >> probably not too many. > > If we go this route, then how general should the mechanism for > "merging" types in Python-space (but keeping them separate in c-space) > be? I.e., should this be an integer-specific change, or a general > mechanism that happens to be used by int? It seems like there might > be other use cases for similar "merges" (string/unicode comes to > mind). I think this is a good idea, I may have some uses for it too. But, more importantly, making this a general mechanism will force us to formulate the exact set of restrictions we place on two such types. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From mwh at python.net Fri Dec 5 06:04:53 2003 From: mwh at python.net (Michael Hudson) Date: Fri Dec 5 06:04:56 2003 Subject: [Python-Dev] are CObjects inherently unsafe? In-Reply-To: ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "04 Dec 2003 20:37:07 +0100") References: <78CB3A69-2669-11D8-9361-0003931DF95C@python.net> Message-ID: <2mk75bwmne.fsf@starship.python.net> martin@v.loewis.de (Martin v. L?wis) writes: > Michael Hudson writes: > >> It seems to me that it would be more sensible to have a dict mapping >> names (maybe just module names...) to cobjects in the interpreter >> state. I guess there might be a way of getting ahold of the dict with >> the gc module (though I can't think of one off hand). This wouldn't >> be a difficult change, and if the CObjects are left in the module >> dicts, it shouldn't even do that much damage to binary compatibility. >> >> Thoughts? > > Good idea. Alternatively, we could make "APIs" a feature of modules: > PyModule_GetAPI, PyModule_SetAPI. We would then define > > typedef struct { > PyObject_HEAD > PyObject *md_dict; > PyObject *md_api; > } PyModuleObject; > > We could restrict md_api to CObjects, which, means we would not need > to change module_traverse. I think you can still break this, though: >>> import _curses >>> import sys >>> sys.modules['cStringIO'] = _curses (doesn't break today because cStringIO and _curses use different attribute names for the CObject). Cheers, mwh -- 31. Simplicity does not precede complexity, but follows it. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html From mwh at python.net Fri Dec 5 06:06:53 2003 From: mwh at python.net (Michael Hudson) Date: Fri Dec 5 06:07:02 2003 Subject: [Python-Dev] are CObjects inherently unsafe? In-Reply-To: <200312050056.hB50uh722511@oma.cosc.canterbury.ac.nz> (Greg Ewing's message of "Fri, 05 Dec 2003 13:56:43 +1300 (NZDT)") References: <200312050056.hB50uh722511@oma.cosc.canterbury.ac.nz> Message-ID: <2mfzfzwmk2.fsf@starship.python.net> Greg Ewing writes: > Michael Hudson : > >> It seems to me that any time there are more than two CObjects around we >> have a good chance of causing mischief: > > Indeed, CObjects seem fundamentally dangerous to me, unless > the modules which create and use them are extremely careful > to make sure they can tell whether they've got the right > kind of CObject. Well, the right *CObject* -- it's CObject identity that matters. > And I suspect that many modules which make use of CObjects > aren't that careful. CStringIO obviously isn't... As things stand, I don't think cPickle *can* be careful enough. Perhaps we could just add a "name" field to CObjects and have module init functions check that. Cheers, mwh -- (Of course SML does have its weaknesses, but by comparison, a discussion of C++'s strengths and flaws always sounds like an argument about whether one should face north or east when one is sacrificing one's goat to the rain god.) -- Thant Tessman From mwh at python.net Fri Dec 5 06:11:00 2003 From: mwh at python.net (Michael Hudson) Date: Fri Dec 5 06:11:03 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312042040.hB4Ke8D06619@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Thu, 04 Dec 2003 12:40:08 -0800") References: <1070546177.3fcf3d01e6f52@mcherm.com> <200312042040.hB4Ke8D06619@c-24-5-183-134.client.comcast.net> Message-ID: <2mbrqnwmd7.fsf@starship.python.net> Guido van Rossum writes: >> If you agree with this premise, then it suggests a different >> approach. Use different implementations in C, but have type(x) in >> Python lie. type(x) would always return the type object which is >> now known as "long". > > If this can be made to work, I like it. I think Armin is ill at the moment, but you're all singing his tune :-) (He has some very similar sounding ideas for PyPy). Cheers, mwh -- If Unicode is a horde of zombies with flaming dung sticks, the hideous intricacies of JIS, Chinese Big-5, Chinese Traditional, KOI-8, et cetera are at least an army of ogres with salt and flensing knives. -- Eric S. Raymond, python-dev From ajsiegel at optonline.net Fri Dec 5 08:28:06 2003 From: ajsiegel at optonline.net (Arthur) Date: Fri Dec 5 08:24:22 2003 Subject: [Python-Dev] re: Tutorial: Brief Introduction to the Standard Message-ID: <0HPF00AVIBWJ3Q@mta5.srv.hcvlny.cv.net> >The opposite question, "What is fundamentally wrong with list[:]..." has >an easy answer: >It takes experience or explanation of slicing to know what it does. >copy(list) is easy even for novices to understand. I agree. But think the problem is even deeper than that. Because it all impacts, in the end, understanding what = does. As Guido says, a favorite subject of mine. Art From Jack.Jansen at cwi.nl Fri Dec 5 08:52:38 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Fri Dec 5 08:52:36 2003 Subject: [Python-Dev] 2.3 maint: test_poll fails on OSX Message-ID: <49E8102C-272A-11D8-989C-000A27B19B96@cwi.nl> With the current state of the 2.3 maintenance branch test_poll fails: File "/Volumes/Vuur/Users/jack/src/python-23/Lib/test/test_poll.py", line 171, in ? test_poll1() File "/Volumes/Vuur/Users/jack/src/python-23/Lib/test/test_poll.py", line 65, in test_poll1 poll_unit_tests() File "/Volumes/Vuur/Users/jack/src/python-23/Lib/test/test_poll.py", line 77, in poll_unit_tests r = p.poll() error: (9, 'Bad file descriptor') This also looks legitimate to me: the test goes to great lengths to make sure it's passing an invalid file descriptor to poll(). But I assume there's something deeper going on that I don't understand... -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From anthony at python.org Fri Dec 5 09:40:53 2003 From: anthony at python.org (Anthony Baxter) Date: Fri Dec 5 09:41:21 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) Message-ID: <200312051440.hB5Eer1P009275@maxim.off.ekorp.com> On behalf of the Python development team and the Python community, I'm happy to announce the release of Python 2.3.3 (release candidate 1). Python 2.3.3 is a bug-fix release of Python 2.3. A couple of serious bugs related to weakrefs and the cyclic garbage collector have been fixed, along with a number of bugs in the standard library. See the release notes on the web page for more details. For more information on Python 2.3.3c1, including download links for various platforms, release notes, and known issues, please see http://www.python.org/2.3.3 Highlights of this new release include: - A couple of serious bugs in the interactions of weakrefs and cyclic GC have been squashed. - At shutdown, the second call to the cyclic garbage collector has been removed. This caused more problems than it solved. - The xml.parsers.expat module now provides Expat 1.95.7. - urllib2's HTTP Digest Auth support works again. - See http://www.python.org/2.3.3/NEWS.html for other bugfixes. Highlights of the previous major Python release (2.3) are available from the Python 2.3 page, at http://www.python.org/2.3/highlights.html A final version of Python 2.3.3 should follow in a couple of weeks, just in time for your Christmas stockings. Enjoy the new release, Anthony Anthony Baxter anthony@python.org Python 2.3.x Release Manager (on behalf of the entire python-dev team) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 226 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20031206/5d033837/attachment.bin From guido at python.org Fri Dec 5 09:41:20 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 5 09:41:25 2003 Subject: [Python-Dev] are CObjects inherently unsafe? In-Reply-To: Your message of "Fri, 05 Dec 2003 11:06:53 GMT." <2mfzfzwmk2.fsf@starship.python.net> References: <200312050056.hB50uh722511@oma.cosc.canterbury.ac.nz> <2mfzfzwmk2.fsf@starship.python.net> Message-ID: <200312051441.hB5EfKv08101@c-24-5-183-134.client.comcast.net> > > Indeed, CObjects seem fundamentally dangerous to me, unless > > the modules which create and use them are extremely careful > > to make sure they can tell whether they've got the right > > kind of CObject. >> > Well, the right *CObject* -- it's CObject identity that matters. I seem to recall that the CObject design contains a feature to avoid using the wrong CObject, but it's not used by any of the CObject implementations in the standard library, and the CObject docs don't describe how it's meant to use (they were clearly reverse-engineered from the code). Several early checkin messages of cobject.c mention Jim Fulton as the author, so maybe we could ask him. (I'm cc'ing Jim in case he remembers the true intention for the description; he might also know whether Zope 2 still uses it; I can only check Zope 3, and it doesn't.) But even without Jim's confirmation, ISTM from memory and looking at the source that the 'desc' field (which stands for description) was intended as a clue that could be checked by the user of a C Object to verify it had gotten the right one: in a shared header (needed anyway to describe the layout of the struct pointer contained in the C Object) you could define a magic word or a magic string value that would be stored as the description, and the users would check that value before believing the struct pointer. Unfortunately the same CObject version that introduced the description field also introduced the convenience function PyCObject_Import(), which doesn't return the description, and returns the void pointer contained in the CObject without returning the CObject itself, leaving its caller without a clue about the description... Anyway, I think it can be fixed by starting to use the description field and adding a new convenience C API, perhaps named PyCObject_ImportEx(), whch takes a description and checks it. We should try to standardize on what kind of thing to use as the description -- if we use a magic word, PyCObject_ImportEx() should do a simple compare, but if we use a string, it should do a strcmp(). Or the API could simply return the description into an output parameter, making the caller responsible for doing the checking. And the docs should be updated to explain the description better (currently at one point the description is called "extra callback data for the destructor function" which seems a rather odd feature). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Dec 5 10:09:21 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 5 10:09:27 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: Your message of "Fri, 05 Dec 2003 00:57:33 EST." <20031205055733.GA12283@panix.com> References: <1070546177.3fcf3d01e6f52@mcherm.com> <20031205055733.GA12283@panix.com> Message-ID: <200312051509.hB5F9Lr08164@c-24-5-183-134.client.comcast.net> > > To my mind, the *ideal* is that there's a single type, probably > > called "int" or "integer" which can take on an arbitrary size and > > precision BUT which, if it happens to be small enough to fit in > > a C int will take up less memory and run faster. The fact that > > it's stored differently should be an implementation detail not > > visible to the user. > > Hmmmm... How important is the "less memory" angle versus the "faster" > angle? Why not just add a few bytes to the long type to store an int? > If you're using the int, the pointer is NULL, and you're saving that > memory, at least. The only harm is some extra memory in the type > object. I'd say that the "less memory" angle is pretty important. The int implementation has been using every trick in the book to save memory from very early days on: it has the most sophisticated special-purpose allocator, *and* it uses a cache for popular values, *and* you really can't squeeze any more bits out of the structure. --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh at python.net Fri Dec 5 10:12:45 2003 From: mwh at python.net (Michael Hudson) Date: Fri Dec 5 10:12:49 2003 Subject: [Python-Dev] are CObjects inherently unsafe? In-Reply-To: <200312051441.hB5EfKv08101@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Fri, 05 Dec 2003 06:41:20 -0800") References: <200312050056.hB50uh722511@oma.cosc.canterbury.ac.nz> <2mfzfzwmk2.fsf@starship.python.net> <200312051441.hB5EfKv08101@c-24-5-183-134.client.comcast.net> Message-ID: <2m7k1bwb6a.fsf@starship.python.net> Guido van Rossum writes: >> > Indeed, CObjects seem fundamentally dangerous to me, unless >> > the modules which create and use them are extremely careful >> > to make sure they can tell whether they've got the right >> > kind of CObject. >>> >> Well, the right *CObject* -- it's CObject identity that matters. > > I seem to recall that the CObject design contains a feature to avoid > using the wrong CObject, but it's not used by any of the CObject > implementations in the standard library, and the CObject docs don't > describe how it's meant to use (they were clearly reverse-engineered > from the code). [...] > And the docs should be updated to explain the description better > (currently at one point the description is called "extra callback data > for the destructor function" which seems a rather odd feature). It had occurred to me that you could use this field for this purpose, but it hadn't occurred to me that this might have been the *intent* of this field. Why a void* then? That's particularly unassertive. char* would be better, surely... Cheers, mwh -- I also fondly recall Paris because that's where I learned to debug Zetalisp while drunk. -- Olin Shivers From theller at python.net Fri Dec 5 10:17:50 2003 From: theller at python.net (Thomas Heller) Date: Fri Dec 5 10:17:59 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312051509.hB5F9Lr08164@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Fri, 05 Dec 2003 07:09:21 -0800") References: <1070546177.3fcf3d01e6f52@mcherm.com> <20031205055733.GA12283@panix.com> <200312051509.hB5F9Lr08164@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: >> Hmmmm... How important is the "less memory" angle versus the "faster" >> angle? Why not just add a few bytes to the long type to store an int? >> If you're using the int, the pointer is NULL, and you're saving that >> memory, at least. The only harm is some extra memory in the type >> object. > > I'd say that the "less memory" angle is pretty important. The int > implementation has been using every trick in the book to save memory > from very early days on: it has the most sophisticated special-purpose > allocator, *and* it uses a cache for popular values, *and* you really > can't squeeze any more bits out of the structure. IIUC, doing range(10000000) in a program somewhere allocates 10 million integers, and these are *never* freed, they live in the cache forever. Correct? I do not have an actual use case for this, but it seems somewhat strange. And the point comes up from time to time on c.l.p. Thomas From anthony at ekit-inc.com Fri Dec 5 10:21:26 2003 From: anthony at ekit-inc.com (Anthony Baxter) Date: Fri Dec 5 10:22:23 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: Message from Thomas Heller of "Fri, 05 Dec 2003 16:17:50 BST." Message-ID: <200312051521.hB5FLRmr009991@maxim.off.ekorp.com> >>> Thomas Heller wrote > Guido van Rossum writes: > IIUC, doing range(10000000) in a program somewhere allocates 10 million > integers, and these are *never* freed, they live in the cache forever. > Correct? My understanding was that it was only ints between -5 and +100 that were cached forever. From guido at python.org Fri Dec 5 10:22:42 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 5 10:23:18 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: Your message of "Thu, 04 Dec 2003 21:18:08 PST." References: Message-ID: <200312051522.hB5FMgQ08216@c-24-5-183-134.client.comcast.net> > Greg Ewing's proposal of a "given" keyword (x.score given x) got me > thinking. I figured I would play around a bit and try to come up with > the most readable version of the original "groupby" idea (for which I > could imagine *some* implementation): > > for group in sequence.groups(using item.score - item.penalty): > ...do stuff with group > > Having written this down, it seems to me the most readable so far. The > keyword "using" creates a new scope, within which "item" is bound to the > arg (or *args?) passed in. I don't know about you all, but the thing I > like least about lambda is having to mention 'x' twice: > > lambda x: x.score > > Why have the programmer bind a custom name to an object we're going to > then use 'anonymously' anyway? I understand its historical necessity, > but it's always struck me as more complex than the concept being > implemented. Ideally, we should be able to reference the passed-in > objects without having to invent names for them. Huh? How can you reference something without having a name for it? Are you proposing to add pronouns to Python? > Now, consider multi-arg lambdas such as: > > sequence.sort(lambda x, y: cmp(x[0], y[0])) > > In these cases, we wish to apply the same operation to each item (that > is, we calculate x[0] and y[0]). If we bind "item" to each argument *in > turn*, we save a lot of syntax. The above might then be written as: > sequence.sort(using cmp(item[0])) # Hard to implement. > > or: > sequence.sort(cmp(using item[0])) # Easier but ugly. Meh. > > or: > sequence.sort(cmp using item[0]) # Oooh. Nice. :) > > or: > # might we assume cmp(), since sort does...? > sequence.sort(using item[0]) > > I like #3, since cmp is explicit but doesn't use cmp(), which looks too > much like a call. Given (cmp using item[0]), the "using block" would > look at the arguments supplied by sort(), call __getitem__[0] for each, > and pass those values in order into cmp, returning the result. There are lots of situations where the comparison lambda is just a bit more complex than this, for example: lambda x, y: cmp(x[0], y[0]) or cmp(x[1], y[1]) And how would you spell lambda x, y: x+y? "+ using item"??? That becomes a syntactical nightmare. (Or what about lambda x, y: 2*(x+y)?) I also think you are cheating by using sort() as the example -- other examples of multi-argument lambdas aren't necessarily so uniform in the arguments. > The "item" keyword functions similarly to Guido's Voodoo.foo() proposal, > now that I think about it. There's no reason it couldn't grow some early > binding, either, as suggested, although multiple operations would become > unwieldy. How would you early-bind this? > > sequence.groups(using divmod(item, 4)[1]) > > ...except perhaps by using multiply-nested scopes to bind the "1" and > then the "4"? I see all sorts of problems with this, but early-binding "1" and "4" aren't amongst them -- early binding only applies to free variables, not to constants. > Hmm. It would have to do some fancy dancing to get everything in the > right order. Too much like reinventing Python to think about at the > moment. :) The point is, passing the "item" instance through such a > scheme should be the easy part. I've read this whole post twice, and I still don't understand what you're really proposing (or how it could ever work given the rest of Python), so I think it's probably not a good idea from a readability perspective... --Guido van Rossum (home page: http://www.python.org/~guido/) From anthony at interlink.com.au Fri Dec 5 10:23:19 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Dec 5 10:23:30 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: Message from Anthony Baxter of "Fri, 05 Dec 2003 00:41:21 +1100." <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> Message-ID: <200312051523.hB5FNJTU010018@maxim.off.ekorp.com> After a short chat with Thomas, we'd like to pencil in the 19th for the final release (so two weeks from the RC1). Does this cause anyone pain? ta, Anthony From guido at python.org Fri Dec 5 10:26:24 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 5 10:26:32 2003 Subject: [Python-Dev] are CObjects inherently unsafe? In-Reply-To: Your message of "Fri, 05 Dec 2003 15:12:45 GMT." <2m7k1bwb6a.fsf@starship.python.net> References: <200312050056.hB50uh722511@oma.cosc.canterbury.ac.nz> <2mfzfzwmk2.fsf@starship.python.net> <200312051441.hB5EfKv08101@c-24-5-183-134.client.comcast.net> <2m7k1bwb6a.fsf@starship.python.net> Message-ID: <200312051526.hB5FQO908272@c-24-5-183-134.client.comcast.net> > It had occurred to me that you could use this field for this purpose, > but it hadn't occurred to me that this might have been the *intent* of > this field. > > Why a void* then? That's particularly unassertive. char* would be > better, surely... Jim only knows... Probably a case of mistaken generality. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Dec 5 10:29:42 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 5 10:29:48 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: Your message of "Fri, 05 Dec 2003 16:17:50 +0100." References: <1070546177.3fcf3d01e6f52@mcherm.com> <20031205055733.GA12283@panix.com> <200312051509.hB5F9Lr08164@c-24-5-183-134.client.comcast.net> Message-ID: <200312051529.hB5FTgq08289@c-24-5-183-134.client.comcast.net> > IIUC, doing range(10000000) in a program somewhere allocates 10 million > integers, and these are *never* freed, they live in the cache forever. > Correct? Wrong. The memory allocated can never be used for other objects than ints, but it will be reused for other ints. IOW if you do for i in range(100): range(i*1000000, (i+1)*1000000) you only allocate space for 1000000 ints, not for 100 times 1000000 ints. (The range(100) OTOH are all in a cache that make the objects indestructible, but this only applies to values in [-5, 100). --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake at acm.org Fri Dec 5 10:30:15 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Dec 5 10:30:43 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: <200312051523.hB5FNJTU010018@maxim.off.ekorp.com> References: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> <200312051523.hB5FNJTU010018@maxim.off.ekorp.com> Message-ID: <16336.42119.508562.747938@grendel.fdrake.net> Anthony Baxter writes: > After a short chat with Thomas, we'd like to pencil in the 19th for > the final release (so two weeks from the RC1). Does this cause anyone > pain? Someone, somewhere, of course. Everything does! +1 -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From recht at netbsd.org Fri Dec 5 10:35:18 2003 From: recht at netbsd.org (Marc Recht) Date: Fri Dec 5 10:35:25 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: <200312051523.hB5FNJTU010018@maxim.off.ekorp.com> References: <200312051523.hB5FNJTU010018@maxim.off.ekorp.com> Message-ID: <16900000.1070638518@leeloo.intern.geht.de> > After a short chat with Thomas, we'd like to pencil in the 19th for > the final release (so two weeks from the RC1). Does this cause anyone > pain? Any change to get the patches #854796 and/or #806800 in ? Cheers, Marc -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 186 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20031205/bf385417/attachment.bin From theller at python.net Fri Dec 5 10:37:10 2003 From: theller at python.net (Thomas Heller) Date: Fri Dec 5 10:37:24 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312051529.hB5FTgq08289@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Fri, 05 Dec 2003 07:29:42 -0800") References: <1070546177.3fcf3d01e6f52@mcherm.com> <20031205055733.GA12283@panix.com> <200312051509.hB5F9Lr08164@c-24-5-183-134.client.comcast.net> <200312051529.hB5FTgq08289@c-24-5-183-134.client.comcast.net> Message-ID: <4qwfuvh5.fsf@python.net> Guido van Rossum writes: >> IIUC, doing range(10000000) in a program somewhere allocates 10 million >> integers, and these are *never* freed, they live in the cache forever. >> Correct? > > Wrong. The memory allocated can never be used for other objects than > ints, but it will be reused for other ints. IOW if you do > > for i in range(100): > range(i*1000000, (i+1)*1000000) > > you only allocate space for 1000000 ints, not for 100 times 1000000 > ints. (The range(100) OTOH are all in a cache that make the objects > indestructible, but this only applies to values in [-5, 100). Ok, then it makes sense. Thanks for the clarification. Thomas From tim.one at comcast.net Fri Dec 5 10:50:58 2003 From: tim.one at comcast.net (Tim Peters) Date: Fri Dec 5 10:50:59 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312051521.hB5FLRmr009991@maxim.off.ekorp.com> Message-ID: [maybe Thomas Heller] >> IIUC, doing range(10000000) in a program somewhere allocates 10 >> million integers, and these are *never* freed, they live in the >> cache forever. Correct? [Anthony Baxter] > My understanding was that it was only ints between -5 and +100 that > were cached forever. Tiny ints are *shared*: no matter how many computations return, and data structures hold, the integer 3, they all point to the same integer object. Space for all ints (tiny or not) is taken from a dedicated freelist, used for ints and nothing but ints. Whenever an int is deallocated, its memory is added to this dedicated freelist. Unlike most dedicated freelists in Python, the int freelist is unbounded in size. So when you do range(10000000), and that list goes away, most of the space for the int objects it held ends up in millions of int objects pushed onto the int freelist. They all stay there until the program shuts down (although they get reused to satisfy requests for new int objects). floats have a very similar (unbounded and immortal) dedicated freelist, so, e.g., [abs(0.5) for i in xrange(10000000)] reserves space for 10 million float objects forever after (although [0.5 for i in xrange(10000000)] only creates 1 float object -- as is obvious to the most casual observer ). From fdrake at acm.org Fri Dec 5 11:11:00 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Dec 5 11:11:42 2003 Subject: [Python-Dev] source-level backward compatibility requirements In-Reply-To: <16900000.1070638518@leeloo.intern.geht.de> References: <200312051523.hB5FNJTU010018@maxim.off.ekorp.com> <16900000.1070638518@leeloo.intern.geht.de> Message-ID: <16336.44564.604153.644072@grendel.fdrake.net> Marc Recht writes: > Any change to get the patches #854796 and/or #806800 in ? It's hard to make changes once the release candidate is out. ;-( I still think the use of apply() in the logging module needs to be fixed. Here's how I think it should be done, retaining source compatibility for Python back to 1.5.2: Add a new module, logging._apply, that defines apply() in terms of the *args, **kw syntax. The modules that use apply, "logging" and "logging.config", would each contain these two lines near the top: if sys.version_info >= (2, 3): from logging._apply import apply That would allow a compatible version of apply() to be used for newer Pythons without triggering the deprecation warning. It would also minimize the changes to the package. Regarding the compatibility requirements in PEP 291, I think we need to decide what in CVS those apply to: only the trunk, or also to the maintenance branches. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From guido at python.org Fri Dec 5 11:15:57 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 5 11:16:18 2003 Subject: [Python-Dev] source-level backward compatibility requirements In-Reply-To: Your message of "Fri, 05 Dec 2003 11:11:00 EST." <16336.44564.604153.644072@grendel.fdrake.net> References: <200312051523.hB5FNJTU010018@maxim.off.ekorp.com> <16900000.1070638518@leeloo.intern.geht.de> <16336.44564.604153.644072@grendel.fdrake.net> Message-ID: <200312051615.hB5GFvW08552@c-24-5-183-134.client.comcast.net> > I still think the use of apply() in the logging module needs to be > fixed. Here's how I think it should be done, retaining source > compatibility for Python back to 1.5.2: > > Add a new module, logging._apply, that defines apply() in terms of the > *args, **kw syntax. The modules that use apply, "logging" and > "logging.config", would each contain these two lines near the top: > > if sys.version_info >= (2, 3): > from logging._apply import apply > > That would allow a compatible version of apply() to be used for newer > Pythons without triggering the deprecation warning. It would also > minimize the changes to the package. Unfortunately, this would make apply even slower than the system version with the silent deprecation warning. :-( I'd rather drop 1.5.2 compatibility for the logging module *as it is in CVS*. 1.5.2 users can continue to use the last version that was distributed by its author independently. --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake at acm.org Fri Dec 5 11:22:06 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Dec 5 11:22:46 2003 Subject: [Python-Dev] source-level backward compatibility requirements In-Reply-To: <200312051615.hB5GFvW08552@c-24-5-183-134.client.comcast.net> References: <200312051523.hB5FNJTU010018@maxim.off.ekorp.com> <16900000.1070638518@leeloo.intern.geht.de> <16336.44564.604153.644072@grendel.fdrake.net> <200312051615.hB5GFvW08552@c-24-5-183-134.client.comcast.net> Message-ID: <16336.45230.749132.305648@grendel.fdrake.net> Guido van Rossum writes: > I'd rather drop 1.5.2 compatibility for the logging module *as it is > in CVS*. 1.5.2 users can continue to use the last version that was > distributed by its author independently. So would I, and this was what I originally proposed (removing the use of apply() altogether). There was some objection to this, mostly based on PEP 291. It wasn't clear how many people actually cared about this case, and I don't recall seeing Vinay chime in. On the other hand, I find I'm not really a fan of deprecating apply() in the first place, so I can think of other solutions. ;-) -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From barry at python.org Fri Dec 5 11:46:17 2003 From: barry at python.org (Barry Warsaw) Date: Fri Dec 5 11:46:22 2003 Subject: [Python-Dev] source-level backward compatibility requirements In-Reply-To: <200312051615.hB5GFvW08552@c-24-5-183-134.client.comcast.net> References: <200312051523.hB5FNJTU010018@maxim.off.ekorp.com> <16900000.1070638518@leeloo.intern.geht.de> <16336.44564.604153.644072@grendel.fdrake.net> <200312051615.hB5GFvW08552@c-24-5-183-134.client.comcast.net> Message-ID: <1070642777.1377.22.camel@anthem> On Fri, 2003-12-05 at 11:15, Guido van Rossum wrote: > I'd rather drop 1.5.2 compatibility for the logging module *as it is > in CVS*. 1.5.2 users can continue to use the last version that was > distributed by its author independently. +1 I'd suggest updating PEP 291 in a way similar to what I did yesterday for the email package. -Barry From guido at python.org Fri Dec 5 11:54:09 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 5 11:54:14 2003 Subject: [Python-Dev] source-level backward compatibility requirements In-Reply-To: Your message of "Fri, 05 Dec 2003 11:22:06 EST." <16336.45230.749132.305648@grendel.fdrake.net> References: <200312051523.hB5FNJTU010018@maxim.off.ekorp.com> <16900000.1070638518@leeloo.intern.geht.de> <16336.44564.604153.644072@grendel.fdrake.net> <200312051615.hB5GFvW08552@c-24-5-183-134.client.comcast.net> <16336.45230.749132.305648@grendel.fdrake.net> Message-ID: <200312051654.hB5Gs9e08643@c-24-5-183-134.client.comcast.net> > On the other hand, I find I'm not really a fan of deprecating apply() > in the first place, so I can think of other solutions. ;-) We could deprecate it in the docs but remove the silent warning which slows it down. Then we could leave it in the logging module until 3.0. --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake at acm.org Fri Dec 5 11:58:25 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Dec 5 11:58:36 2003 Subject: [Python-Dev] source-level backward compatibility requirements In-Reply-To: <200312051654.hB5Gs9e08643@c-24-5-183-134.client.comcast.net> References: <200312051523.hB5FNJTU010018@maxim.off.ekorp.com> <16900000.1070638518@leeloo.intern.geht.de> <16336.44564.604153.644072@grendel.fdrake.net> <200312051615.hB5GFvW08552@c-24-5-183-134.client.comcast.net> <16336.45230.749132.305648@grendel.fdrake.net> <200312051654.hB5Gs9e08643@c-24-5-183-134.client.comcast.net> Message-ID: <16336.47409.107855.751131@grendel.fdrake.net> Guido van Rossum writes: > We could deprecate it in the docs but remove the silent warning which > slows it down. Then we could leave it in the logging module until > 3.0. +42 Does this warrant a c2 release? I'm happy to make the changes right now. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tim.one at comcast.net Fri Dec 5 12:12:38 2003 From: tim.one at comcast.net (Tim Peters) Date: Fri Dec 5 12:12:41 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: Message-ID: [Tim] > ... > So when you do range(10000000), and that list goes away, most of the > space for the int objects it held ends up in millions of int objects > pushed onto the int freelist. They all stay there until the program > shuts down (although they get reused to satisfy requests for new int > objects). BTW, using pymalloc instead wouldn't eliminate that allocated memory is never returned to the OS. pymalloc grabs large "arenas" from malloc(), and never free()s them. The fundamental difference is that pymalloc is type-agnostic: space gotten from pymalloc for, say, a dict, can later be reused for, say, a string. pymalloc's freelists are segregated by raw memory size rather than by type, and a given chunk of pymalloc memory may even move from one size list to another (any number of times). From guido at python.org Fri Dec 5 12:19:11 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 5 12:20:19 2003 Subject: [Python-Dev] source-level backward compatibility requirements In-Reply-To: Your message of "Fri, 05 Dec 2003 11:58:25 EST." <16336.47409.107855.751131@grendel.fdrake.net> References: <200312051523.hB5FNJTU010018@maxim.off.ekorp.com> <16900000.1070638518@leeloo.intern.geht.de> <16336.44564.604153.644072@grendel.fdrake.net> <200312051615.hB5GFvW08552@c-24-5-183-134.client.comcast.net> <16336.45230.749132.305648@grendel.fdrake.net> <200312051654.hB5Gs9e08643@c-24-5-183-134.client.comcast.net> <16336.47409.107855.751131@grendel.fdrake.net> Message-ID: <200312051719.hB5HJBp08708@c-24-5-183-134.client.comcast.net> > Guido van Rossum writes: > > We could deprecate it in the docs but remove the silent warning which > > slows it down. Then we could leave it in the logging module until > > 3.0. > > +42 > > Does this warrant a c2 release? I'm happy to make the changes right > now. > > -Fred Tht's a question for Anthony. I suppose there will be a c2 release anyway. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Dec 5 12:26:09 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 5 12:26:14 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: Your message of "Fri, 05 Dec 2003 12:12:38 EST." References: Message-ID: <200312051726.hB5HQ9Y08752@c-24-5-183-134.client.comcast.net> > BTW, using pymalloc instead wouldn't eliminate that allocated memory is > never returned to the OS. pymalloc grabs large "arenas" from malloc(), and > never free()s them. The fundamental difference is that pymalloc is > type-agnostic: space gotten from pymalloc for, say, a dict, can later be > reused for, say, a string. pymalloc's freelists are segregated by raw > memory size rather than by type, and a given chunk of pymalloc memory may > even move from one size list to another (any number of times). But there aren't a lot of other objects that fit in the size of an int, right? I know of no other types that are <= 12 bytes on a 32-bit machine, and assuming pymalloc rounds up, only one that is <= 16 (float). There may be some obscure ones (super may be only 16 bytes too) but they'll never need that many instances to fill the void left behind by range(10000000). (I'm assuming that pymalloc doesn't join adjacent small blocks to form a larger block -- that way lies a full malloc reimplementation, and what's the point of that...) --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Fri Dec 5 12:30:00 2003 From: aahz at pythoncraft.com (Aahz) Date: Fri Dec 5 12:30:03 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312051509.hB5F9Lr08164@c-24-5-183-134.client.comcast.net> References: <1070546177.3fcf3d01e6f52@mcherm.com> <20031205055733.GA12283@panix.com> <200312051509.hB5F9Lr08164@c-24-5-183-134.client.comcast.net> Message-ID: <20031205172959.GA1278@panix.com> On Fri, Dec 05, 2003, Guido van Rossum wrote: >Aahz: >> >> Hmmmm... How important is the "less memory" angle versus the "faster" >> angle? Why not just add a few bytes to the long type to store an int? >> If you're using the int, the pointer is NULL, and you're saving that >> memory, at least. The only harm is some extra memory in the type >> object. > > I'd say that the "less memory" angle is pretty important. The int > implementation has been using every trick in the book to save memory > from very early days on: it has the most sophisticated special-purpose > allocator, *and* it uses a cache for popular values, *and* you really > can't squeeze any more bits out of the structure. Right, that's what's historically true. Question is, by the time 3.0 comes out, *should* it still be true? As pointed out downthread, we've already got a problem with the int free list; earlier discussions on python-dev have talked about ways of dealing with that (the simplest of which will be to make range() an iterator, since that's the most common problem source). Why not use some of that memory savings to make the int object easier to handle? "Simple is better than complex"; I suspect that we might have a win-win situation. Perhaps we could push some of the memory-saving techniques into the long object? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From fumanchu at amor.org Fri Dec 5 12:33:08 2003 From: fumanchu at amor.org (Robert Brewer) Date: Fri Dec 5 12:38:31 2003 Subject: [Python-Dev] Re: "groupby" iterator Message-ID: I wrote: > > Why have the programmer bind a custom name to an object > we're going to > > then use 'anonymously' anyway? I understand its historical > necessity, > > but it's always struck me as more complex than the concept being > > implemented. Ideally, we should be able to reference the passed-in > > objects without having to invent names for them. And Guido van Rossum answered: > Huh? How can you reference something without having a name for it? > Are you proposing to add pronouns to Python? That's one way of looking at it. In a sense, names are already personal pronouns with mutable referents. I'm thinking about a language (and maybe Python wouldn't be a good candidate) where it also has standard impersonal pronouns like "it" and "they" in English. But in an already-restricted context, that is, lambda. > There are lots of situations where the comparison lambda is just a bit > more complex than this, for example: > > lambda x, y: cmp(x[0], y[0]) or cmp(x[1], y[1]) > > And how would you spell lambda x, y: x+y? "+ using item"??? That > becomes a syntactical nightmare. (Or what about lambda x, y: > 2*(x+y)?) Good questions. > I've read this whole post twice, and I still don't understand what > you're really proposing (or how it could ever work given the rest of > Python), so I think it's probably not a good idea from a readability > perspective... That's fine. I'm really just musing; I'll refrain from that in the future on this list unless I have code. ;) I work far too intuitively sometimes. Maybe in the Spring, when I'll have time to dig into internals and form a sample implementation. Thanks for your comments. I *do* think this thread needs to get back to, "are we going to have 'groupby' in 2.4 or later?" Robert Brewer MIS Amor Ministries fumanchu@amor.org From guido at python.org Fri Dec 5 12:45:31 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 5 12:46:07 2003 Subject: [Python-Dev] Re: "groupby" iterator In-Reply-To: Your message of "Fri, 05 Dec 2003 09:33:08 PST." References: Message-ID: <200312051745.hB5HjVG08871@c-24-5-183-134.client.comcast.net> > I *do* think this thread needs to get back to, "are we going to have > 'groupby' in 2.4 or later?" AFAIC that's decided. We have received a high-quality implementation; Raymond just has to find the time to check it in. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Fri Dec 5 13:05:39 2003 From: tim.one at comcast.net (Tim Peters) Date: Fri Dec 5 13:05:41 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <200312051726.hB5HQ9Y08752@c-24-5-183-134.client.comcast.net> Message-ID: [Tim] >> BTW, using pymalloc instead wouldn't eliminate that allocated memory >> is never returned to the OS. pymalloc grabs large "arenas" from >> malloc(), and never free()s them. The fundamental difference is >> that pymalloc is type-agnostic: space gotten from pymalloc for, >> say, a dict, can later be reused for, say, a string. pymalloc's >> freelists are segregated by raw memory size rather than by type, and >> a given chunk of pymalloc memory may even move from one size list to >> another (any number of times). [Guido] > But there aren't a lot of other objects that fit in the size of an > int, right? That's true, but doesn't matter . > I know of no other types that are <= 12 bytes on a 32-bit machine, and > assuming pymalloc rounds up, only one that is <= 16 (float). There > may be some obscure ones (super may be only 16 bytes too) but they'll > never need that many instances to fill the void left behind by range > (10000000). > > (I'm assuming that pymalloc doesn't join adjacent small blocks to form > a larger block That's the part that's missing -- it does, but cleverly and efficiently. > -- that way lies a full malloc reimplementation, and what's the > point of that...) Large "arenas" are cut into many equal-sized "pools", and pools are in turn cut into equal-size objects. All objects within a given pool have the same size. When all the objects in a given pool are freed, the entire pool is available for reuse by objects of any size. There's a little pressure favoring reusing an entirely-free pool for objects of the same size as it was last used for, because that saves some overhead (for example, some size-dependent info is computed and stored in the pool header, and that part can be skipped if the pool is used for objects of the same size as last time). pymalloc will change the size associated with an entirely-free pool before it will ask for another arena, though. So, e.g., if ints used pymalloc, range(10000000) would carve out enough pools to hold 10 million ints, but when range(10000000) went away almost all those pools would become entirely free, and so available for reuse by objects of any size. The coalescing here is *not* done on an object-by-object basis. Instead the pool header keeps a count of the number of slots in that pool currently allocated. When an object in the pool is freed, that count is decremented. If the count becomes 0 then, the entire pool can be recycled without further ado, in constant time. If the count does not become 0 then, the newly-freed object is put at the head of a singly-linked list of free objects within the pool, which is also a constant-time operation (IOW, the objects within a pool act a lot like the dedicated int freelist; if the pool is recycled to hold objects of a different size, the freelist links within it are simply ignored and overwritten). A more-general malloc has to muck around asking whether the objects on either side of a newly-freed object are also free, so that it can coalesce them. pymalloc never does that. OTOH, pymalloc is emphatically not a general-purpose allocator: it doesn't even try to deal with requests for more than about 256 bytes (those get passed on to the system malloc). It relies (for performance) on that the vast bulk of memory requests Python makes are for small chunks of memory. Vladimir did a great job of designing this! Ignoring cache effects, the dedicated int freelist is faster. I'm not sure it's always faster if cache effects are considered too (it's complex, and pymalloc is very cache-conscious). From python at rcn.com Fri Dec 5 14:58:22 2003 From: python at rcn.com (Raymond Hettinger) Date: Fri Dec 5 14:58:41 2003 Subject: [Python-Dev] Re: "groupby" iterator References: <200312051745.hB5HjVG08871@c-24-5-183-134.client.comcast.net> Message-ID: <007801c3bb6a$2c9510c0$e841fea9@oemcomputer> > > I *do* think this thread needs to get back to, "are we going to have > > 'groupby' in 2.4 or later?" > > AFAIC that's decided. We have received a high-quality implementation; > Raymond just has to find the time to check it in. Expect it to go in this weekend :-) Raymond From python at rcn.com Fri Dec 5 15:00:24 2003 From: python at rcn.com (Raymond Hettinger) Date: Fri Dec 5 15:00:28 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python bltinmodule.c, 2.292.10.3, 2.292.10.4 References: Message-ID: <008a01c3bb6a$6bee1b40$e841fea9@oemcomputer> FYI, I have a few more doc fixes for Py2.3.3. Anthony asked me to wait until Sunday to put them in. Raymond From fdrake at acm.org Fri Dec 5 15:02:54 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Dec 5 15:03:27 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python bltinmodule.c, 2.292.10.3, 2.292.10.4 In-Reply-To: <008a01c3bb6a$6bee1b40$e841fea9@oemcomputer> References: <008a01c3bb6a$6bee1b40$e841fea9@oemcomputer> Message-ID: <16336.58478.552434.65123@grendel.fdrake.net> Raymond Hettinger writes: > FYI, I have a few more doc fixes for Py2.3.3. > Anthony asked me to wait until Sunday to put them in. Any particular reason for the delay? -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From mcherm at mcherm.com Fri Dec 5 15:14:38 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Fri Dec 5 15:14:43 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs Message-ID: <1070655278.3fd0e72e6c665@mcherm.com> I proposed: > If you agree with this premise, then it suggests a different > approach. Use different implementations in C, but have type(x) in > Python lie. type(x) would always return the type object which is > now known as "long". Greg Ewing writes: > What would happen under this scheme if someone did try > to subclass int, or long? What would isinstance() make of > such subclasses? Well, you could subclass long (probably renamed, but I'll use today's names to avoid confusion) just fine. But you couldn't subclass int because it wouldn't even be visible to a Python program. (Of course we can't stop you from trying to subclass it in C, but then you get what you deserve.) Edward Loper writes: > If we go this route, then how general should the mechanism for "merging" > types in Python-space (but keeping them separate in c-space) be? I.e., > should this be an integer-specific change, or a general mechanism that > happens to be used by int? It seems like there might be other use cases > for similar "merges" (string/unicode comes to mind). I could be wrong, but I can't think of any other cases where this would be useful. Your example of string/unicode is definitely NOT such a case: unicode is a useful type, and string is ALSO a useful type, one with slightly different behavior (eg: one is useful for binary data, the other is handy for storing arabic text). You wouldn't want to go with a solution where one of those types didn't even seem to exist from Python. Really, the int/long thing is pretty unique, and it's because ints are a bit special. With almost any other data type, we could just create separate types which inherit from each other, but with ints we want them to appear seamlessly to be the same thing as longs, yet we are willing to grant them almost NO memory overhead. The only other type I can think of which we want to be so small is float, and if we decided to introduce a arbitrary precision binary based floating point type to complement Decimal and Rational, then I think we'd need our heads examined. -- Michael Chermside From tim.one at comcast.net Fri Dec 5 15:16:05 2003 From: tim.one at comcast.net (Tim Peters) Date: Fri Dec 5 15:16:16 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re:2.3.3 cycle) In-Reply-To: <200312051523.hB5FNJTU010018@maxim.off.ekorp.com> Message-ID: [Anthony Baxter] > After a short chat with Thomas, we'd like to pencil in the 19th for > the final release (so two weeks from the RC1). Does this cause anyone > pain? Lots of pain, but it's a *good* kind of pain. I'll take more pain like that, if you have it. Thanks for your work on this! From theller at python.net Fri Dec 5 15:48:56 2003 From: theller at python.net (Thomas Heller) Date: Fri Dec 5 15:49:05 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) In-Reply-To: <200312051440.hB5Eer1P009275@maxim.off.ekorp.com> (Anthony Baxter's message of "Sat, 06 Dec 2003 01:40:53 +1100") References: <200312051440.hB5Eer1P009275@maxim.off.ekorp.com> Message-ID: <4qwft2h3.fsf@python.net> While building and trying out the windows installer yesterday, I had some crashes in pydocgui.pyw (the module documentation server), both on win98SE and on XP Pro. The crash ocurred when it was closed, and it seems only when the test suite was also installed (usually it also finds something in the test suite). Win98 was unhelpful as usual in only showing the top of the machine stack, winXP displayed a message box showing an access violation in TclNotifier. I were not able to debug this, because I couldn't get Tkinter to work in the version running from the build dir, and I had no more time - I assume there are some env vars to be set so that it finds the tcl and tk dlls. I could not reproduce the problem with the final built which is no out in the cold, so I assume it was a temporary problem. But it would be nice if other people could try this out as well before this comes back later on me. Thanks, Thomas From Jack.Jansen at cwi.nl Fri Dec 5 18:17:15 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Fri Dec 5 18:17:21 2003 Subject: [Python-Dev] Constants from C headers and int/long unification Message-ID: <29CBBE16-2779-11D8-BAB7-000A27B19B96@cwi.nl> The discussions on unification of int and long have now gone on for so many months that I no longer know how to handle them for the following scenario, please enlighten me. I have C header files which I parse and to which I can do some simple processing before outputting them as Python code. An example C file will contain enum { kConstantOne = 0xffffffff, kConstantTwo = -1, }; Regexps will grab the two constants and output them to Python. The Python programmer will pass these constants to an API routine, at which point I need a PyArg_Parse format char that will convert both of these values to the 32 bit pattern 0xffffffff. Question 1: Do I need to add anything to the constants before outputting them as Python code? Question 2: Is there a format char that will convert both of the values produced by the answer to question 1 to the 32 bit pattern 0xffffffff? -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From guido at python.org Fri Dec 5 18:24:05 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 5 18:24:13 2003 Subject: [Python-Dev] Constants from C headers and int/long unification In-Reply-To: Your message of "Sat, 06 Dec 2003 00:17:15 +0100." <29CBBE16-2779-11D8-BAB7-000A27B19B96@cwi.nl> References: <29CBBE16-2779-11D8-BAB7-000A27B19B96@cwi.nl> Message-ID: <200312052324.hB5NO5o09561@c-24-5-183-134.client.comcast.net> > I have C header files which I parse and to which I can do some simple > processing before outputting them as Python code. An example C file > will contain > > enum { > kConstantOne = 0xffffffff, > kConstantTwo = -1, > }; > > Regexps will grab the two constants and output them to Python. The > Python programmer will pass these constants to an API routine, at which > point I need a PyArg_Parse format char that will convert both of these > values to the 32 bit pattern 0xffffffff. > > Question 1: Do I need to add anything to the constants before > outputting them as Python code? No -- kConstantTwo automatically become a long with value 2**32-1 in Python 2.3 and up. > Question 2: Is there a format char that will convert both of the values > produced by the answer to question 1 to the 32 bit pattern 0xffffffff? If you mean a PyArg_Parse format char, yes: use 'k'. If you meant a Python % format char, you'd have to write "%x" % (x&0xffffffff). --Guido van Rossum (home page: http://www.python.org/~guido/) From anthony at ekit-inc.com Fri Dec 5 22:39:39 2003 From: anthony at ekit-inc.com (Anthony Baxter) Date: Fri Dec 5 22:39:59 2003 Subject: [Python-Dev] source-level backward compatibility requirements In-Reply-To: Message from Guido van Rossum of "Fri, 05 Dec 2003 09:19:11 -0800." <200312051719.hB5HJBp08708@c-24-5-183-134.client.comcast.net> Message-ID: <200312060339.hB63dd8F022401@maxim.off.ekorp.com> >>> Guido van Rossum wrote > > Does this warrant a c2 release? I'm happy to make the changes right > > now. > Tht's a question for Anthony. I suppose there will be a c2 release > anyway. I hadn't planned a c2 release - the c2 release would be only for brown-paper-bag kinda bugs. I'm not sure that merely removing a warning for apply() really merits a whole nother release candidate, but I'm willing to be convinced otherwise. From anthony at ekit-inc.com Fri Dec 5 22:41:54 2003 From: anthony at ekit-inc.com (Anthony Baxter) Date: Fri Dec 5 22:42:07 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python bltinmodule.c, 2.292.10.3, 2.292.10.4 In-Reply-To: Message from "Fred L. Drake, Jr." of "Fri, 05 Dec 2003 15:02:54 CDT." <16336.58478.552434.65123@grendel.fdrake.net> Message-ID: <200312060341.hB63fsJO022461@maxim.off.ekorp.com> >>> "Fred L. Drake, Jr." wrote > > Raymond Hettinger writes: > > FYI, I have a few more doc fixes for Py2.3.3. > > Anthony asked me to wait until Sunday to put them in. > > Any particular reason for the delay? In case there was a packaging bug in the rc1 release, and I wanted to cut a new one. From tim.one at comcast.net Sat Dec 6 00:00:08 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Dec 6 00:00:13 2003 Subject: [Python-Dev] source-level backward compatibility requirements In-Reply-To: <200312060339.hB63dd8F022401@maxim.off.ekorp.com> Message-ID: [Anthony Baxter] > I hadn't planned a c2 release - the c2 release would be only for > brown-paper-bag kinda bugs. I'm not sure that merely removing a > warning for apply() really merits a whole nother release candidate, > but I'm willing to be convinced otherwise. I won't be trying to convince you: I reviewed Fred's checkin, and it's not going to break anything. Of course, if it does, Fred will be in really big trouble, not me . From skip at pobox.com Sat Dec 6 12:48:19 2003 From: skip at pobox.com (Skip Montanaro) Date: Sat Dec 6 12:48:18 2003 Subject: [Python-Dev] test_unicode_file failing on Mac OS X Message-ID: <16338.5731.362994.497561@montanaro.dyndns.org> Two of the test_unicode_file began failing on my Mac today (fresh cvs up, OS X 10.2.8, vanilla unix-style build): ====================================================================== FAIL: test_directories (__main__.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "../Lib/test/test_unicode_file.py", line 155, in test_directories self._do_directory(TESTFN_ENCODED+ext, TESTFN_ENCODED+ext, os.getcwd) File "../Lib/test/test_unicode_file.py", line 103, in _do_directory make_name) AssertionError: '@test-a\xcc\x80o\xcc\x80.dir' != '@test-\xc3\xa0\xc3\xb2.dir' ====================================================================== FAIL: test_single_files (__main__.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "../Lib/test/test_unicode_file.py", line 141, in test_single_files self._test_single(TESTFN_ENCODED) File "../Lib/test/test_unicode_file.py", line 116, in _test_single self._do_single(filename) File "../Lib/test/test_unicode_file.py", line 41, in _do_single self.failUnless(base in os.listdir(path)) AssertionError Skip From raymond.hettinger at verizon.net Sat Dec 6 15:54:21 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sat Dec 6 15:54:27 2003 Subject: [Python-Dev] list.sorted() Message-ID: <005201c3bc3b$20069320$e841fea9@oemcomputer> When working on itertools.groupby(), the tutorial updates, and exercising the newer features, I've made frequent use of list.sorted(). The good news is that after using it many times, I no longer hate the name ;-) Also, it fits well inside list comps (and is expected to do the same with gen exps) plus it works well with itertools. In fact, it is a delight to use. One surprise use case was how nicely it works with unittest.TestCase.assertEqual() when verifying that two collections are equal with respect to permutation. The bad news is that the list dot prefix adds no value, feels awkward, and interferes with the eye's ability to parse expressions containing list.sorted(). Part of the reason for making it a classmethod was that it *is* effectively an alternative constructor for lists. OTOH, that could be said for any function that returns a list. The two disadvantages of having it as a classmethod are the mandatory list dot prefix and that it is confusing when called with an existing list: [f(elem, arg) for elem in [3,2,1].sorted(anotherseq)] It's up to someone else to propose the obvious solutions -- I've already gone over my quota and am running low on asbestos ;-) Whatever the answer, it's important that the result look clean when used in gen exps or as an argument to functions expecting a sequence. Raymond Hettinger From python at rcn.com Sat Dec 6 17:32:35 2003 From: python at rcn.com (Raymond Hettinger) Date: Sat Dec 6 17:32:43 2003 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Doc/lib libitertools.tex, 1.23, 1.24 In-Reply-To: Message-ID: <005d01c3bc48$d8d4d580$e841fea9@oemcomputer> Thank you. Raymond From bac at OCF.Berkeley.EDU Sat Dec 6 17:32:37 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sat Dec 6 17:32:49 2003 Subject: [Python-Dev] python-dev Summary for 2003-11-16 through 2003-11-30 [rough draft] Message-ID: <3FD25905.4010003@ocf.berkeley.edu> OK, you guys know the drill. I plan on sending this out no later than the end of next week. ----------------------------- python-dev Summary for 2003-11-16 through 2003-11-30 ++++++++++++++++++++++++++++++++++++++++++++++++++++ This is a summary of traffic on the `python-dev mailing list`_ from November 16, 2003 through November 30, 2003. It is intended to inform the wider Python community of on-going developments on the list. To comment on anything mentioned here, just post to `comp.lang.python`_ (or email python-list@python.org which is a gateway to the newsgroup) with a subject line mentioning what you are discussing. All python-dev members are interested in seeing ideas discussed by the community, so don't hesitate to take a stance on something. And if all of this really interests you then get involved and join `python-dev`_! This is the thirtieth summary written by Brett Cannon (you like me! You really like me! =). To contact me, please send email to brett at python.org ; I do not have the time to keep up on comp.lang.python and thus do not always catch follow-ups posted there. All summaries are archived at http://www.python.org/dev/summary/ . Please note that this summary is written using reStructuredText_ which can be found at http://docutils.sf.net/rst.html . Any unfamiliar punctuation is probably markup for reST_ (otherwise it is probably regular expression syntax or a typo =); you can safely ignore it, although I suggest learning reST; it's simple and is accepted for `PEP markup`_ and gives some perks for the HTML output. Also, because of the wonders of programs that like to reformat text, I cannot guarantee you will be able to run the text version of this summary through Docutils_ as-is unless it is from the original text file. .. _PEP Markup: http://www.python.org/peps/pep-0012.html The in-development version of the documentation for Python can be found at http://www.python.org/dev/doc/devel/ and should be used when looking up any documentation on something mentioned here. PEPs (Python Enhancement Proposals) are located at http://www.python.org/peps/ . To view files in the Python CVS online, go to http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/ . Reported bugs and suggested patches can be found at the SourceForge_ project page. .. _python-dev: http://www.python.org/dev/ .. _SourceForge: http://sourceforge.net/tracker/?group_id=5470 .. _python-dev mailing list: http://mail.python.org/mailman/listinfo/python-dev .. _comp.lang.python: http://groups.google.com/groups?q=comp.lang.python .. _Docutils: http://docutils.sf.net/ .. _reST: .. _reStructuredText: http://docutils.sf.net/rst.html .. contents:: .. _last summary: http://www.python.org/dev/summary/2003-10-16_2003-11-15.html ===================== Summary Announcements ===================== Wow, I actually managed to get a summary out less than a week when its coverage dates ended. Perks of procrastinating for studying for finals. =) First, an errata on the last summary. I said generator expressions were Peter Norvig's idea. Turns out it was Raymond Hettinger, the man who has so many new ideas that his flame retardant suit has his initials embroidered on it, who came up with the original idea in `PEP 289`_. PyCon_ registration has gone live! Go to the page to register. And if you don't we will have a possessed Barry Warsaw come after you (to get the reference, go to http://www.pycon.org/ and take a look at the banner graphic; Barry is the second from the right)! =) The system for accepting proposals for PyCon is still being worked on. It should be up in the very near future. Since it was not up in time for the original deadline, the new proposal deadline has been extended. This summary is brought to you by the `iTunes Music Store`_ and the following songs from the `Verve Remixed 2 - Exclusive EP `__ album (which are linked to the iTMS and thus require iTunes to be installed): - `Manteca, by Dizzy Gillespie & Funky Lowlives `__ - `Sinnerman, by Felix da Housecat & Nina Simone `__ .. _PEP 289: http://www.python.org/peps/pep-0289.html .. _PyCon: http://www.pycon.org/ .. _iTunes Music Store: http://www.apple.com/itunes/store/ ========= Summaries ========= -------------------------------------------------- Aggregate refcounting is not always a hard science -------------------------------------------------- Christian Tismer discovered that the aggregate refcount stored in _Py_RefTotal (only available in a Py_REF_DEBUG build which is part of a debug build; see Misc/SpecialBuilds.txt for all the special builds that are available) can be wrong when code starts to do funky things in __del__ methods and manipulating the refcount of objects directly. The offending code has been fixed in the 2.3.x branch and up. Contributing threads: - `_Py_RefTotal wrong in Py 2.2.3 `__ ------------------------------------------------ Bulk up reversed() with __reversed__ and __len__ ------------------------------------------------ When the new 'reversed' built-in was being discussed, the idea of having a __reversed__ magic method for it was proposed. But because it might get misused by applying 'reversed' to iterators that were infinite (think generators and itertools iterators), Guido had it removed. But Raymond Hettinger, who came up with 'reversed' in the first place, came up with a modified idea; only have 'reversed' use __reversed__ if both that magic method and __len__ are defined. That way 'reversed' will only try to reverse its argument if it has a known bounded length (as of right now it only works if the object has __getitem__ and __len__ defined). As of right now it has not been pronounced upon. Contributing threads: - ` __reversed__ protocol `__ ------------------------------------------------------ Change in object.__setattr__ semantics from 2.2 to 2.3 ------------------------------------------------------ A change in how object.__setattr__ works went unmentioned in the 'What's New...' doc for 2.3 by accident. In case you have not run into this, you cannot use object.__setattr__ if a derived type defines __setattr__ itself. It is not an issue if you just use the built-in setattr. Contributing threads: - `2.2=>2.3 object.__setattr__(cls,attr,value) `__ ----------------------------- Say goodbye to MacOS9 support ----------------------------- Starting with 2.4, Python will not support MacOS9. Jack Jansen has ripped out the support so that all Mac-related efforts can focus on OS X work. This does not affect any previous versions of Python. Contributing threads: - `Ripping out Macintosh support `__ ------------------ 2.3.3 in the works ------------------ 2.3.3c1 has been released as of this writing. You can find it at http://www.python.org/2.3.3/ . As usual, please download it and run the test suite along with your own code. Contributing threads: - `Time for 2.3.3? `__ -------------------- Project ideas galore -------------------- Last month I asked for ideas for a masters thesis. This led to a huge response with various project ideas. After going through them all I posted an annotated list of all the ideas presented to me (see the contributing thread for the email link). If you are ever bored for say, a month or more, you can try to tackle one of these projects to keep yourself occupied. Contributing threads: - `Thesis ideas list `__ ------------------------- No more hex/oct warnings! ------------------------- The warnings from 2.3 about how hex/oct constants were going to change in 2.4 are now gone since the change has been checked into CVS. `PEP 237`_, which specified this change, also mentioned warnings of the change in 2.4 along with changes to repr for longs. The warnings are not going to exist in Python 2.4 . It was felt the warnings in 2.3 were visible and noisy enough to get the point across. As for the repr change for longs, that is not going to happen. The repr of a long will continue to have an "L" appended to the end of it. .. _PEP 237: http://www.python.org/peps/pep_0237.html Contributing threads: - `Int FutureWarnings and other 2.4 TODOs `__ ------------------------------ Backticks to go bye-bye in 3.0 ------------------------------ Consider yourselves warned; backticks will be removed in Python 3.0 . Contributing threads: - `Banishing apply(), buffer(), coerce(), and intern() `__ ----------------------- itertools grows groupby ----------------------- itertools.groupby takes an iterable and a key-accessing function and returns a tuple of the key and an iterator containing items that match the current key. Read the documentation for a more thorough explanation. Contributing threads: - `"groupby" iterator `__ From fincher.8 at osu.edu Sat Dec 6 19:00:31 2003 From: fincher.8 at osu.edu (Jeremy Fincher) Date: Sat Dec 6 18:02:39 2003 Subject: [Python-Dev] list.sorted() In-Reply-To: <005201c3bc3b$20069320$e841fea9@oemcomputer> References: <005201c3bc3b$20069320$e841fea9@oemcomputer> Message-ID: <200312061900.31919.fincher.8@osu.edu> On Saturday 06 December 2003 03:54 pm, Raymond Hettinger wrote: > It's up to someone else to propose the obvious solutions -- I've already > gone over my quota and am running low on asbestos ;-) I remember someone, somewhere (comp.lang.python, I think) proposing a "consumers" module for the new ideas people are having now that generator expressions will be unleashed in 2.4. A "sorted" consumer would seem to fit right in, I would think. Jeremy From oussoren at cistron.nl Sun Dec 7 00:35:22 2003 From: oussoren at cistron.nl (Ronald Oussoren) Date: Sun Dec 7 00:35:26 2003 Subject: [Python-Dev] list.sorted() In-Reply-To: <005201c3bc3b$20069320$e841fea9@oemcomputer> References: <005201c3bc3b$20069320$e841fea9@oemcomputer> Message-ID: <26D718AD-2877-11D8-BB3A-0003931CFE24@cistron.nl> On 6 dec 2003, at 21:54, Raymond Hettinger wrote: > The two disadvantages of having it as a classmethod are the mandatory > list dot prefix and that it is confusing when called with an existing > list: > > [f(elem, arg) for elem in [3,2,1].sorted(anotherseq)] > The latter problem is a generic problem of class methods, e.g. someDict.fromKeys(seq). One way to fix this would be to make sure that class methods are no longer accessible through instances. Ronald From raymond.hettinger at verizon.net Sun Dec 7 07:38:43 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sun Dec 7 07:39:43 2003 Subject: [Python-Dev] Py2.4 pre-alpha snapshot Message-ID: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> I think it would be worthwhile to occasionally (every 3 months or so) package a Py2.4 pre-alpha release. My feel is that a number of people without compilers (Windows users especially) would enjoy working with the latest python if it were an easy thing to do (has an installer, etc). Besides increasing community involvement, this could open up a whole new stream of user feedback so we can discover issues sooner rather than later. Since non-developers stress the system in different ways, they are more likely to surface various documentation, usability, and integration bugs. An opening message could be added that clearly states that everything is subject to change, under development, non-production, etc. Raymond Hettinger -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031207/47a7d9e1/attachment-0001.html From skip at manatee.mojam.com Sun Dec 7 08:00:52 2003 From: skip at manatee.mojam.com (Skip Montanaro) Date: Sun Dec 7 08:01:01 2003 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200312071300.hB7D0qvo023090@manatee.mojam.com> Bug/Patch Summary ----------------- 593 open / 4407 total bugs (+50) 217 open / 2492 total patches (+24) New Bugs -------- Can't send Apple Events without WindowServer connection (2003-12-01) http://python.org/sf/852150 add support for cjkcodecs to Python email (2003-12-01) http://python.org/sf/852347 ^$ won't split on empty line (2003-12-02) http://python.org/sf/852532 bz2:compress empty string inconsistent (2003-12-02) http://python.org/sf/853061 After fork in subthread, signals are blocked (2003-12-03) http://python.org/sf/853411 IP6 address parsing i sgmllib (2003-12-03) http://python.org/sf/853506 socket.recv() raises MemoryError exception (WindowsXP ONLY) (2003-12-03) http://python.org/sf/853507 new _Launch module includes useless deprecated functions (2003-12-03) http://python.org/sf/853558 Carbon.CF.CFURLRef should be easier to create (2003-12-03) http://python.org/sf/853656 No documentation for zipimport module (2003-12-03) http://python.org/sf/853800 'Demo/embed/' , It crash (2003-12-03) http://python.org/sf/853862 Message.get_filename() problem (2003-12-04) http://python.org/sf/854102 Python-2.3.3c1, Solaris 2.7: socketmodule does not compile (2003-12-05) http://python.org/sf/854823 New Patches ----------- urllib2 CacheFTPHandler doesn't work on multiple dirs (2003-11-30) http://python.org/sf/851736 urllib2 silently returns None when auth_uri is mismatched (2003-11-30) http://python.org/sf/851752 bugfix for [Bugs-850964](optparse) (2003-11-30) http://python.org/sf/851902 Refactored patch #815911 for logging with SocketHandler (2003-12-01) http://python.org/sf/851993 broken url in colorsys documentation (2003-12-01) http://python.org/sf/852236 Replace backticks with repr() (2003-12-01) http://python.org/sf/852334 tests and processors patch for urllib2 (2003-12-02) http://python.org/sf/852995 Tix hlist missing entry_configure and entry_cget methods (2003-12-03) http://python.org/sf/853488 Optional keyword unicode args not handled correctly (2003-12-04) http://python.org/sf/853890 Implement lazy read for sockets (2003-12-04) http://python.org/sf/853963 ConfigParser should raise an error for duplicate options (2003-12-04) http://python.org/sf/854484 shutil.copyfile destroys hard links (Bug [ 851123 ]) (2003-12-05) http://python.org/sf/854853 fix typos (2003-12-05) http://python.org/sf/855195 Closed Bugs ----------- os.strerror doesn't understand windows error codes (2003-10-14) http://python.org/sf/823672 Windows installer 2.3.2 leaves old version in control panel (2003-11-05) http://python.org/sf/836515 Typos in the docs (Extending/Embedding + Python/C API) (2003-11-09) http://python.org/sf/838938 tkFileDialog.Open is broken (2003-11-17) http://python.org/sf/843999 "up" instead of "down" in turtle module documentation (2003-11-17) http://python.org/sf/844123 couple of new list.sort bugs (2003-11-25) http://python.org/sf/848856 winreg can segfault (2003-11-28) http://python.org/sf/851056 Item out of order on builtin function page (2003-11-29) http://python.org/sf/851152 Bug tracker page asks for login even when logged in (2003-11-29) http://python.org/sf/851156 New-style classes with __eq__ but not __hash__ are hashable (2003-11-29) http://python.org/sf/851449 Closed Patches -------------- Better output for unittest (2003-04-16) http://python.org/sf/722638 whichdb not ID'ing dbm files with GDBM backend (2003-07-06) http://python.org/sf/766650 unittest's main don't handle TestSuite in command line (2003-09-10) http://python.org/sf/804212 From aahz at pythoncraft.com Sun Dec 7 08:29:59 2003 From: aahz at pythoncraft.com (Aahz) Date: Sun Dec 7 08:30:03 2003 Subject: [Python-Dev] Py2.4 pre-alpha snapshot In-Reply-To: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> References: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> Message-ID: <20031207132959.GA26733@panix.com> On Sun, Dec 07, 2003, Raymond Hettinger wrote: > > I think it would be worthwhile to occasionally (every 3 months or so) > package a Py2.4 pre-alpha release. My feel is that a number of people > without compilers (Windows users especially) would enjoy working with > the latest python if it were an easy thing to do (has an installer, > etc). > > Besides increasing community involvement, this could open up a whole new > stream of user feedback so we can discover issues sooner rather than > later. Since non-developers stress the system in different ways, they > are more likely to surface various documentation, usability, and > integration bugs. Let's see if I can channel Tim correctly: "The number of people who have historically downloaded beta and candidate releases indicates that it's unlikely that people will download pre-alpha releases." OTOH, David Goodger has been reasonably successful with docutils having snapshots. It's certainly a worthwhile question about whether people would be more likely to download beta and candidate releases if snapshots were regularly available. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From perky at i18n.org Sun Dec 7 09:36:20 2003 From: perky at i18n.org (Hye-Shik Chang) Date: Sun Dec 7 09:36:25 2003 Subject: [Python-Dev] Py2.4 pre-alpha snapshot In-Reply-To: <20031207132959.GA26733@panix.com> References: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> <20031207132959.GA26733@panix.com> Message-ID: <20031207143620.GA32893@i18n.org> On Sun, Dec 07, 2003 at 08:29:59AM -0500, Aahz wrote: > On Sun, Dec 07, 2003, Raymond Hettinger wrote: > > > > I think it would be worthwhile to occasionally (every 3 months or so) > > package a Py2.4 pre-alpha release. My feel is that a number of people > > without compilers (Windows users especially) would enjoy working with > > the latest python if it were an easy thing to do (has an installer, > > etc). > > > > Besides increasing community involvement, this could open up a whole new > > stream of user feedback so we can discover issues sooner rather than > > later. Since non-developers stress the system in different ways, they > > are more likely to surface various documentation, usability, and > > integration bugs. > > Let's see if I can channel Tim correctly: "The number of people who have > historically downloaded beta and candidate releases indicates that it's > unlikely that people will download pre-alpha releases." JFYI, FreeBSD distributes its own unofficial snapshot of Python 2.4. http://www.freshports.org/lang/python-devel/ Looking at download statistics, it seems that about 60~80 users install FreeBSD's python-devel snapshot for every updates. (the count can be much more because downloads from many mirror sites isn't counted.) Hye-Shik From fredrik at pythonware.com Sun Dec 7 09:29:58 2003 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sun Dec 7 09:50:23 2003 Subject: [Python-Dev] Re: Py2.4 pre-alpha snapshot References: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> <20031207132959.GA26733@panix.com> Message-ID: Aahz wrote: > OTOH, David Goodger has been reasonably successful with docutils having > snapshots. well, that's not that strange, given that the docutils page says that the latest official release is 18 months old [1], and is named 0.3, and that users should avoid using official releases. for some reason, I'm not sure that's the right approach for Python. 1) that appears to be a typo, though. the effbot's 89'th law of software releases: if you don't release stuff often enough to notice a typo on your release page in 6 months, you're not releasing stuff often enough ;-) From pje at telecommunity.com Sun Dec 7 10:37:05 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun Dec 7 10:35:03 2003 Subject: [Python-Dev] Where to report PyPI problem? Message-ID: <5.1.0.14.0.20031207103226.03d88bd0@mail.telecommunity.com> I was updating some release information today, and got this traceback: Traceback (most recent call last): File "/usr/local/pypi/lib/pypi/webui.py", line 99, in run self.inner_run() File "/usr/local/pypi/lib/pypi/webui.py", line 374, in inner_run getattr(self, action)() File "/usr/local/pypi/lib/pypi/webui.py", line 1209, in pkg_edit info = reldict[ver] KeyError: 0.9.2%20%28bug%20fix%20release%29 It looks rather like PyPI's not decoding URL-escaped field names. I'd submit a patch, but I'm not sure where to find the source. I poked around the nondist tree in CVS, but didn't see it. From aahz at pythoncraft.com Sun Dec 7 10:58:36 2003 From: aahz at pythoncraft.com (Aahz) Date: Sun Dec 7 10:58:39 2003 Subject: [Python-Dev] Where to report PyPI problem? In-Reply-To: <5.1.0.14.0.20031207103226.03d88bd0@mail.telecommunity.com> References: <5.1.0.14.0.20031207103226.03d88bd0@mail.telecommunity.com> Message-ID: <20031207155836.GB6153@panix.com> On Sun, Dec 07, 2003, Phillip J. Eby wrote: > > I was updating some release information today, and got this traceback: > > Traceback (most recent call last): > File "/usr/local/pypi/lib/pypi/webui.py", line 99, in run > self.inner_run() > File "/usr/local/pypi/lib/pypi/webui.py", line 374, in inner_run > getattr(self, action)() > File "/usr/local/pypi/lib/pypi/webui.py", line 1209, in pkg_edit > info = reldict[ver] > KeyError: 0.9.2%20%28bug%20fix%20release%29 > > It looks rather like PyPI's not decoding URL-escaped field names. I'd > submit a patch, but I'm not sure where to find the source. I poked around > the nondist tree in CVS, but didn't see it. Didja see the "Contact Us" section on http://www.python.org/pypi -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From Jack.Jansen at cwi.nl Sun Dec 7 11:32:23 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Sun Dec 7 11:32:30 2003 Subject: [Python-Dev] test_unicode_file failing on Mac OS X In-Reply-To: <16338.5731.362994.497561@montanaro.dyndns.org> References: <16338.5731.362994.497561@montanaro.dyndns.org> Message-ID: On 6-dec-03, at 18:48, Skip Montanaro wrote: > Two of the test_unicode_file began failing on my Mac today (fresh cvs > up, OS > X 10.2.8, vanilla unix-style build): > > > ====================================================================== > FAIL: test_directories (__main__.TestUnicodeFiles) > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "../Lib/test/test_unicode_file.py", line 155, in > test_directories > self._do_directory(TESTFN_ENCODED+ext, TESTFN_ENCODED+ext, > os.getcwd) > File "../Lib/test/test_unicode_file.py", line 103, in > _do_directory > make_name) > AssertionError: '@test-a\xcc\x80o\xcc\x80.dir' != > '@test-\xc3\xa0\xc3\xb2.dir' This is probably related to the two flavors of unicode there are, one which prefers to have all accents separately from the letters as much as possible and one which prefers the reverse. I keep forgetting the names of the two, they're somewhat silly. But the problem is that Python prefers to represent the string "?" as the two characters "a" and "umlaut on the previous char", and MacOSX prefers to represent the same string as "a with umlaut on it". Or the other way around, this is something else I always forget. And while there are algorithms to convert the combined form of unicode to the uncombined form and vice versa there are no Python codecs to do this. The OSX system calls do the right thing (convert both forms to what it prefers), but when you do a readdir() you don't get the string back you put it. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From pf_moore at yahoo.co.uk Sun Dec 7 11:57:26 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Sun Dec 7 11:57:12 2003 Subject: [Python-Dev] Re: Py2.4 pre-alpha snapshot References: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> Message-ID: "Raymond Hettinger" writes: > I think it would be worthwhile to occasionally (every 3 months or so) > package a Py2.4 pre-alpha release. My feel is that a number of people > without compilers (Windows users especially) would enjoy working with > the latest python if it were an easy thing to do (has an installer, > etc). Speaking as a "latest release" junkie with not enough time to build for myself, I'd say it might be nice. But the killer is that you need extensions. On Windows, for instance, you'd need to prevail on Mark Hammond to produce 2.4-compatible win32all builds (pretty much essential on Windows, and a serious pain to build by hand, I believe). Also, you'd need an installer which would sit happily alongside the production version (install in a separate directory, include a python24.exe so as not to clash in terms of command names, etc). > Besides increasing community involvement, this could open up a whole new > stream of user feedback so we can discover issues sooner rather than > later. Since non-developers stress the system in different ways, they > are more likely to surface various documentation, usability, and > integration bugs. A compromise may just be a simple zipped up compile. That helps people without compilers, but leaves the effort of "installing" to the user. It still doesn't help with the extensions issue, though... Paul. -- This signature intentionally left blank From kbk at shore.net Sun Dec 7 12:14:42 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Sun Dec 7 12:14:46 2003 Subject: [Python-Dev] Py2.4 pre-alpha snapshot In-Reply-To: <20031207132959.GA26733@panix.com> (Aahz's message of "Sun, 7 Dec 2003 08:29:59 -0500") References: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> <20031207132959.GA26733@panix.com> Message-ID: Aahz writes: > On Sun, Dec 07, 2003, Raymond Hettinger wrote: >> >> I think it would be worthwhile to occasionally (every 3 months or so) >> package a Py2.4 pre-alpha release. My feel is that a number of people >> without compilers (Windows users especially) would enjoy working with >> the latest python if it were an easy thing to do (has an installer, >> etc). >> >> Besides increasing community involvement, this could open up a whole new >> stream of user feedback so we can discover issues sooner rather than >> later. Since non-developers stress the system in different ways, they >> are more likely to surface various documentation, usability, and >> integration bugs. > > Let's see if I can channel Tim correctly: "The number of people who have > historically downloaded beta and candidate releases indicates that it's > unlikely that people will download pre-alpha releases." > > OTOH, David Goodger has been reasonably successful with docutils having > snapshots. It's certainly a worthwhile question about whether people > would be more likely to download beta and candidate releases if snapshots > were regularly available. In my case, I can no longer compile Python on Windows (VC5 worked up to 2.3). So to test IDLE against 2.4 on that platform, I need a Windows installer. As I recollect, there was discussion about moving to VC7 which has a free compiler available. Is that going to happen on 2.4? -- KBK From pedronis at bluewin.ch Sun Dec 7 12:43:42 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Sun Dec 7 12:40:34 2003 Subject: [Python-Dev] source-level backward compatibility requirements In-Reply-To: <200312051654.hB5Gs9e08643@c-24-5-183-134.client.comcast.ne t> References: <200312051523.hB5FNJTU010018@maxim.off.ekorp.com> <16900000.1070638518@leeloo.intern.geht.de> <16336.44564.604153.644072@grendel.fdrake.net> <200312051615.hB5GFvW08552@c-24-5-183-134.client.comcast.net> <16336.45230.749132.305648@grendel.fdrake.net> Message-ID: <5.2.1.1.0.20031207183758.02fd16c0@pop.bluewin.ch> At 08:54 05.12.2003 -0800, Guido van Rossum wrote: > > On the other hand, I find I'm not really a fan of deprecating apply() > > in the first place, so I can think of other solutions. ;-) > >We could deprecate it in the docs but remove the silent warning which >slows it down. Then we could leave it in the logging module until >3.0. Btw, maybe this is known to everybody, but the python layers of SWIG produced Python bindings still use apply all over the place. From martin at v.loewis.de Sun Dec 7 12:47:37 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sun Dec 7 12:47:55 2003 Subject: [Python-Dev] Py2.4 pre-alpha snapshot In-Reply-To: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> References: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> Message-ID: "Raymond Hettinger" writes: > I think it would be worthwhile to occasionally (every 3 months or so) > package a Py2.4 pre-alpha release. My feel is that a number of people > without compilers (Windows users especially) would enjoy working with > the latest python if it were an easy thing to do (has an installer, > etc). Anybody could do that, without any official blessing. For example, I'm distributing Itanium binaries at my own leisure, and I plan to provide Itanium binaries for 2.4-in-progress, once I complete fixing the pending issues. So if you want to provide snapshots, just go ahead. If you need hyperlinks from Python.org, or disk space on python.org: that can be provided. OTOH, I would consider it a wastage of resources if the regular packagers would put any effort into snapshots (unless they really want to do this on their own). Regards, Martin From martin at v.loewis.de Sun Dec 7 12:50:08 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sun Dec 7 12:50:23 2003 Subject: [Python-Dev] Py2.4 pre-alpha snapshot In-Reply-To: References: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> <20031207132959.GA26733@panix.com> Message-ID: kbk@shore.net (Kurt B. Kaiser) writes: > As I recollect, there was discussion about moving to VC7 which has > a free compiler available. Is that going to happen on 2.4? Yes. If nobody beats me in that, I'll do it over Christmas. Regards, Martin From martin at v.loewis.de Sun Dec 7 12:56:54 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sun Dec 7 12:57:28 2003 Subject: [Python-Dev] test_unicode_file failing on Mac OS X In-Reply-To: References: <16338.5731.362994.497561@montanaro.dyndns.org> Message-ID: Jack Jansen writes: > This is probably related to the two flavors of unicode there are, one > which prefers to have all accents separately from the letters as much > as possible and one which prefers the reverse. I keep forgetting the > names of the two, they're somewhat silly. OS X uses what is called the "decomposed normal form", splitting combined characters into the base character and the combining accent. Python supports either form, but will use precomposed characters more often than not. > And while there are algorithms to convert the combined form of unicode > to the uncombined form and vice versa there are no Python codecs to do > this. Not as a codec, but as unicodedata.normalize. If you do unicodedata.normalize(composed_string, "NFD") you get the string that OS X wants you to use. Of course, with Unicode-on-Windows, the story is mostly vice-versa. NTFS/Win32 does not perform any normalization, so you can actually store the precomposed and the decomposed string simultaneously in the same directory (which is confusing). The platform codecs always generate the precomposed form, though, so you are more likely to find the precomposed form on disk. For the test, it would be best to compare normal forms, and have the test pass if the normal forms (NFD) are equal. Regards, Martin From pje at telecommunity.com Sun Dec 7 13:26:03 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun Dec 7 13:24:04 2003 Subject: [Python-Dev] Where to report PyPI problem? In-Reply-To: <20031207155836.GB6153@panix.com> References: <5.1.0.14.0.20031207103226.03d88bd0@mail.telecommunity.com> <5.1.0.14.0.20031207103226.03d88bd0@mail.telecommunity.com> Message-ID: <5.1.0.14.0.20031207132430.03e25bc0@mail.telecommunity.com> At 10:58 AM 12/7/03 -0500, Aahz wrote: >On Sun, Dec 07, 2003, Phillip J. Eby wrote: > > > > I was updating some release information today, and got this traceback: > > > > Traceback (most recent call last): > > File "/usr/local/pypi/lib/pypi/webui.py", line 99, in run > > self.inner_run() > > File "/usr/local/pypi/lib/pypi/webui.py", line 374, in inner_run > > getattr(self, action)() > > File "/usr/local/pypi/lib/pypi/webui.py", line 1209, in pkg_edit > > info = reldict[ver] > > KeyError: 0.9.2%20%28bug%20fix%20release%29 > > > > It looks rather like PyPI's not decoding URL-escaped field names. I'd > > submit a patch, but I'm not sure where to find the source. I poked around > > the nondist tree in CVS, but didn't see it. > >Didja see the "Contact Us" section on http://www.python.org/pypi Aha. When I had hovered the "Bug Reports" link and saw a SourceForge link, I assumed it was the general Python bug reports page, not PyPI-specific. Sorry about that. I'll pop in a bug report there. Thanks. From theller at python.net Sun Dec 7 14:46:27 2003 From: theller at python.net (Thomas Heller) Date: Sun Dec 7 14:46:44 2003 Subject: [Python-Dev] Re: Py2.4 pre-alpha snapshot In-Reply-To: (Paul Moore's message of "Sun, 07 Dec 2003 16:57:26 +0000") References: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> Message-ID: Paul Moore writes: > "Raymond Hettinger" writes: > >> I think it would be worthwhile to occasionally (every 3 months or so) >> package a Py2.4 pre-alpha release. My feel is that a number of people >> without compilers (Windows users especially) would enjoy working with >> the latest python if it were an easy thing to do (has an installer, >> etc). > > Speaking as a "latest release" junkie with not enough time to build > for myself, I'd say it might be nice. But the killer is that you need > extensions. On Windows, for instance, you'd need to prevail on Mark > Hammond to produce 2.4-compatible win32all builds (pretty much > essential on Windows, and a serious pain to build by hand, I believe). It has been a pain, but not any longer. win32all can now be built with a distutils setup script - although there is no official release built with that. > Also, you'd need an installer which would sit happily alongside the > production version (install in a separate directory, include a > python24.exe so as not to clash in terms of command names, etc). There's no problem having different major Python versions installed on one machine, even if there are *also* CVS builds. It can all be hidden in batch scripts calling python.exe in different directories. >> Besides increasing community involvement, this could open up a whole new >> stream of user feedback so we can discover issues sooner rather than >> later. Since non-developers stress the system in different ways, they >> are more likely to surface various documentation, usability, and >> integration bugs. > > A compromise may just be a simple zipped up compile. That helps people > without compilers, but leaves the effort of "installing" to the user. > It still doesn't help with the extensions issue, though... Then there's the possibility to include the compiled files (even for different python versions, if you don't care too much about the download size) in a source distribution prepared by distutils. Thomas From eppstein at ics.uci.edu Sun Dec 7 14:46:57 2003 From: eppstein at ics.uci.edu (David Eppstein) Date: Sun Dec 7 14:47:01 2003 Subject: [Python-Dev] Re: test_unicode_file failing on Mac OS X References: <16338.5731.362994.497561@montanaro.dyndns.org> Message-ID: In article , martin@v.loewis.de (Martin v. Lowis) wrote: > For the test, it would be best to compare normal forms, and have the > test pass if the normal forms (NFD) are equal. Shouldn't that be what happens in general for equality testing of unicodes, not just for this test? -- David Eppstein http://www.ics.uci.edu/~eppstein/ Univ. of California, Irvine, School of Information & Computer Science From pf_moore at yahoo.co.uk Sun Dec 7 15:51:39 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Sun Dec 7 15:51:37 2003 Subject: [Python-Dev] Re: Py2.4 pre-alpha snapshot References: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> <20031207132959.GA26733@panix.com> Message-ID: martin@v.loewis.de (Martin v. L?wis) writes: > kbk@shore.net (Kurt B. Kaiser) writes: > >> As I recollect, there was discussion about moving to VC7 which has >> a free compiler available. Is that going to happen on 2.4? > > Yes. If nobody beats me in that, I'll do it over Christmas. What exactly does that imply? Will VC6 no longer work, or will you just add extra build files etc for VC7? How will the free VC7 work (I assume Kurt means the one in the .NET SDK)? AFAIK, it doesn't support Visual Studio project files. It also doesn't include an optimizer, what effect will that have on build compatibility, etc? Paul. -- This signature intentionally left blank From kbk at shore.net Sun Dec 7 17:53:28 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Sun Dec 7 17:53:31 2003 Subject: [Python-Dev] Re: Py2.4 pre-alpha snapshot In-Reply-To: (Paul Moore's message of "Sun, 07 Dec 2003 20:51:39 +0000") References: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> <20031207132959.GA26733@panix.com> Message-ID: Paul Moore writes: > How will the free VC7 work (I assume Kurt means the one in the .NET > SDK)? AFAIK, it doesn't support Visual Studio project files. It also > doesn't include an optimizer, what effect will that have on build > compatibility, etc? Here are the pertinent messages from last spring: -- KBK ===================================================================== From: Guido van Rossum Subject: Re: [Python-Dev] MS VC 7 offer To: phil@riverbankcomputing.co.uk cc: python-dev@python.org Date: Tue, 06 May 2003 14:55:15 -0400 > How do we get hold of the free VC 7 compilers? Here's the info Nick sent me: | We offer as part of the .NET Framework SDK each of the compilers that | comprise our Visual Studio tool - including C++. The caveat here is | that we don't yet ship the full CRT or STL with this distribution - | this will be changing. Also, the 64bit C++ compilers ship for free as | part of the Windows Platform SDK. All of this is available on | msdn.microsoft.com. [...] | Here are the links to the SDKs. But so you aren't surprised, these are | NOT low-overhead downloads or installs... | | .NET Framework 1.1 | | http://www.msdn.microsoft.com/netframework/downloads/howtoget.aspx | | Platform SDK | | http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sdkintro/sdkintro/obtaining_the_complete_sdk.asp --Guido van Rossum (home page: http://www.python.org/~guido/) _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev From: Tim Peters Subject: RE: [Python-Dev] MS VC 7 offer To: phil@riverbankcomputing.co.uk Cc: python-dev@python.org Date: Tue, 06 May 2003 14:56:52 -0400 [Phil Thompson] > How do we get hold of the free VC 7 compilers? Part of the 100+ MB .NET Framework 1.1 SDK: http://www.msdn.microsoft.com/netframework/downloads/howtoget.aspx Note that this requires Win2K minimum. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev From: Chad Netzer Subject: Re: [Python-Dev] MS VC 7 offer To: python-dev@python.org Date: 06 May 2003 15:37:30 -0700 Organization: On Tue, 2003-05-06 at 15:06, Guido van Rossum wrote: > Part of the offer was: > > | Potentially we can even figure out how to enable anyone to > | build Python using the freely downloadable compilers I mentioned > | above... Which would seem to exclude building on Win98 machines (or WinME *snort*, or even Win NT 4). Those platforms still have a huge installed base, and I would assume a not insignificant developer base. Is offering a MSVC6 version along with a more recent compiler version an option? -- Chad Netzer (any opinion expressed is my own and not NASA's or my employer's) =============================================================================== From tdelaney at avaya.com Sun Dec 7 17:57:18 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Sun Dec 7 17:57:26 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEF5FF10@au3010avexu1.global.avaya.com> > From: Aahz > > Hmmmm... How important is the "less memory" angle versus the "faster" > angle? Why not just add a few bytes to the long type to store an int? > If you're using the int, the pointer is NULL, and you're saving that > memory, at least. The only harm is some extra memory in the type > object. One thing I can immediately think of is that you're likely to see cache effects much sooner. In that case, you'll probably find that you've ended up with higher memory usage *and* lower performance. The best of both worlds :) Tim Delaney From greg at cosc.canterbury.ac.nz Sun Dec 7 19:05:58 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun Dec 7 19:06:08 2003 Subject: [Python-Dev] are CObjects inherently unsafe? In-Reply-To: <2mfzfzwmk2.fsf@starship.python.net> Message-ID: <200312080005.hB805wS08108@oma.cosc.canterbury.ac.nz> Michael Hudson : > As things stand, I don't think cPickle *can* be careful enough. > Perhaps we could just add a "name" field to CObjects and have module > init functions check that. That would certainly be a huge improvement over the current situation! Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Sun Dec 7 19:37:06 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun Dec 7 19:37:11 2003 Subject: [Python-Dev] are CObjects inherently unsafe? In-Reply-To: <200312051441.hB5EfKv08101@c-24-5-183-134.client.comcast.net> Message-ID: <200312080037.hB80b6Q08173@oma.cosc.canterbury.ac.nz> Guido: > And the docs should be updated to explain the description better > (currently at one point the description is called "extra callback data > for the destructor function" which seems a rather odd feature). Indeed! That comment completely misled me about the purpose of the "desc". Ironically, if that sentence simply hadn't been there, I probably would have correctly guessed what the desc was meant for. Another API suggestion: a function such as void* PyCObject_AsVoidPtrWithDesc(PyObject* self, void *expected_desc) which checks the desc and raises an exception if it doesn't match the expected value. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Sun Dec 7 19:43:37 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun Dec 7 19:43:42 2003 Subject: [Python-Dev] are CObjects inherently unsafe? In-Reply-To: <2m7k1bwb6a.fsf@starship.python.net> Message-ID: <200312080043.hB80hba08180@oma.cosc.canterbury.ac.nz> Michael Hudson : > Why a void* then? That's particularly unassertive. char* would be > better, surely... A string isn't the only thing that could usefully be used as a descriptor. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From jjl at pobox.com Sun Dec 7 19:51:00 2003 From: jjl at pobox.com (John J Lee) Date: Sun Dec 7 19:51:07 2003 Subject: [Python-Dev] Conventions for functional tests, PyUnit? Message-ID: First, it seems the standard place for functional tests is in if __name__ == "__main__" right? Some (most?) of the standard library functional tests seem to be a bunch of commands that are simply supposed not to crash (urllib2) or that are little test programs that need to be invoked with the correct arguments (ftplib). Would it be a good thing to attempt to PyUnit-ize these functional tests so you can run them with no command line arguments and get clear indications of how many tests are passing and failing? Or, if not PyUnit, just ad-hoc tests that indicate success / failure? Should some be left intact and new functional tests added? For example, ftplib has a little test program tacked on at its end, perhaps that should be retained in some form? John From raymond.hettinger at verizon.net Sun Dec 7 19:59:26 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sun Dec 7 19:59:32 2003 Subject: [Python-Dev] New developer Message-ID: <001601c3bd26$877e15a0$0705a044@oemcomputer> Please welcome Hye-Shik Chang as a new python developer. He hails from Seoul, Korea and has interests in implementing generator expressions and in i18n projects such as CJK codecs. Most recently, he has worked on the code and tests implementing itertools.groupby(). Raymond Hettinger His Bio (slightly edited): """ I work for a linux company for a military service replacement. (Korean military service duty is 2 years but we can work for defense industry or IT company for 3 years instead.) My company sells AAA and DHCP solutions for Korean major ISPs. They are all based on python. And even the ISPs' operators use python interactive interpreter as a command line interface. I'm a commit-privileged developer on FreeBSD project since 2002. I maintain package infrastructure for python stuffs named "bsd.python.mk", the python package itself and FreeBSD packages for 80+ python 3rd party modules. (But it's easy thank to distutils.) After starting to use python on my company, I'm distributing python extension modules I developed as opensource licensed as possible: CJKCodecs, py-rrdtool, py-freebsd, py-fibheap, coolwater and iconvcodec. """ From aahz at pythoncraft.com Sun Dec 7 20:04:54 2003 From: aahz at pythoncraft.com (Aahz) Date: Sun Dec 7 20:04:56 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEF5FF10@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DEF5FF10@au3010avexu1.global.avaya.com> Message-ID: <20031208010454.GB19111@panix.com> On Mon, Dec 08, 2003, Delaney, Timothy C (Timothy) wrote: > From: Aahz >> >> Hmmmm... How important is the "less memory" angle versus the "faster" >> angle? Why not just add a few bytes to the long type to store an int? >> If you're using the int, the pointer is NULL, and you're saving that >> memory, at least. The only harm is some extra memory in the type >> object. > > One thing I can immediately think of is that you're likely to see > cache effects much sooner. In that case, you'll probably find that > you've ended up with higher memory usage *and* lower performance. The > best of both worlds :) While there's generally some truth to that, I have a suspicion that the tricks we'll need to perform in order hide the int/long dichotomy at the Python level will also be cache-unfriendly. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From python at rcn.com Sun Dec 7 20:19:09 2003 From: python at rcn.com (Raymond Hettinger) Date: Sun Dec 7 20:19:16 2003 Subject: [Python-Dev] Conventions for functional tests, PyUnit? In-Reply-To: Message-ID: <001701c3bd29$4878f020$0705a044@oemcomputer> > First, it seems the standard place for functional tests is in > > if __name__ == "__main__" > > right? > > Some (most?) of the standard library functional tests seem to be a bunch > of commands that are simply supposed not to crash (urllib2) or that are > little test programs that need to be invoked with the correct arguments > (ftplib). > > Would it be a good thing to attempt to PyUnit-ize these functional tests > so you can run them with no command line arguments and get clear > indications of how many tests are passing and failing? Or, if not PyUnit, > just ad-hoc tests that indicate success / failure? > > Should some be left intact and new functional tests added? For example, > ftplib has a little test program tacked on at its end, perhaps that should > be retained in some form? Ideally, all repeatable, self verifying tests should be coded using the unittest module and placed in Lib/test. Walter Dorwald is actively working on converting tests to unittest form. At the same time, he is running a coverage analysis tool and adding tests to increase coverage. Also, he is factoring out some tests that are common to multiple modules. If you want to assist, I recommend working with him directly and having him give a 100% review to any proposed changes. For an example of that work, see: www.python.org/sf/736962 Some of tests in the module itself are being left intact for various reasons. For instance, the random module still has a few tests that need a human eye to authenticate. Some, like the ones in urllib2 are normally only run when someone updates the module -- they should be left in for convenience but could also have a counterpart in Lib/test so long as the test were marked to run only when -u network was enabled. As you pointed out, some are run interactively using command line arguments -- these are a development tool and should be left in. Also, some modules need to be left alone because they are maintained elsewhere. Raymond Hettinger From anthony at ekit-inc.com Mon Dec 8 00:01:37 2003 From: anthony at ekit-inc.com (Anthony Baxter) Date: Mon Dec 8 00:01:51 2003 Subject: [Python-Dev] PEP0004: audioop deprecated? Message-ID: <200312080501.hB851bP7000438@maxim.off.ekorp.com> According to PEP-0004 and PEP-0320, the audioop module is deprecated, and scheduled for removal in 2.4. I hadn't noticed this before, but I'd like to ask that it be kept. I've a number of pieces of code using it - both internal (our unified messaging server makes heavy use of it) and external (the shtoom python SIP phone uses it). Anthony From guido at python.org Mon Dec 8 00:08:04 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 8 00:08:11 2003 Subject: [Python-Dev] Int FutureWarnings and other 2.4 TODOs In-Reply-To: Your message of "Sun, 07 Dec 2003 20:04:54 EST." <20031208010454.GB19111@panix.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DEF5FF10@au3010avexu1.global.avaya.com> <20031208010454.GB19111@panix.com> Message-ID: <200312080508.hB8584E23041@c-24-5-183-134.client.comcast.net> I've figured the whole int/short/long thing out now. Jack may be happy to know that it happened on a long bike ride. :-) There will be two classes, and they can both be user-visible, but users will generally deal in terms of the class called 'int', which is the base class. It is called 'int', but its implementation is what we currently call 'long'. The subclass will be called 'short', implemented as what we currently call 'int'. The 'short' class will not be subclassable. The 'int' class can tell the difference between the two representations by looking at the class pointer -- if it's 'short' then the short representation applies. Both will automatically return an object of the other class if the size of the return value requires/allows it. The new 'short', the current 'int', already does this -- the new 'int', currently 'long' will have to be adapted. But (unlike 'short') it doesn't have to be perfect... (If 'int' is subclassed, it may return an instance of the subclass always, to make subclassing int more useful -- "contagiousness" helps in this case I think.) --Guido van Rossum (home page: http://www.python.org/~guido/) From sjoerd at acm.org Mon Dec 8 04:42:15 2003 From: sjoerd at acm.org (Sjoerd Mullender) Date: Mon Dec 8 04:42:19 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/PCbuild make_versioninfo.dsp, 1.1.2.1, 1.1.2.2 In-Reply-To: References: Message-ID: <3FD44777.2090107@acm.org> theller@users.sourceforge.net wrote: > Update of /cvsroot/python/python/dist/src/PCbuild > In directory sc8-pr-cvs1:/tmp/cvs-serv29575 > > Modified Files: > Tag: release23-maint > make_versioninfo.dsp > Log Message: > Sigh. dsp files must be marked as binary files for cvs, otherwise > MSVC isn't able to read them. Thanks to David Rushby. Actually, this is not entirely true. They must be marked with -ko, i.e. CVS keywords must not be expanded. But it is ok if they are treated as text files. -- Sjoerd Mullender From theller at python.net Mon Dec 8 04:59:35 2003 From: theller at python.net (Thomas Heller) Date: Mon Dec 8 04:59:47 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/PCbuild make_versioninfo.dsp, 1.1.2.1, 1.1.2.2 In-Reply-To: <3FD44777.2090107@acm.org> (Sjoerd Mullender's message of "Mon, 08 Dec 2003 10:42:15 +0100") References: <3FD44777.2090107@acm.org> Message-ID: Sjoerd Mullender writes: > theller@users.sourceforge.net wrote: >> Update of /cvsroot/python/python/dist/src/PCbuild >> In directory sc8-pr-cvs1:/tmp/cvs-serv29575 >> Modified Files: >> Tag: release23-maint >> make_versioninfo.dsp Log Message: >> Sigh. dsp files must be marked as binary files for cvs, otherwise >> MSVC isn't able to read them. Thanks to David Rushby. > > Actually, this is not entirely true. > They must be marked with -ko, i.e. CVS keywords must not be > expanded. But it is ok if they are treated as text files. That may be, but the real problem is that MSVC refuses to load them if they have unix line endings. And the source tarball is built on a unix like system. Thomas From theller at python.net Mon Dec 8 05:01:34 2003 From: theller at python.net (Thomas Heller) Date: Mon Dec 8 05:01:40 2003 Subject: [Python-Dev] Py2.4 pre-alpha snapshot In-Reply-To: (Martin v.'s message of "07 Dec 2003 18:50:08 +0100") References: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> <20031207132959.GA26733@panix.com> Message-ID: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) writes: > kbk@shore.net (Kurt B. Kaiser) writes: > >> As I recollect, there was discussion about moving to VC7 which has >> a free compiler available. Is that going to happen on 2.4? > > Yes. If nobody beats me in that, I'll do it over Christmas. How's your msi builder going? I would be interested to try it out - although I probably don't have too much time over Christmas. Thomas From mwh at python.net Mon Dec 8 05:02:32 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 8 05:02:36 2003 Subject: [Python-Dev] are CObjects inherently unsafe? In-Reply-To: <200312080043.hB80hba08180@oma.cosc.canterbury.ac.nz> (Greg Ewing's message of "Mon, 08 Dec 2003 13:43:37 +1300 (NZDT)") References: <200312080043.hB80hba08180@oma.cosc.canterbury.ac.nz> Message-ID: <2mk757tyo7.fsf@starship.python.net> Greg Ewing writes: > Michael Hudson : > >> Why a void* then? That's particularly unassertive. char* would be >> better, surely... > > A string isn't the only thing that could usefully be used > as a descriptor. What else would you consider useful? I'm only really aware of the most straightforward uses of CObjects (and wonder if the advanced uses actually see much use...). Cheers, mhw -- Just point your web browser at http://www.python.org/search/ and look for "program", "doesn't", "work", or "my". Whenever you find someone else whose program didn't work, don't do what they did. Repeat as needed. -- Tim Peters, on python-help, 16 Jun 1998 From jim at zope.com Mon Dec 8 05:03:05 2003 From: jim at zope.com (Jim Fulton) Date: Mon Dec 8 05:03:52 2003 Subject: [Python-Dev] are CObjects inherently unsafe? In-Reply-To: <200312051441.hB5EfKv08101@c-24-5-183-134.client.comcast.net> References: <200312050056.hB50uh722511@oma.cosc.canterbury.ac.nz> <2mfzfzwmk2.fsf@starship.python.net> <200312051441.hB5EfKv08101@c-24-5-183-134.client.comcast.net> Message-ID: <3FD44C59.1070701@zope.com> Guido van Rossum wrote: >>>Indeed, CObjects seem fundamentally dangerous to me, unless >>>the modules which create and use them are extremely careful >>>to make sure they can tell whether they've got the right >>>kind of CObject. >>> >> >>Well, the right *CObject* -- it's CObject identity that matters. > > > I seem to recall that the CObject design contains a feature to avoid > using the wrong CObject, but it's not used by any of the CObject > implementations in the standard library, and the CObject docs don't > describe how it's meant to use (they were clearly reverse-engineered > from the code). > > Several early checkin messages of cobject.c mention Jim Fulton as the > author, so maybe we could ask him. Yup, I'm the one who proposed CObject. > (I'm cc'ing Jim in case he > remembers the true intention for the description; he might also know > whether Zope 2 still uses it; I can only check Zope 3, and it > doesn't.) What "it" are you refering to. I don't really remember what the description was for. I also don't have the original email. > But even without Jim's confirmation, ISTM from memory and looking at > the source that the 'desc' field (which stands for description) I now hate abbreviations like that btw. :) > was > intended as a clue that could be checked by the user of a C Object to > verify it had gotten the right one: in a shared header (needed anyway > to describe the layout of the struct pointer contained in the C > Object) you could define a magic word or a magic string value that > would be stored as the description, and the users would check that > value before believing the struct pointer. I vaguely remember that such a clue was one possible application of the extra data. I really imagined that the destructor could need the extra data, but, again, it's been a long time. Too bad there's not a PEP. :) > Unfortunately the same CObject version that introduced the description > field also introduced the convenience function PyCObject_Import(), > which doesn't return the description, and returns the void pointer > contained in the CObject without returning the CObject itself, leaving > its caller without a clue about the description... That's because I don't think that the description was primarily intended as a fingerprint. > Anyway, I think it can be fixed by starting to use the description > field and adding a new convenience C API, perhaps named > PyCObject_ImportEx(), whch takes a description and checks it. We > should try to standardize on what kind of thing to use as the > description -- if we use a magic word, PyCObject_ImportEx() should do > a simple compare, but if we use a string, it should do a strcmp(). Or > the API could simply return the description into an output parameter, > making the caller responsible for doing the checking. > > And the docs should be updated to explain the description better > (currently at one point the description is called "extra callback data > for the destructor function" which seems a rather odd feature). I like the idea of adding this use of the description. That is, the description is really a marker, that is obtained from an include file and checked against the value in the CObject. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From mwh at python.net Mon Dec 8 05:30:06 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 8 05:30:10 2003 Subject: [Python-Dev] are CObjects inherently unsafe? In-Reply-To: <3FD44C59.1070701@zope.com> (Jim Fulton's message of "Mon, 08 Dec 2003 05:03:05 -0500") References: <200312050056.hB50uh722511@oma.cosc.canterbury.ac.nz> <2mfzfzwmk2.fsf@starship.python.net> <200312051441.hB5EfKv08101@c-24-5-183-134.client.comcast.net> <3FD44C59.1070701@zope.com> Message-ID: <2mk757sitt.fsf@starship.python.net> Jim Fulton writes: > > (I'm cc'ing Jim in case he >> remembers the true intention for the description; he might also know >> whether Zope 2 still uses it; I can only check Zope 3, and it >> doesn't.) > > What "it" are you refering to. The desc field in a CObject. > I don't really remember what the description was for. I also > don't have the original email. http://mail.python.org/pipermail/python-dev/2003-December/040702.html >> But even without Jim's confirmation, ISTM from memory and looking at >> the source that the 'desc' field (which stands for description) > > I now hate abbreviations like that btw. :) Good :-) > > was >> intended as a clue that could be checked by the user of a C Object to >> verify it had gotten the right one: in a shared header (needed anyway >> to describe the layout of the struct pointer contained in the C >> Object) you could define a magic word or a magic string value that >> would be stored as the description, and the users would check that >> value before believing the struct pointer. > > I vaguely remember that such a clue was one possible application > of the extra data. I really imagined that the destructor could need the > extra data, but, again, it's been a long time. Too bad there's not a PEP. :) > >> Unfortunately the same CObject version that introduced the description >> field also introduced the convenience function PyCObject_Import(), >> which doesn't return the description, and returns the void pointer >> contained in the CObject without returning the CObject itself, leaving >> its caller without a clue about the description... > > That's because I don't think that the description was primarily intended > as a fingerprint. I feel a retcon coming on. >> Anyway, I think it can be fixed by starting to use the description >> field and adding a new convenience C API, perhaps named >> PyCObject_ImportEx(), whch takes a description and checks it. We >> should try to standardize on what kind of thing to use as the >> description -- if we use a magic word, PyCObject_ImportEx() should do >> a simple compare, but if we use a string, it should do a strcmp(). Or >> the API could simply return the description into an output parameter, >> making the caller responsible for doing the checking. >> And the docs should be updated to explain the description better >> (currently at one point the description is called "extra callback data >> for the destructor function" which seems a rather odd feature). > > I like the idea of adding this use of the description. That is, > the description is really a marker, that is obtained from an include file > and checked against the value in the CObject. Right. What should we use as the description object? A good old C string? Seems sensible enough to me, but Greg Ewing seemed to have some other use in mind for it... Cheers, mwh -- 6. The code definitely is not portable - it will produce incorrect results if run from the surface of Mars. -- James Bonfield, http://www.ioccc.org/2000/rince.hint From fredrik at pythonware.com Mon Dec 8 05:54:31 2003 From: fredrik at pythonware.com (Fredrik Lundh) Date: Mon Dec 8 05:54:39 2003 Subject: [Python-Dev] Re: are CObjects inherently unsafe? References: <200312080043.hB80hba08180@oma.cosc.canterbury.ac.nz> <2mk757tyo7.fsf@starship.python.net> Message-ID: Michael Hudson wrote: > > A string isn't the only thing that could usefully be used > > as a descriptor. > > What else would you consider useful? I'm only really aware of the > most straightforward uses of CObjects (and wonder if the advanced uses > actually see much use...). the descriptor is passed to the destructor, so it can be used to hold data associated with the object. checking our repository, I cannot find a single example where I haven't used the descriptor at all; the most common use is to point to a "MAGIC" string, and there's one case where I've used it to keep a reference to a PyObject that's related to the CObject pointer. From fredrik at pythonware.com Mon Dec 8 05:59:18 2003 From: fredrik at pythonware.com (Fredrik Lundh) Date: Mon Dec 8 06:00:24 2003 Subject: [Python-Dev] Re: are CObjects inherently unsafe? References: <200312051441.hB5EfKv08101@c-24-5-183-134.client.comcast.net> <200312080037.hB80b6Q08173@oma.cosc.canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Another API suggestion: a function such as > > void* PyCObject_AsVoidPtrWithDesc(PyObject* self, void *expected_desc) > > which checks the desc and raises an exception if it doesn't > match the expected value. which should probably be void* PyCObject_AsVoidPtrWithStringDesc(PyObject* self, const char *expected_desc) /* uses desc && !strcmp(desc, expected_desc), and can still crash... */ void* PyCObject_AsVoidPtrWithPtrDesc(PyObject* self, void *expected_desc) /* uses desc == expected_desc */ From jim at zope.com Mon Dec 8 07:00:32 2003 From: jim at zope.com (Jim Fulton) Date: Mon Dec 8 07:01:20 2003 Subject: [Python-Dev] are CObjects inherently unsafe? In-Reply-To: <2mk757sitt.fsf@starship.python.net> References: <200312050056.hB50uh722511@oma.cosc.canterbury.ac.nz> <2mfzfzwmk2.fsf@starship.python.net> <200312051441.hB5EfKv08101@c-24-5-183-134.client.comcast.net> <3FD44C59.1070701@zope.com> <2mk757sitt.fsf@starship.python.net> Message-ID: <3FD467E0.4030508@zope.com> Michael Hudson wrote: > Jim Fulton writes: > > >> > (I'm cc'ing Jim in case he >> >>>remembers the true intention for the description; he might also know >>>whether Zope 2 still uses it; I can only check Zope 3, and it >>>doesn't.) >> >>What "it" are you refering to. > > > The desc field in a CObject. > > >>I don't really remember what the description was for. I also >>don't have the original email. > > > http://mail.python.org/pipermail/python-dev/2003-December/040702.html > > >>>But even without Jim's confirmation, ISTM from memory and looking at >>>the source that the 'desc' field (which stands for description) >> >>I now hate abbreviations like that btw. :) > > > Good :-) > > >> > was >> >>>intended as a clue that could be checked by the user of a C Object to >>>verify it had gotten the right one: in a shared header (needed anyway >>>to describe the layout of the struct pointer contained in the C >>>Object) you could define a magic word or a magic string value that >>>would be stored as the description, and the users would check that >>>value before believing the struct pointer. >> >>I vaguely remember that such a clue was one possible application >>of the extra data. I really imagined that the destructor could need the >>extra data, but, again, it's been a long time. Too bad there's not a PEP. :) >> >> >>>Unfortunately the same CObject version that introduced the description >>>field also introduced the convenience function PyCObject_Import(), >>>which doesn't return the description, and returns the void pointer >>>contained in the CObject without returning the CObject itself, leaving >>>its caller without a clue about the description... >> >>That's because I don't think that the description was primarily intended >>as a fingerprint. > > > I feel a retcon coming on. > > >>>Anyway, I think it can be fixed by starting to use the description >>>field and adding a new convenience C API, perhaps named >>>PyCObject_ImportEx(), whch takes a description and checks it. We >>>should try to standardize on what kind of thing to use as the >>>description -- if we use a magic word, PyCObject_ImportEx() should do >>>a simple compare, but if we use a string, it should do a strcmp(). Or >>>the API could simply return the description into an output parameter, >>>making the caller responsible for doing the checking. >>>And the docs should be updated to explain the description better >>>(currently at one point the description is called "extra callback data >>>for the destructor function" which seems a rather odd feature). >> >>I like the idea of adding this use of the description. That is, >>the description is really a marker, that is obtained from an include file >>and checked against the value in the CObject. > > > Right. What should we use as the description object? A good old C > string? Probably. I don't have a strong opinion on it. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From vinay_sajip at red-dove.com Mon Dec 8 07:05:33 2003 From: vinay_sajip at red-dove.com (Vinay Sajip) Date: Mon Dec 8 07:03:19 2003 Subject: [Python-Dev] source-level backward compatibility requirements - logging and apply() Message-ID: <00f101c3bd83$96117dc0$652b6992@alpha> Sorry that this post starts a new thread - I have python-dev on digest mode and have not been following the list as closely as I ought to have done. [Guido] >> We could deprecate it in the docs but remove the silent warning which >> slows it down. Then we could leave it in the logging module until >> 3.0. [Fred] >+42 >Does this warrant a c2 release? I'm happy to make the changes right >now. +1 on deprecating apply() in the docs but not in the code. As Raymond originally said, the only "problem" with apply() and coerce() is that while they are useful, they are not useful in "mainstream" situations and could use up learning bandwidth. However, I think apply() is a useful idiom to have at one's disposal, though it only finds application here and there. If Raymond's "all-builtins-have-at-least-one-friend" comment is not true for apply(), I'd like to nominate myself as that friend :-) Regards, Vinay Sajip From tim.one at comcast.net Mon Dec 8 11:17:00 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 8 11:16:59 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/PCbuild make_versioninfo.dsp, 1.1.2.1, 1.1.2.2 In-Reply-To: <3FD44777.2090107@acm.org> Message-ID: >> Sigh. dsp files must be marked as binary files for cvs, otherwise >> MSVC isn't able to read them. Thanks to David Rushby. [Sjoerd Mullender, on Windows .dsp files] > Actually, this is not entirely true. > They must be marked with -ko, i.e. CVS keywords must not be expanded. > But it is ok if they are treated as text files. We used to, and that doesn't work -- someone ends up editing a .dsp file on Linux then, or builds a distro from Linux, and the \r\n line ends get lost. MSVC indeed does not tolerate a .dsp file with plain \n line ends. Since the exact line end characters used are crucial, it really is a binary file. From martin at v.loewis.de Mon Dec 8 12:46:14 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Mon Dec 8 12:46:40 2003 Subject: [Python-Dev] Re: Py2.4 pre-alpha snapshot In-Reply-To: References: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> <20031207132959.GA26733@panix.com> Message-ID: Paul Moore writes: > What exactly does that imply? Will VC6 no longer work, or will you > just add extra build files etc for VC7? I will remove the VC6 project files, and restore them in a different place. They will become unmaintained. > How will the free VC7 work (I assume Kurt means the one in the .NET > SDK)? AFAIK, it doesn't support Visual Studio project files. It also > doesn't include an optimizer, what effect will that have on build > compatibility, etc? I don't know, I don't use the VC7 SDK compiler. I cannot understand your question on build compatibility, etc. Regards, Martin From martin at v.loewis.de Mon Dec 8 12:50:01 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Mon Dec 8 12:54:22 2003 Subject: [Python-Dev] Py2.4 pre-alpha snapshot In-Reply-To: References: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> <20031207132959.GA26733@panix.com> Message-ID: Thomas Heller writes: > > Yes. If nobody beats me in that, I'll do it over Christmas. > > How's your msi builder going? I would be interested to try it out - > although I probably don't have too much time over Christmas. I got so far that I can initialize the schema, i.e. I have code that takes the current schema, and dumps it into a Python source file, which, when run, will restore the schema. This includes data for the validation and install sequence tables. I was next looking at including the actual files, and was planning to supply a single CAB stream. I got stuck finding a CAB API, and plan to invoke cabarc.exe instead to generate the CAB file. This was a couple of weeks ago, I have made no progress since. Regards, Martin From martin at v.loewis.de Mon Dec 8 12:53:37 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Mon Dec 8 12:55:05 2003 Subject: [Python-Dev] Re: test_unicode_file failing on Mac OS X In-Reply-To: References: <16338.5731.362994.497561@montanaro.dyndns.org> Message-ID: David Eppstein writes: > > For the test, it would be best to compare normal forms, and have the > > test pass if the normal forms (NFD) are equal. > > Shouldn't that be what happens in general for equality testing of > unicodes, not just for this test? There was a BDFL pronouncement once that this should not be done automatically, in general. Normalization is a very slow algorithm, and it might not be meaningful in all cases. E.g. XML 1.1 will require that all documents are in NFC. So the XML-generating application will have to normalize on output; all XML-processing applications can then assume that all strings are normalized. Converting them to NFD all the time would be wasteful. Python is still lacking an efficient test function to determine whether a string is in normal form already; reportedly, a yes-no-maybe function with a reasonably slow rate of maybe answers can be implemented much more efficiently than performing the actual conversion. Regards, Martin From martin at v.loewis.de Mon Dec 8 12:57:23 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Mon Dec 8 12:58:55 2003 Subject: [Python-Dev] New developer In-Reply-To: <001601c3bd26$877e15a0$0705a044@oemcomputer> References: <001601c3bd26$877e15a0$0705a044@oemcomputer> Message-ID: "Raymond Hettinger" writes: > Please welcome Hye-Shik Chang as a new python developer. > > He hails from Seoul, Korea and has interests in implementing generator > expressions and in i18n projects such as CJK codecs. Most recently, he > has worked on the code and tests implementing itertools.groupby(). Does that mean we get CJKCodecs in 2.4 ?-) (I'm actually serious here; I would welcome this addition) Regards, Martin From barry at python.org Mon Dec 8 13:37:48 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 8 13:37:53 2003 Subject: [Python-Dev] New developer In-Reply-To: References: <001601c3bd26$877e15a0$0705a044@oemcomputer> Message-ID: <1070908667.25704.17.camel@anthem> On Mon, 2003-12-08 at 12:57, Martin v. L?wis wrote: > Does that mean we get CJKCodecs in 2.4 ?-) > > (I'm actually serious here; I would welcome this addition) Me too! -Barry From perky at i18n.org Mon Dec 8 14:35:46 2003 From: perky at i18n.org (Hye-Shik Chang) Date: Mon Dec 8 14:35:52 2003 Subject: [Python-Dev] New developer In-Reply-To: References: <001601c3bd26$877e15a0$0705a044@oemcomputer> Message-ID: <20031208193546.GA17282@i18n.org> On Mon, Dec 08, 2003 at 06:57:23PM +0100, Martin v. L?wis wrote: > "Raymond Hettinger" writes: > > > Please welcome Hye-Shik Chang as a new python developer. > > > > He hails from Seoul, Korea and has interests in implementing generator > > expressions and in i18n projects such as CJK codecs. Most recently, he > > has worked on the code and tests implementing itertools.groupby(). Hello! I'm him. :) As Raymond said, I have interests in implementing PEP 289 as well as i18n stuffs such as CJK codecs or richer file.encoding support. I'm currently doing some reviews for list.sort and str.rsplit following Raymond's worth instructions. > Does that mean we get CJKCodecs in 2.4 ?-) > > (I'm actually serious here; I would welcome this addition) I'd like. CJKCodecs got enough compatibility and standard conformance to be merged into python to my thinking. And source size (500kB in tar.gz) and installed binary size (1500kB on i386) may be reasonable enough for most of people. Hye-Shik From walter at livinglogic.de Mon Dec 8 18:16:26 2003 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon Dec 8 18:16:43 2003 Subject: [Python-Dev] Conventions for functional tests, PyUnit? In-Reply-To: <001701c3bd29$4878f020$0705a044@oemcomputer> References: <001701c3bd29$4878f020$0705a044@oemcomputer> Message-ID: <3FD5064A.7020201@livinglogic.de> Raymond Hettinger wrote: > [...] > Ideally, all repeatable, self verifying tests should be coded using the > unittest module and placed in Lib/test. > > Walter Dorwald is actively working on converting tests to unittest form. > At the same time, he is running a coverage analysis tool and adding > tests to increase coverage. BTW, the results of the coverage tests are available at: http://coverage.livinglogic.de/ Currently updates are done once a month. > Also, he is factoring out some tests that > are common to multiple modules. This not only reduces the number of tests that have to be written, but it helps to keep similar APIs consistent. > [...] Bye, Walter D?rwald From vinay_sajip at red-dove.com Mon Dec 8 18:40:02 2003 From: vinay_sajip at red-dove.com (Vinay Sajip) Date: Mon Dec 8 18:37:49 2003 Subject: [Python-Dev] Source-level backward compatibility requirements - logging and apply() - addendum Message-ID: <002901c3bde4$9b069980$652b6992@alpha> > If Raymond's "all-builtins-have-at-least-one-friend" comment is not true for > apply(), I'd like to nominate myself as that friend :-) Raymond has very tactfully shown me the error of my ways, with respect to the now-I-see-it-was-misplaced friendship with apply(). I hereby cut it loose from me, and shall shed no tears when it departs in Python 3.0. Regards, Vinay From fredrik at pythonware.com Tue Dec 9 03:21:22 2003 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue Dec 9 03:21:34 2003 Subject: [Python-Dev] Re: Source-level backward compatibility requirements -logging and apply() - addendum References: <002901c3bde4$9b069980$652b6992@alpha> Message-ID: Vinay Sajip wrote_ > Raymond has very tactfully shown me the error of my ways, with respect to > the now-I-see-it-was-misplaced friendship with apply(). I hereby cut it > loose from me, and shall shed no tears when it departs in Python 3.0. can python-dev perhaps provide a downloadable version of Raymond, that I can bring to my Python training sessions? in my experience, you always lose your audience when you introduce the callable(*args, **kwargs) syntax, and you don't get them back until you explain that the notion is apply(callable, args, kwargs). whoever decided that the new calling syntax is "better" obviously didn't spend many seconds on user testing. a foolish consistency etc. From theller at python.net Tue Dec 9 06:41:44 2003 From: theller at python.net (Thomas Heller) Date: Tue Dec 9 06:41:52 2003 Subject: [Python-Dev] Py2.4 pre-alpha snapshot In-Reply-To: (Martin v.'s message of "08 Dec 2003 18:50:01 +0100") References: <000201c3bcbf$2d1c2320$f705a044@oemcomputer> <20031207132959.GA26733@panix.com> Message-ID: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) writes: > Thomas Heller writes: > >> > Yes. If nobody beats me in that, I'll do it over Christmas. >> >> How's your msi builder going? I would be interested to try it out - >> although I probably don't have too much time over Christmas. > > I got so far that I can initialize the schema, i.e. I have code that > takes the current schema, and dumps it into a Python source file, > which, when run, will restore the schema. This includes data for the > validation and install sequence tables. > > I was next looking at including the actual files, and was planning to > supply a single CAB stream. I got stuck finding a CAB API, and plan to > invoke cabarc.exe instead to generate the CAB file. I hacked together a very simple cab creator using ctypes , maybe you find it useful - there's also a link to the Cabinet SDK on the page, which is not required but contains the docs (it is only 400kB or so). http://starship.python.net/crew/theller/moin.cgi/CreateCab > This was a couple of weeks ago, I have made no progress since. > Thomas From smi at srasys.co.in Tue Dec 9 07:28:46 2003 From: smi at srasys.co.in (Mythili Madhav) Date: Tue Dec 9 07:26:48 2003 Subject: [Python-Dev] Help on MSAA Message-ID: <002d01c3be4f$fe0f5f60$980110ac@3mts.com> Hi, I need to write a wrapper around MSAA in Python for having my own control classes like MyApplication, MyWindow, MyPushButton etc which I think is basically to subclass from MSAA's classes. I have downloaded pyAA but I don't know how and from what to subclass. Has anybody worked on this? If yes, kindly revert. Thanks in advance, SMI -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031209/7336d06c/attachment.html From karthik19_20 at yahoo.com Tue Dec 9 07:43:15 2003 From: karthik19_20 at yahoo.com (karthik karthik) Date: Tue Dec 9 07:43:26 2003 Subject: [Python-Dev] help on MSAA Message-ID: <20031209124315.94658.qmail@web11905.mail.yahoo.com> hi!! my task is to develop a wrapper to the MSAA using python. for this i'm developing a module. the class design is: class Wrapper: class Application: class Window: class PushButton: class EditBox: in my main module i create an object for the Wrapper class and when i say: wrapper.Application, it should enable me to access the application.. similarly if i say, wrapper.PushButton, it should enable me to access the pushbutton... in other words its a generic class tht gives me the respective handlers. i inherit this class and user for specific purposes. thank u. sha. __________________________________ Do you Yahoo!? New Yahoo! Photos - easier uploading and sharing. http://photos.yahoo.com/ From aahz at pythoncraft.com Tue Dec 9 08:03:51 2003 From: aahz at pythoncraft.com (Aahz) Date: Tue Dec 9 08:03:54 2003 Subject: [Python-Dev] help on MSAA In-Reply-To: <20031209124315.94658.qmail@web11905.mail.yahoo.com> References: <20031209124315.94658.qmail@web11905.mail.yahoo.com> Message-ID: <20031209130351.GA22221@panix.com> On Tue, Dec 09, 2003, karthik karthik wrote: > > my task is to develop a wrapper to the MSAA using python. for this i'm > developing a module. the class design is: python-dev is for discussion of features and bugfixes for Python, not creating programs in Python. Please restrict future posts like this to comp.lang.python (or the python-list gateway). -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From Jack.Jansen at cwi.nl Tue Dec 9 08:25:40 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Tue Dec 9 08:25:11 2003 Subject: [Python-Dev] Re: Source-level backward compatibility requirements -logging and apply() - addendum In-Reply-To: References: <002901c3bde4$9b069980$652b6992@alpha> Message-ID: <2F1E9273-2A4B-11D8-A5D7-0030655234CE@cwi.nl> On 9 Dec 2003, at 9:21, Fredrik Lundh wrote: > in my experience, you always lose your audience when you introduce the > callable(*args, **kwargs) syntax, and you don't get them back until you > explain that the notion is apply(callable, args, kwargs). This is a good point. For most other language constructs that we have considered dropping the new construct was more readable than the old one (think repr() versus backticks) but for this one that isn't true: apply() is much more readable than the * and ** notation in some situations. In other situations the * notation is much easier, I really don't want to write callable(1, a=2, *args, **kwargs) in apply notation anymore. But still apply is easy to understand for easy cases. -- Jack Jansen http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From fredrik at pythonware.com Tue Dec 9 08:31:35 2003 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue Dec 9 08:31:47 2003 Subject: [Python-Dev] Re: Source-level backward compatibility requirements-logging and apply() - addendum References: <002901c3bde4$9b069980$652b6992@alpha> Message-ID: > in my experience, you always lose your audience when you introduce the > callable(*args, **kwargs) syntax, and you don't get them back until you > explain that the notion is apply(callable, args, kwargs). missing an "old" in there: > explain that the old notion is apply(callable, args, kwargs). From Scott.Daniels at Acm.Org Wed Dec 10 08:28:46 2003 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Tue Dec 9 09:27:10 2003 Subject: [Python-Dev] Re: test_unicode_file failing on Mac OS X In-Reply-To: References: Message-ID: <3FD71F8E.6030307@Acm.Org> martin@v.loewis.de (Martin v. L?wis) wrote: > David Eppstein writes: >>>For the test, it would be best to compare normal forms, and have the >>>test pass if the normal forms (NFD) are equal. >>Shouldn't that be what happens in general for equality testing of >>unicodes, not just for this test? > There was a BDFL pronouncement once that this should not be done > automatically, in general. Normalization is a very slow algorithm, and > it might not be meaningful in all cases. Could we perhaps use a comparison that, in effect, did: def uni_equal(first, second): if first == second: return True return first.normalize() == second.normalize() That is, take advantage of the fact that normalization is often unnecessary for "trivial" reasons. -Scott David Daniels Scott.Daniels@Acm.Org From jjl at pobox.com Tue Dec 9 11:01:53 2003 From: jjl at pobox.com (John J Lee) Date: Tue Dec 9 11:02:05 2003 Subject: [Python-Dev] Conventions for functional tests, PyUnit? In-Reply-To: <001701c3bd29$4878f020$0705a044@oemcomputer> References: <001701c3bd29$4878f020$0705a044@oemcomputer> Message-ID: On Sun, 7 Dec 2003, Raymond Hettinger wrote: [...] > Ideally, all repeatable, self verifying tests should be coded using the > unittest module and placed in Lib/test. [...] > Some of tests in the module itself are being left intact for various > reasons. For instance, the random module still has a few tests that > need a human eye to authenticate. Some, like the ones in urllib2 are > normally only run when someone updates the module -- they should be left > in for convenience but could also have a counterpart in Lib/test so long > as the test were marked to run only when -u network was enabled. As you (Ah, hadn't seen the test runner script, so didn't know about -u network, thanks.) OK, so I can PyUnit-ize the urllib2 functional tests (but leave them there so functional tests can conveniently be run separately), and link them to Lib/test (so all self-verifying tests get run there). One problem, though: doesn't putting functional tests in Lib/test throw off Walter Dorwald's unit test coverage numbers? Perhaps there should be a 'functional' resource for test_support.use_resources (so the tests can be run with -uall, -functional for coverage measurements)? Thanks John From Jack.Jansen at cwi.nl Tue Dec 9 11:31:34 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Tue Dec 9 11:31:44 2003 Subject: [Python-Dev] Re: test_unicode_file failing on Mac OS X In-Reply-To: <3FD71F8E.6030307@Acm.Org> References: <3FD71F8E.6030307@Acm.Org> Message-ID: <275ADB90-2A65-11D8-BACE-000A27B19B96@cwi.nl> On 10-dec-03, at 14:28, Scott David Daniels wrote: > martin@v.loewis.de (Martin v. L?wis) wrote: >> David Eppstein writes: >>>> For the test, it would be best to compare normal forms, and have the >>>> test pass if the normal forms (NFD) are equal. >>> Shouldn't that be what happens in general for equality testing of >>> unicodes, not just for this test? >> There was a BDFL pronouncement once that this should not be done >> automatically, in general. Normalization is a very slow algorithm, and >> it might not be meaningful in all cases. > > Could we perhaps use a comparison that, in effect, did: > > def uni_equal(first, second): > if first == second: > return True > return first.normalize() == second.normalize() > > That is, take advantage of the fact that normalization is often > unnecessary for "trivial" reasons. It helps, but only in 50% of the comparisons:-) -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From eppstein at ics.uci.edu Tue Dec 9 12:06:03 2003 From: eppstein at ics.uci.edu (David Eppstein) Date: Tue Dec 9 12:06:06 2003 Subject: [Python-Dev] Re: test_unicode_file failing on Mac OS X References: <3FD71F8E.6030307@Acm.Org> <275ADB90-2A65-11D8-BACE-000A27B19B96@cwi.nl> Message-ID: In article <275ADB90-2A65-11D8-BACE-000A27B19B96@cwi.nl>, Jack Jansen wrote: > > Could we perhaps use a comparison that, in effect, did: > > > > def uni_equal(first, second): > > if first == second: > > return True > > return first.normalize() == second.normalize() > > > > That is, take advantage of the fact that normalization is often > > unnecessary for "trivial" reasons. > > It helps, but only in 50% of the comparisons:-) One could imagine a test that also quickly shortcutted most of the unequal comparisons. A possibly bigger problem with allowing a more general definition of equality for unicodes might be that it could be incompatible with the ordering given by < and <=. -- David Eppstein http://www.ics.uci.edu/~eppstein/ Univ. of California, Irvine, School of Information & Computer Science From guido at python.org Tue Dec 9 12:23:10 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 9 12:23:18 2003 Subject: [Python-Dev] Re: Source-level backward compatibility requirements -logging and apply() - addendum In-Reply-To: Your message of "Tue, 09 Dec 2003 09:21:22 +0100." References: <002901c3bde4$9b069980$652b6992@alpha> Message-ID: <200312091723.hB9HNAM25939@c-24-5-183-134.client.comcast.net> > in my experience, you always lose your audience when you introduce the > callable(*args, **kwargs) syntax, and you don't get them back until you > explain that the notion is apply(callable, args, kwargs). Maybe it depends on the audience? In my experience (admitted it was a while ago), the desire to call a function with arguments that are already given as an array often comes to relative newbies who've never heard of apply in any other language (isn't it a Lisp thing?). Sometimes they even ask why a = (1, 2, 3) f(a) isn't equivalent to f(1, 2, 3) and telling them about f(*a) usually makes them very happy. --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Tue Dec 9 12:29:45 2003 From: aahz at pythoncraft.com (Aahz) Date: Tue Dec 9 12:29:48 2003 Subject: [Python-Dev] Re: Source-level backward compatibility requirements -logging and apply() - addendum In-Reply-To: <200312091723.hB9HNAM25939@c-24-5-183-134.client.comcast.net> References: <002901c3bde4$9b069980$652b6992@alpha> <200312091723.hB9HNAM25939@c-24-5-183-134.client.comcast.net> Message-ID: <20031209172944.GA22672@panix.com> On Tue, Dec 09, 2003, Guido van Rossum wrote: >Fredrik >> >> in my experience, you always lose your audience when you introduce the >> callable(*args, **kwargs) syntax, and you don't get them back until you >> explain that the notion is apply(callable, args, kwargs). > > Maybe it depends on the audience? > > In my experience (admitted it was a while ago), the desire to call a > function with arguments that are already given as an array often comes > to relative newbies who've never heard of apply in any other language > (isn't it a Lisp thing?). There's also the issue that IME teaching apply() usually comes after teaching def foo(*args, **kwargs): which makes the symmetry easier to understand, IMO. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From martin at v.loewis.de Tue Dec 9 14:51:45 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Dec 9 14:52:25 2003 Subject: [Python-Dev] Re: test_unicode_file failing on Mac OS X In-Reply-To: <3FD71F8E.6030307@Acm.Org> References: <3FD71F8E.6030307@Acm.Org> Message-ID: Scott David Daniels writes: > Could we perhaps use a comparison that, in effect, did: > > def uni_equal(first, second): > if first == second: > return True > return first.normalize() == second.normalize() > > That is, take advantage of the fact that normalization is often > unnecessary for "trivial" reasons. It also affects hashing, if Unicode objects are used as dictionary keys. Objects that compare equal need to hash equal. Regards, Martin From walter at livinglogic.de Tue Dec 9 16:20:01 2003 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Tue Dec 9 16:20:27 2003 Subject: [Python-Dev] Conventions for functional tests, PyUnit? In-Reply-To: References: <001701c3bd29$4878f020$0705a044@oemcomputer> Message-ID: <3FD63C81.20506@livinglogic.de> John J Lee wrote: > On Sun, 7 Dec 2003, Raymond Hettinger wrote: > [...] > >>Ideally, all repeatable, self verifying tests should be coded using the >>unittest module and placed in Lib/test. > > [...] > >>Some of tests in the module itself are being left intact for various >>reasons. For instance, the random module still has a few tests that >>need a human eye to authenticate. Some, like the ones in urllib2 are >>normally only run when someone updates the module -- they should be left >>in for convenience but could also have a counterpart in Lib/test so long >>as the test were marked to run only when -u network was enabled. As you > > > (Ah, hadn't seen the test runner script, so didn't know about -u network, > thanks.) > > OK, so I can PyUnit-ize the urllib2 functional tests (but leave them there > so functional tests can conveniently be run separately), and link them to > Lib/test (so all self-verifying tests get run there). Personally I'd prefer all tests to be in the test scripts. Otherwise they clutter up the module without serving any real purpose (except maybe as documentation). > One problem, though: doesn't putting functional tests in Lib/test throw > off Walter Dorwald's unit test coverage numbers? Perhaps there should be > a 'functional' resource for test_support.use_resources (so the tests can > be run with -uall, -functional for coverage measurements)? What exactly do you mean with 'functional' tests? We certainly don't need non-functional tests! ;) And why would additional test scripts throw off the coverage numbers? Bye, Walter D?rwald From greg at cosc.canterbury.ac.nz Tue Dec 9 19:40:06 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Dec 9 19:40:12 2003 Subject: [Python-Dev] Re: Source-level backward compatibility requirements -logging and apply() - addendum In-Reply-To: Message-ID: <200312100040.hBA0e6225446@oma.cosc.canterbury.ac.nz> Fredrik Lundh : > in my experience, you always lose your audience when you introduce the > callable(*args, **kwargs) syntax, and you don't get them back until you > explain that the notion is apply(callable, args, kwargs). What sort of example are you using to introduce it? Have you taught them about the * and ** syntaxes in function definitions first? The main use case motivating it is passing on * and ** arguments to an overridden method, and when seen in that context it looks almost self-explanatory, in my opinion. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From raymond.hettinger at verizon.net Tue Dec 9 21:55:55 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue Dec 9 21:56:26 2003 Subject: [Python-Dev] PEP 292 and templating Message-ID: <001601c3bec9$2f258d80$e841fea9@oemcomputer> Is there interest in having a templating module with two functions one for simple substitutions and the other with more tools? The first would be Barry's simple substitutions using only $name or ${name} for templates exposed to the user. The second would extend the first with Cheetah style dotted names for more advanced templates controlled by the programmer. Raymond Hettinger -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031209/c8aada22/attachment.html From fredrik at pythonware.com Wed Dec 10 03:34:47 2003 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed Dec 10 03:35:04 2003 Subject: [Python-Dev] Re: PEP 292 and templating References: <001601c3bec9$2f258d80$e841fea9@oemcomputer> Message-ID: Raymond Hettinger wrote: > The first would be Barry's simple substitutions using only > $name or ${name} for templates exposed to the user. > > The second would extend the first with Cheetah style > dotted names for more advanced templates controlled > by the programmer. time for a "retools" module, perhaps? some code snippets to consider: http://www.effbot.org/zone/re-sub.htm http://www.effbot.org/zone/template-language.htm a collection of "cookbook REs" would also be nice. any takers? From kdart at kdart.com Wed Dec 10 05:23:02 2003 From: kdart at kdart.com (Keith Dart) Date: Wed Dec 10 05:23:08 2003 Subject: [Python-Dev] Greetings Message-ID: <1071051781.14243.25.camel@leviathan.kdart.com> Hello all. I just joined this list, but I have been a full-time Python developer for several years now. My goal here, if possible, would be to contribute some of my code to the the standard library. I have some more time now to clean up and contribute some of my own library code since I am, alas, unemployed. I look forward to working with you all. Sincerely, Keith Dart -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031210/4f4e6bff/attachment.bin From Jack.Jansen at cwi.nl Wed Dec 10 06:59:23 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Wed Dec 10 06:58:59 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> References: <200312041341.hB4DfLcp015502@maxim.off.ekorp.com> Message-ID: <4B7BF5A7-2B08-11D8-A7AA-0030655234CE@cwi.nl> Anthony, I would like to do a 2.3.3 release for MacOS9 this time, so after you've finished with the general 2.3.3 release could you please hand over ownership of the branch to me, and tell the crowd to refrain from fiddling with the branch until I'm done? Thanks, -- Jack Jansen http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From anthony at ekit-inc.com Wed Dec 10 08:31:50 2003 From: anthony at ekit-inc.com (Anthony Baxter) Date: Wed Dec 10 08:32:16 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: Message from Jack Jansen of "Wed, 10 Dec 2003 12:59:23 BST." <4B7BF5A7-2B08-11D8-A7AA-0030655234CE@cwi.nl> Message-ID: <200312101331.hBADVo7A019516@maxim.off.ekorp.com> >>> Jack Jansen wrote > Anthony, > I would like to do a 2.3.3 release for MacOS9 this time, so after > you've finished > with the general 2.3.3 release could you please hand over ownership > of the branch to me, and tell the crowd to refrain from fiddling with > the branch until I'm done? Not a problem. Were you going to do an OS X release as well? Assuming we get the 2.3.3 release on Friday the 19th (AU time), how long do you anticipate you'll want the branch for? (Guesstimate, obviously). Anthony From Jack.Jansen at cwi.nl Wed Dec 10 09:24:16 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Wed Dec 10 09:23:50 2003 Subject: [Python-Dev] release23-maint branch CLOSED for release (was Re: 2.3.3 cycle) In-Reply-To: <200312101331.hBADVo7A019516@maxim.off.ekorp.com> References: <200312101331.hBADVo7A019516@maxim.off.ekorp.com> Message-ID: <890803F8-2B1C-11D8-BC8B-0030655234CE@cwi.nl> On 10 Dec 2003, at 14:31, Anthony Baxter wrote: >>>> Jack Jansen wrote >> Anthony, >> I would like to do a 2.3.3 release for MacOS9 this time, so after >> you've finished >> with the general 2.3.3 release could you please hand over ownership >> of the branch to me, and tell the crowd to refrain from fiddling with >> the branch until I'm done? > > Not a problem. Were you going to do an OS X release as well? Yes. But that's done from the standard unix sources (maybe with one or two version numbers of installers changed, but that's all). For MacOS9 I often have to add some last second fixes. > Assuming we get the 2.3.3 release on Friday the 19th (AU time), how > long do you anticipate you'll want the branch for? (Guesstimate, > obviously). The christmas-newyear break would be a good time to do this, but I'll try to get to it earlier. -- Jack Jansen http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From guido at python.org Wed Dec 10 10:13:37 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 10 10:13:42 2003 Subject: [Python-Dev] Greetings In-Reply-To: Your message of "Wed, 10 Dec 2003 02:23:02 PST." <1071051781.14243.25.camel@leviathan.kdart.com> References: <1071051781.14243.25.camel@leviathan.kdart.com> Message-ID: <200312101513.hBAFDbi27677@c-24-5-183-134.client.comcast.net> > Hello all. I just joined this list, but I have been a full-time Python > developer for several years now.=20 > > My goal here, if possible, would be to contribute some of my code to the > the standard library. I have some more time now to clean up and > contribute some of my own library code since I am, alas, unemployed. > > I look forward to working with you all. Sure! Welcome. Sorry to hear about your job loss. Have you read the developer docs at http://www.python.org/dev/? Depending on the size of your contributions, the patch submission manager at SourceForge or a PEP describing the functionality would seem the best way to get the process started. Feel free to ask more directed questions here. --Guido van Rossum (home page: http://www.python.org/~guido/) From Scott.Daniels at Acm.Org Thu Dec 11 09:27:59 2003 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Wed Dec 10 10:26:24 2003 Subject: [Python-Dev] Re: test_unicode_file failing on Mac OS X In-Reply-To: References: <3FD71F8E.6030307@Acm.Org> Message-ID: <3FD87EEF.8060506@Acm.Org> I na?vely wrote: >Could we perhaps use a comparison that, in effect, did: > def uni_equal(first, second): > if first == second: > return True > return first.normalize() == second.normalize() >That is, take advantage of the fact that normalization is often >unnecessary for "trivial" reasons. This works, and a similar "unequal" trick may be constructible. Ordering is certainly trickier (assuring we have a total order given new equalities, so that we cannot choose a, b, and c where: a < b = c > a is True. But, Martin v. L?wis points out: > It also affects hashing, if Unicode objects are used as dictionary > keys. Objects that compare equal need to hash equal. Still not disgusting, _but_ unicode strings must hash equal to the corresponding "plain" string. I am not certain about this requirement for non-ASCII characters, but I expect we are stuck with matching hashes in the range ord(' ') through ord('~') and probably for all character values from 0 through 127. We might be able to classify UTF-16 code units into three groups: 1) matches base ASCII character 2) diacritical or combining 3) definitely distinct from any ASCII or combining form. If we map the group 1 entries to the corresponding ASCII code, skip the group 2s, and take the group 3s separately (probably remapping to another set), we might come up with a hash that used only the map results as elements contributing to the hash. Are we stuck with the current hash for unicode? If so, there is little hope. If not, this might bear further investigation. -Scott David Daniels Scott.Daniels@Acm.Org From jjl at pobox.com Wed Dec 10 12:14:39 2003 From: jjl at pobox.com (John J Lee) Date: Wed Dec 10 12:14:49 2003 Subject: [Python-Dev] Conventions for functional tests, PyUnit? In-Reply-To: <3FD63C81.20506@livinglogic.de> References: <001701c3bd29$4878f020$0705a044@oemcomputer> <3FD63C81.20506@livinglogic.de> Message-ID: On Tue, 9 Dec 2003, [ISO-8859-1] Walter D=F6rwald wrote: [...] > > One problem, though: doesn't putting functional tests in Lib/test throw > > off Walter Dorwald's unit test coverage numbers? Perhaps there should = be > > a 'functional' resource for test_support.use_resources (so the tests ca= n > > be run with -uall, -functional for coverage measurements)? > > What exactly do you mean with 'functional' tests? We certainly don't > need non-functional tests! ;) And why would additional test scripts > throw off the coverage numbers? http://c2.com/cgi/wiki?FunctionalTest I had assumed you were measuring unit test coverage, but maybe there are lots of functional tests in Lib/test too. BTW, I had a look at your coverage page, and it looked as if it would be useful and interesting, if I understood what all those little graphs &c. were. :-/ John From martin at v.loewis.de Wed Dec 10 12:16:15 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Wed Dec 10 12:16:23 2003 Subject: [Python-Dev] Re: test_unicode_file failing on Mac OS X In-Reply-To: <3FD87EEF.8060506@Acm.Org> References: <3FD71F8E.6030307@Acm.Org> <3FD87EEF.8060506@Acm.Org> Message-ID: Scott David Daniels writes: > Still not disgusting, _but_ unicode strings must hash equal to > the corresponding "plain" string. I am not certain about this > requirement for non-ASCII characters, but I expect we are stuck > with matching hashes in the range ord(' ') through ord('~') and > probably for all character values from 0 through 127. Strictly speaking, unicode and byte strings must hash equal if they convert to each other through the system encoding. As a side effect, this really means that the system encoding currently must be ASCII, as the current hash function won't otherwise hash equal (latin-1 may also be correct as the system encoding). Fortunately, plain ASCII strings are in all applicable normal forms (NFC, NFKC, NFD, NFKD). Regards, Martin From edloper at gradient.cis.upenn.edu Wed Dec 10 13:32:22 2003 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Wed Dec 10 12:31:10 2003 Subject: [Python-Dev] Normalizing unicode? (was: Re: test_unicode_file failing on Mac OS X) In-Reply-To: References: Message-ID: <3FD766B6.8000204@gradient.cis.upenn.edu> Scott David Daniels wrote: > I na?vely wrote: > >Could we perhaps use a comparison that, in effect, did: > > def uni_equal(first, second): > > if first == second: > > return True > > return first.normalize() == second.normalize() > >That is, take advantage of the fact that normalization is often > >unnecessary for "trivial" reasons. > > [...] Before we start considering how it's possible to make unicode.__equal__ act encoding-insensitively[1], I think we need to consider whether that's really the behavior we want. In some ways, this seems like case-insensitive equality to me: it's certainly a useful operation, but I don't think it should be the object's builtin notion of equality.. - I think people will be confused if s1==s2 but s1[0]!=s2[0]. - Sometimes you might *want* to distinguish different encodings of the "same" string; a "normalized" equality test makes that very difficult. And if you *do* want unicode objects to act normalized, then I think that the right way to do it is to normalize them at creation time. Then all the right hash/eq/cmp stuff just falls out. But since some people will may want to distinguish different encodings of the same string, I think that the most sensible alternative is to add a new subclass to unicode -- something like "normalized_unicode." It would normalize itself at construction time; and when combined with other unicode strings (eg by +), the result would be normalized (so unicode+normalized_unicode -> normalized_unicode). It's possible that the normalized unicode class would be more useful to people (and therefore more widely used?), but the non-normalized version would still be available for people who want it. (or we could just leave things as they are now, and force people to do any normalization themselves. :) ) -Edward [1] I don't think that "encoding" is the right technical term here, but I'm not sure what the right term is. I mean insensitive to the difference between separated diacritics & unified diacritics. From guido at python.org Wed Dec 10 12:39:16 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 10 12:39:24 2003 Subject: [Python-Dev] Normalizing unicode? (was: Re: test_unicode_file failing on Mac OS X) In-Reply-To: Your message of "Wed, 10 Dec 2003 12:32:22 CST." <3FD766B6.8000204@gradient.cis.upenn.edu> References: <3FD766B6.8000204@gradient.cis.upenn.edu> Message-ID: <200312101739.hBAHdGx27967@c-24-5-183-134.client.comcast.net> > Before we start considering how it's possible to make unicode.__equal__ > act encoding-insensitively[1], I think we need to consider whether > that's really the behavior we want. In some ways, this seems like > case-insensitive equality to me: it's certainly a useful operation, but > I don't think it should be the object's builtin notion of equality.. > - I think people will be confused if s1==s2 but s1[0]!=s2[0]. > - Sometimes you might *want* to distinguish different encodings of > the "same" string; a "normalized" equality test makes that very > difficult. Right. Couldn't have said it better myself. > And if you *do* want unicode objects to act normalized, then I think > that the right way to do it is to normalize them at creation time. Then > all the right hash/eq/cmp stuff just falls out. Exactly. > But since some people will may want to distinguish different encodings > of the same string, I think that the most sensible alternative is to add > a new subclass to unicode -- something like "normalized_unicode." It > would normalize itself at construction time; and when combined with > other unicode strings (eg by +), the result would be normalized (so > unicode+normalized_unicode -> normalized_unicode). It's possible that > the normalized unicode class would be more useful to people (and > therefore more widely used?), but the non-normalized version would still > be available for people who want it. Works for me. I recomment that someone try this approach as a user subclass first -- this should be easy enough, right? > (or we could just leave things as they are now, and force people to do > any normalization themselves. :) ) Do we even have normalization code in core Python? --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Wed Dec 10 12:43:36 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Wed Dec 10 12:44:02 2003 Subject: [Python-Dev] Normalizing unicode? (was: Re: test_unicode_file failing on Mac OS X) In-Reply-To: <200312101739.hBAHdGx27967@c-24-5-183-134.client.comcast.net> References: <3FD766B6.8000204@gradient.cis.upenn.edu> <200312101739.hBAHdGx27967@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: > > (or we could just leave things as they are now, and force people to do > > any normalization themselves. :) ) > > Do we even have normalization code in core Python? Yes, in unicodedata.normalize, since 2.3. I agree that having == operate as if strings where normalized is undesirable. Regards, Martin From walter at livinglogic.de Wed Dec 10 13:03:47 2003 From: walter at livinglogic.de (=?ISO-8859-15?Q?Walter_D=F6rwald?=) Date: Wed Dec 10 13:04:03 2003 Subject: [Python-Dev] Conventions for functional tests, PyUnit? In-Reply-To: References: <001701c3bd29$4878f020$0705a044@oemcomputer> <3FD63C81.20506@livinglogic.de> Message-ID: <3FD76003.3030809@livinglogic.de> John J Lee wrote: > On Tue, 9 Dec 2003, [ISO-8859-1] Walter D?rwald wrote: > [...] > >>>One problem, though: doesn't putting functional tests in Lib/test throw >>>off Walter Dorwald's unit test coverage numbers? Perhaps there should be >>>a 'functional' resource for test_support.use_resources (so the tests can >>>be run with -uall, -functional for coverage measurements)? >> >>What exactly do you mean with 'functional' tests? We certainly don't >>need non-functional tests! ;) And why would additional test scripts >>throw off the coverage numbers? > > > http://c2.com/cgi/wiki?FunctionalTest > > I had assumed you were measuring unit test coverage, but maybe there are > lots of functional tests in Lib/test too. What I'm doing is compiling the Python source with profiling options and running Lib/test/regrtest.py via Lib/trace.py. After that the script calls gcov for every .c file to get C coverage info and checks the coverage files generated by trace.py to get Python coverage info. > BTW, I had a look at your coverage page, and it looked as if it would be > useful and interesting, if I understood what all those little graphs &c. > were. :-/ I guess a little explanation would help. There are four types of source code lines: 1) Unknown: The status of the line can not be determined, because gcov or trace.py didn't generate a coverage file (color black) 2) Uncoverable: Can not be executed (Comment or empty line) (color grey) 3) Uncovered (but coverable): Has not been executed (but is executable) (color red) 4) Covered: Has been executed at least once during the test run. (color green) For files: 1) Unknown: A coverage file hasn't been created. 2) Uncoverable: The file consists only of uncoverable lines (e.g. a file with comments only). 3) Uncovered: The file contains coverable lines, but none of these lines have been executed. 4) Covered: The file contains at least one covered line. Note that the web application isn't exactly fast. The machine is rather old and the Postgres database seems to have a few problems with tables that contain 10 million records. :-/ Bye, Walter D?rwald From kdart at kdart.com Wed Dec 10 16:14:17 2003 From: kdart at kdart.com (Keith Dart) Date: Wed Dec 10 16:14:21 2003 Subject: [Python-Dev] Greetings In-Reply-To: <200312101513.hBAFDbi27677@c-24-5-183-134.client.comcast.net> References: <1071051781.14243.25.camel@leviathan.kdart.com> <200312101513.hBAFDbi27677@c-24-5-183-134.client.comcast.net> Message-ID: <1071090857.14245.36.camel@leviathan.kdart.com> On Wed, 2003-12-10 at 07:13, Guido van Rossum wrote: > Sure! Welcome. Sorry to hear about your job loss. Have you read the > developer docs at http://www.python.org/dev/? Depending on the size > of your contributions, the patch submission manager at SourceForge or > a PEP describing the functionality would seem the best way to get the > process started. Feel free to ask more directed questions here. Thank you. I have read the dev page. So, suppose one wanted to add a new module. Should I write PEP for it? -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031210/99fe04d5/attachment.bin From guido at python.org Wed Dec 10 16:30:52 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 10 16:30:58 2003 Subject: [Python-Dev] Greetings In-Reply-To: Your message of "Wed, 10 Dec 2003 13:14:17 PST." <1071090857.14245.36.camel@leviathan.kdart.com> References: <1071051781.14243.25.camel@leviathan.kdart.com> <200312101513.hBAFDbi27677@c-24-5-183-134.client.comcast.net> <1071090857.14245.36.camel@leviathan.kdart.com> Message-ID: <200312102130.hBALUq328562@c-24-5-183-134.client.comcast.net> > Thank you. I have read the dev page. So, suppose one wanted to add a new > module. Should I write PEP for it? That question can't be answered in general; I suggest presenting the new module you'd like to see added here (in just a few lines or paragraphs), with motivation why you think it's useful for many Python users, and see what kind of feedback you get. --Guido van Rossum (home page: http://www.python.org/~guido/) From danmil at shore.net Wed Dec 10 17:05:56 2003 From: danmil at shore.net (Dan Milstein) Date: Wed Dec 10 17:05:41 2003 Subject: [Python-Dev] Experimental Macro System Message-ID: <3FD798C4.6040709@shore.net> As a way to learn more about Python (and for fun), I'm building an experimental Dylan- and Lisp-inspired Macro system for Python. I've just put up a 0.1 release on Sourceforge at: https://sourceforge.net/projects/pymac/ ...if anyone is interested. It's pretty primitive at the moment, but it is possible to write macros and use them to expand and run actual Python code. It's currently implemented as a preprocessor (written in Python). More details in the README file in the project. -Dan Milstein From aleaxit at yahoo.com Wed Dec 10 18:35:46 2003 From: aleaxit at yahoo.com (Alex Martelli) Date: Wed Dec 10 18:35:53 2003 Subject: [Python-Dev] a serious threat to 2.3's speed? Message-ID: <200312110035.46217.aleaxit@yahoo.com> According to: http://primates.ximian.com/~miguel/archive/2003/Dec-09.html Hugunin's new Ironpython (Python on .NET) project is 70% faster than CPython-2.3 on the standard pystone benchmark. That's all I know about it, but it may need checking out (dunno if parrot can take some tricks from this Ironpython thing, but, who knows... we'd better preemptively see if there's anything we can take for CPython!). Alex From tdelaney at avaya.com Wed Dec 10 18:46:04 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Wed Dec 10 18:46:13 2003 Subject: [Python-Dev] a serious threat to 2.3's speed? Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEF60595@au3010avexu1.global.avaya.com> > From: Alex Martelli > > http://primates.ximian.com/~miguel/archive/2003/Dec-09.html > > Hugunin's new Ironpython (Python on .NET) project is 70% faster > than CPython-2.3 on the standard pystone benchmark. That's all > I know about it, but it may need checking out (dunno if parrot can > take some tricks from this Ironpython thing, but, who knows... we'd > better preemptively see if there's anything we can take for CPython!). >From the microbenchmarks, it looks like function calls are the big bottlenecks. Another option for a masters thesis Brett - speeding up function calls :) Tim Delaney From tdelaney at avaya.com Wed Dec 10 19:58:06 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Wed Dec 10 19:58:13 2003 Subject: [Python-Dev] a serious threat to 2.3's speed? Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEF605DB@au3010avexu1.global.avaya.com> > >From the microbenchmarks, it looks like function calls are > the big bottlenecks. BTW the above is my analysis - not Alex's - obviously I got the cursor on the wrong line ... my apologies. Tim Delaney From bac at OCF.Berkeley.EDU Wed Dec 10 23:06:39 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Wed Dec 10 23:06:47 2003 Subject: [Python-Dev] a serious threat to 2.3's speed? In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEF60595@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DEF60595@au3010avexu1.global.avaya.com> Message-ID: <3FD7ED4F.5060708@ocf.berkeley.edu> Delaney, Timothy C (Timothy) wrote: >>From: Alex Martelli >> >>http://primates.ximian.com/~miguel/archive/2003/Dec-09.html >> >>Hugunin's new Ironpython (Python on .NET) project is 70% faster >>than CPython-2.3 on the standard pystone benchmark. That's all >>I know about it, but it may need checking out (dunno if parrot can >>take some tricks from this Ironpython thing, but, who knows... we'd >>better preemptively see if there's anything we can take for CPython!). > > >>From the microbenchmarks, it looks like function calls are the big bottlenecks. > > Another option for a masters thesis Brett - speeding up function calls :) > =) Maybe. Just a quick update for people: I talked to my adviser and my possible topics as been thinned out considerably. I have my last final tomorrow at which point I will be free, so to speak, and thus plan on making the time to post the list again to get another round of comments on what people see as beneficial, etc. -Brett From martin at v.loewis.de Thu Dec 11 02:21:40 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 11 02:21:51 2003 Subject: [Python-Dev] Greetings In-Reply-To: <200312102130.hBALUq328562@c-24-5-183-134.client.comcast.net> References: <1071051781.14243.25.camel@leviathan.kdart.com> <200312101513.hBAFDbi27677@c-24-5-183-134.client.comcast.net> <1071090857.14245.36.camel@leviathan.kdart.com> <200312102130.hBALUq328562@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: > That question can't be answered in general; I suggest presenting the > new module you'd like to see added here (in just a few lines or > paragraphs), with motivation why you think it's useful for many Python > users, and see what kind of feedback you get. In addition, you should realize that there are many ways of publishing Python modules; including them in the Python distribution is just one way. Regards, Martin From martin at v.loewis.de Thu Dec 11 02:23:56 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 11 02:24:20 2003 Subject: [Python-Dev] a serious threat to 2.3's speed? In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEF605DB@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DEF605DB@au3010avexu1.global.avaya.com> Message-ID: "Delaney, Timothy C (Timothy)" writes: > > >From the microbenchmarks, it looks like function calls are > > the big bottlenecks. > BTW the above is my analysis - not Alex's - obviously I got the > cursor on the wrong line ... my apologies. It's not that obvious. More obviously, it's a line starting with "From", and your mailer has inserted a greater-than sign automatically. Regards, Martin From kdart at kdart.com Thu Dec 11 02:32:44 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 02:32:48 2003 Subject: [Python-Dev] distributing modules In-Reply-To: References: <1071051781.14243.25.camel@leviathan.kdart.com> <200312101513.hBAFDbi27677@c-24-5-183-134.client.comcast.net> <1071090857.14245.36.camel@leviathan.kdart.com> <200312102130.hBALUq328562@c-24-5-183-134.client.comcast.net> Message-ID: <1071127964.14245.62.camel@leviathan.kdart.com> On Wed, 2003-12-10 at 23:21, Martin v. L?wis wrote: > In addition, you should realize that there are many ways of publishing > Python modules; including them in the Python distribution is just one > way. That's true. In fact, I do distribute some from sourceforge.net (in the pyNMS project) now. But that is a large package that does contain some general and unrelated modules that others might find useful. It is also limited to a tarball download or CVS access. That limits it's user base a bit. Since we currently don't have a CPAN for Python, I thought I might try this route. Also, I know it is often said that Python has "batteries included". However, in my experience the "batteries" are often inadequate for the job. Many modules are incomplete, or otherwise like some key functionality. I have had to extend and/or rewrite many of the "stock" libraries over the past few years. I would like to fold my additions back into the core, if possible. -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031210/74167065/attachment.bin From karthik19_20 at yahoo.com Thu Dec 11 03:31:58 2003 From: karthik19_20 at yahoo.com (karthik karthik) Date: Thu Dec 11 03:32:02 2003 Subject: [Python-Dev] prolem in running AAIE demo in pyAA.py Message-ID: <20031211083158.63563.qmail@web11901.mail.yahoo.com> hi!! when i'm trying to run the AAIE demo in pyAA.py i got this error message: from wxPython.wx import * ImportError: No module named wxPython.wx can anyone provide me the link to download wxPython.wx? regards, Sha __________________________________ Do you Yahoo!? New Yahoo! Photos - easier uploading and sharing. http://photos.yahoo.com/ From fredrik at pythonware.com Thu Dec 11 05:16:58 2003 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu Dec 11 05:17:12 2003 Subject: [Python-Dev] Re: a serious threat to 2.3's speed? References: <338366A6D2E2CA4C9DAEAE652E12A1DEF60595@au3010avexu1.global.avaya.com> Message-ID: Tim Delaney wrote: > From the microbenchmarks, it looks like function calls are the big > bottlenecks. in my pytte1 experiment from last year, I was able to run pystone about 25 times faster than a stock 2.1 interpreter. (that would be, say, about 20 times faster than 2.3) key features contributing to this is true gc, call-site caching, and "traditional" (C-style) argument passing for common cases. not sure if any of that can be applied any of that to today's C python, though (but that doesn't mean it cannot be done ;-). > Another option for a masters thesis Brett - speeding up function calls :) that would be most excellent. From mwh at python.net Thu Dec 11 05:32:17 2003 From: mwh at python.net (Michael Hudson) Date: Thu Dec 11 05:32:20 2003 Subject: [Python-Dev] distributing modules In-Reply-To: <1071127964.14245.62.camel@leviathan.kdart.com> (Keith Dart's message of "Wed, 10 Dec 2003 23:32:44 -0800") References: <1071051781.14243.25.camel@leviathan.kdart.com> <200312101513.hBAFDbi27677@c-24-5-183-134.client.comcast.net> <1071090857.14245.36.camel@leviathan.kdart.com> <200312102130.hBALUq328562@c-24-5-183-134.client.comcast.net> <1071127964.14245.62.camel@leviathan.kdart.com> Message-ID: <2mad5zodam.fsf@starship.python.net> Keith Dart writes: > Also, I know it is often said that Python has "batteries included". > However, in my experience the "batteries" are often inadequate for the > job. Many modules are incomplete, or otherwise like some key > functionality. I have had to extend and/or rewrite many of the "stock" > libraries over the past few years. I would like to fold my additions > back into the core, if possible. I think improving existing library modules is a much less contentious issue than adding new ones... Cheers, mwh -- Darned confusing, unless you have that magic ingredient coffee, of which I can pay you Tuesday for a couple pounds of extra-special grind today. -- John Mitchell, 11 Jan 1999 From mwh at python.net Thu Dec 11 05:34:37 2003 From: mwh at python.net (Michael Hudson) Date: Thu Dec 11 05:34:40 2003 Subject: [Python-Dev] Normalizing unicode? In-Reply-To: <3FD766B6.8000204@gradient.cis.upenn.edu> (Edward Loper's message of "Wed, 10 Dec 2003 12:32:22 -0600") References: <3FD766B6.8000204@gradient.cis.upenn.edu> Message-ID: <2m65gnod6q.fsf@starship.python.net> Edward Loper writes: > Scott David Daniels wrote: >> I na?vely wrote: >> >Could we perhaps use a comparison that, in effect, did: >> > def uni_equal(first, second): >> > if first == second: >> > return True >> > return first.normalize() == second.normalize() >> >That is, take advantage of the fact that normalization is often >> >unnecessary for "trivial" reasons. >> [...] > > Before we start considering how it's possible to make > unicode.__equal__ act encoding-insensitively[1], I think we need to > consider whether that's really the behavior we want. In some ways, > this seems like case-insensitive equality to me: it's certainly a > useful operation, but I don't think it should be the object's builtin > notion of equality.. > - I think people will be confused if s1==s2 but s1[0]!=s2[0]. > - Sometimes you might *want* to distinguish different encodings of > the "same" string; a "normalized" equality test makes that very > difficult. In general it seems to me that == should, given a choice, err on the side of being an overly tight equivalence relation -- i.e. return True less often. Cheers, mwh -- 81. In computing, turning the obvious into the useful is a living definition of the word "frustration". -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html From martin at v.loewis.de Thu Dec 11 06:34:14 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 11 06:36:13 2003 Subject: [Python-Dev] Re: distributing modules In-Reply-To: <1071127964.14245.62.camel@leviathan.kdart.com> References: <1071051781.14243.25.camel@leviathan.kdart.com> <200312101513.hBAFDbi27677@c-24-5-183-134.client.comcast.net> <1071090857.14245.36.camel@leviathan.kdart.com> <200312102130.hBALUq328562@c-24-5-183-134.client.comcast.net> <1071127964.14245.62.camel@leviathan.kdart.com> Message-ID: Keith Dart writes: > That's true. In fact, I do distribute some from sourceforge.net (in the > pyNMS project) now. But that is a large package that does contain some > general and unrelated modules that others might find useful. As the first step, you should make use of distutils, then generate binary packages for the platforms you care about (e.g. bdist_wininst, bdist_rpm). As the second step, you should register your packages with PyPI: http://www.python.org/pypi > Also, I know it is often said that Python has "batteries included". > However, in my experience the "batteries" are often inadequate for the > job. Many modules are incomplete, or otherwise like some key > functionality. I have had to extend and/or rewrite many of the "stock" > libraries over the past few years. As Michael explains, improvements on existing libraries are very much appreciated. You should upload each improvement individually to SF, indicating what bug it fixes or what feature it adds. I would be concerned about rewrites, though, because they might break backwards compatibility. Regards, Martin From jeremy at alum.mit.edu Thu Dec 11 10:04:30 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Thu Dec 11 10:07:49 2003 Subject: [Python-Dev] Re: a serious threat to 2.3's speed? In-Reply-To: References: <338366A6D2E2CA4C9DAEAE652E12A1DEF60595@au3010avexu1.global.avaya.com> Message-ID: <1071155070.15985.583.camel@localhost.localdomain> On Thu, 2003-12-11 at 05:16, Fredrik Lundh wrote: > Tim Delaney wrote: > > > From the microbenchmarks, it looks like function calls are the big > > bottlenecks. > > in my pytte1 experiment from last year, I was able to run pystone about 25 > times faster than a stock 2.1 interpreter. what is pyttel? I don't think I've heard of it. Jeremy From skip at pobox.com Thu Dec 11 10:09:03 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 11 10:09:21 2003 Subject: [Python-Dev] Re: a serious threat to 2.3's speed? In-Reply-To: References: <338366A6D2E2CA4C9DAEAE652E12A1DEF60595@au3010avexu1.global.avaya.com> Message-ID: <16344.34959.580835.583077@montanaro.dyndns.org> >> Another option for a masters thesis Brett - speeding up function >> calls :) Fredrik> that would be most excellent. Indeed. We should make sure we're comparing apples to apples when looking at Jim's benchmark figures though. It's not clear to me that his current system supports all the different types of callables which CPython supports. That alone might streamline his function call code a lot. Skip From fredrik at pythonware.com Thu Dec 11 10:17:35 2003 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu Dec 11 10:17:43 2003 Subject: [Python-Dev] Re: Re: a serious threat to 2.3's speed? References: <338366A6D2E2CA4C9DAEAE652E12A1DEF60595@au3010avexu1.global.avaya.com> <1071155070.15985.583.camel@localhost.localdomain> Message-ID: Jeremy Hylton wrote: > > > From the microbenchmarks, it looks like function calls are the big > > > bottlenecks. > > > > in my pytte1 experiment from last year, I was able to run pystone about 25 > > times faster than a stock 2.1 interpreter. > > what is pyttel? I don't think I've heard of it. google is your friend (at least if you're using the right font ;-) http://effbot.org/zone/pytte.htm From aleax at aleax.it Thu Dec 11 10:40:18 2003 From: aleax at aleax.it (Alex Martelli) Date: Thu Dec 11 10:40:25 2003 Subject: [Python-Dev] Re: a serious threat to 2.3's speed? In-Reply-To: <16344.34959.580835.583077@montanaro.dyndns.org> References: <338366A6D2E2CA4C9DAEAE652E12A1DEF60595@au3010avexu1.global.avaya.com> <16344.34959.580835.583077@montanaro.dyndns.org> Message-ID: <200312111640.18085.aleax@aleax.it> On Thursday 11 December 2003 04:09 pm, Skip Montanaro wrote: > >> Another option for a masters thesis Brett - speeding up function > >> calls :) > > Fredrik> that would be most excellent. > > Indeed. We should make sure we're comparing apples to apples when looking > at Jim's benchmark figures though. It's not clear to me that his current > system supports all the different types of callables which CPython > supports. That alone might streamline his function call code a lot. How so? Isn't one "indirecting" through the tp_call slot anyway? Supporting two or two thousand types of callables should make function calls just about the same cost -- or am I missing something...? Alex From skip at pobox.com Thu Dec 11 11:06:03 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 11 11:06:12 2003 Subject: [Python-Dev] Re: a serious threat to 2.3's speed? In-Reply-To: <200312111640.18085.aleax@aleax.it> References: <338366A6D2E2CA4C9DAEAE652E12A1DEF60595@au3010avexu1.global.avaya.com> <16344.34959.580835.583077@montanaro.dyndns.org> <200312111640.18085.aleax@aleax.it> Message-ID: <16344.38379.89674.176847@montanaro.dyndns.org> >> It's not clear to me that his current system supports all the >> different types of callables which CPython supports. That alone might >> streamline his function call code a lot. Alex> How so? Isn't one "indirecting" through the tp_call slot anyway? Alex> Supporting two or two thousand types of callables should make Alex> function calls just about the same cost -- or am I missing Alex> something...? Python's function call machinery is complex, mostly in the argument marshalling area I think. Take a look in Python/ceval.c at call_function fast_function do_call ext_do_call update_keyword_args update_star_args If Jim's compiler doesn't yet support any of varargs, keyword args, calling C functions or calling bound method objects, I suspect his code is more streamlined. Skip From duncan at rcp.co.uk Thu Dec 11 11:35:57 2003 From: duncan at rcp.co.uk (Duncan Booth) Date: Thu Dec 11 11:36:08 2003 Subject: [Python-Dev] Re: a serious threat to 2.3's speed? References: <338366A6D2E2CA4C9DAEAE652E12A1DEF60595@au3010avexu1.global.avaya.com> <16344.34959.580835.583077@montanaro.dyndns.org> <200312111640.18085.aleax@aleax.it> <16344.38379.89674.176847@montanaro.dyndns.org> Message-ID: Skip Montanaro wrote in news:16344.38379.89674.176847@montanaro.dyndns.org: > Python's function call machinery is complex, mostly in the argument > marshalling area I think. Take a look in Python/ceval.c at > > call_function > fast_function > do_call > ext_do_call > update_keyword_args > update_star_args > > If Jim's compiler doesn't yet support any of varargs, keyword args, > calling C functions or calling bound method objects, I suspect his > code is more streamlined. I've been working on my own implementation of Python for .Net, although I've now emailed Jim to ask if I can help him with his since he seems to be further on that I am with my version. The way I was planning to do function calls (and had partially implemented it), was to have an IPyCallable interface, and a set of classes that implement that interface. The interface includes methods: PyObject __call__(); PyObject __call__(PyObject arg1); PyObject __call__(PyObject arg1, PyObject arg2); PyObject __call__(PyObject arg1, PyObject arg2, PyObject arg3); PyObject __call__(PyObject[] args); PyObject __call__(PyObject[] args, PyDict keywordArgs); Concrete classes exist for functions and for bound methods taking 0, 1, 2, 3 or more arguments. The constructor for a function object takes a delegate, a list of argument names and a list of defaults. The idea was that, say you had a function taking 3 arguments and you call it with 3 arguments, the call can be passed straight through without doing any checking at all for default args or keyword args, so one virtual call then a delegate call. If you call it with fewer arguments the earlier overloads simply add in the missing defaults. This should allow pretty well all calls to be streamlined so you only pay for the overhead of variable number of arguments or keywords if you actually use them in the call, and even then you don't have a string to interpret at runtime. I was planning to create function wrappers at runtime, so that a method that took say a string and an integer would be wrapped by a static function taking three PyObject arguments and casting them appropriately before forwarding to the wrapped method. So a Python call to a method would get directed through the function object which calls the delegate to the wrapper which does any necessary casts and calls the underlying method. In practice some of these nested calls should be optimisable by the JIT. Anyway, that was my scheme. I wonder what Jim's is. -- Duncan Booth duncan@rcp.co.uk int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3" "\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure? From pje at telecommunity.com Thu Dec 11 12:06:17 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Dec 11 12:06:35 2003 Subject: [Python-Dev] Re: a serious threat to 2.3's speed? In-Reply-To: References: <338366A6D2E2CA4C9DAEAE652E12A1DEF60595@au3010avexu1.global.avaya.com> <16344.34959.580835.583077@montanaro.dyndns.org> <200312111640.18085.aleax@aleax.it> <16344.38379.89674.176847@montanaro.dyndns.org> Message-ID: <5.1.1.6.0.20031211115334.02259ec0@telecommunity.com> At 04:35 PM 12/11/03 +0000, Duncan Booth wrote: >The way I was planning to do function calls (and had partially implemented >it), was to have an IPyCallable interface, and a set of classes that >implement that interface. > >The interface includes methods: > > PyObject __call__(); > PyObject __call__(PyObject arg1); > PyObject __call__(PyObject arg1, PyObject arg2); > PyObject __call__(PyObject arg1, PyObject arg2, PyObject arg3); > PyObject __call__(PyObject[] args); > PyObject __call__(PyObject[] args, PyDict keywordArgs); > >Concrete classes exist for functions and for bound methods taking 0, 1, 2, >3 or more arguments. The constructor for a function object takes a >delegate, a list of argument names and a list of defaults. Hm. We could steal this exact same idea for use in CPython, although it'd be tedious to implement. The current tp_call would cover the last two cases shown, but adding a tp_call0 through tp_call3 or so might speed up a *lot* of function calls. Specialized bytecodes for these calls would have very short implementations. The tp_call_N slots could have default implementations in 'object' that delegate back to the regular tp_call, so that end-user types wouldn't need to reimplement them. Hm. Actually, this'd be *really* tedious to implement right, because of all the custom bound method types needed, and the C multiple wrappers for __call__ methods written in Python. OTOH, this might be an "opportunity" to write generic leftcurry and rightcurry functions, and use rightcurry for default arguments... Bleah. What this *really* seems to want is to generate a new type on the fly, that is precisely coded to do the necessary left and right currying as part of C-level code invocation. A JIT, in other words. Ah well, it sounded cool at first. :) From tjreedy at udel.edu Thu Dec 11 14:13:45 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Thu Dec 11 14:13:49 2003 Subject: [Python-Dev] Re: prolem in running AAIE demo in pyAA.py References: <20031211083158.63563.qmail@web11901.mail.yahoo.com> Message-ID: "karthik karthik" wrote in message news:20031211083158.63563.qmail@web11901.mail.yahoo.com... > when i'm trying to run the AAIE demo in pyAA.py i got > this error message: > > > from wxPython.wx import * > ImportError: No module named wxPython.wx > > > can anyone provide me the link to download > wxPython.wx? PyDev is for developing future Python versions. Please direct usage questions like this to the general Python mailing list (or comp.lang.python) or to application specific lists (there is one for wxPython, I believe). TJR From tdelaney at avaya.com Thu Dec 11 14:50:40 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Thu Dec 11 14:50:47 2003 Subject: [Python-Dev] a serious threat to 2.3's speed? Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0456@au3010avexu1.global.avaya.com> > From: martin@v.loewis.de [mailto:martin@v.loewis.de] > > > > >From the microbenchmarks, it looks like function calls are > > > the big bottlenecks. > > > BTW the above is my analysis - not Alex's - obviously I got the > > cursor on the wrong line ... my apologies. > > It's not that obvious. More obviously, it's a line starting with > "From", and your mailer has inserted a greater-than sign > automatically. What I meant was that the line has an extra indent, and the cause is most likely that when I trimmed the original message I accidentally left the cursor on the wrong line when I started typing and didn't notice it until after the message was sent. Hence the use of "obviously" - as in, that was obviously the cause of the error - not that it's obvious that it *wasn't* Alex's comment. Tim Delaney From tdelaney at avaya.com Thu Dec 11 14:55:09 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Thu Dec 11 14:55:17 2003 Subject: [Python-Dev] Re: a serious threat to 2.3's speed? Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0458@au3010avexu1.global.avaya.com> > From: Fredrik Lundh > > key features contributing to this is true gc, call-site > caching, and "traditional" > (C-style) argument passing for common cases. Did you collect stats on how much garbage collection actually occurred during the tests? How did it compare to stock CPython? One of the big flaws in benchmarking early versions of Java of course was that most tests didn't end up invoking the garbage collector - hence showing an artificially good result that could not be observed in real-world usage. Tim Delaney From tdelaney at avaya.com Thu Dec 11 14:57:12 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Thu Dec 11 14:57:19 2003 Subject: [Python-Dev] Re: a serious threat to 2.3's speed? Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0459@au3010avexu1.global.avaya.com> > From: Duncan Booth > > The interface includes methods: > > PyObject __call__(); > PyObject __call__(PyObject arg1); > PyObject __call__(PyObject arg1, PyObject arg2); > PyObject __call__(PyObject arg1, PyObject arg2, PyObject arg3); > PyObject __call__(PyObject[] args); > PyObject __call__(PyObject[] args, PyDict keywordArgs); Hmm - IIRC that's how Jython does it. Tim Delaney From pedronis at bluewin.ch Thu Dec 11 15:52:54 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Thu Dec 11 15:49:45 2003 Subject: [Python-Dev] Re: a serious threat to 2.3's speed? In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0459@au3010avexu1.global .avaya.com> Message-ID: <5.2.1.1.0.20031211211518.0294e900@pop.bluewin.ch> At 06:57 12.12.2003 +1100, Delaney, Timothy C (Timothy) wrote: > > From: Duncan Booth > > > > The interface includes methods: > > > > PyObject __call__(); > > PyObject __call__(PyObject arg1); > > PyObject __call__(PyObject arg1, PyObject arg2); > > PyObject __call__(PyObject arg1, PyObject arg2, PyObject arg3); > > PyObject __call__(PyObject[] args); > > PyObject __call__(PyObject[] args, PyDict keywordArgs); > >Hmm - IIRC that's how Jython does it. yup, but it doesn't try to profit from this organization as much as it could, and it queries some ad-hoc per-thread storage at each function call, instead of passing the data along from caller to callee, which would be possibly speedier. So there are probably margins for speeding up also Jython. OTOH I expect .NET to allow a larger range of tricks. My pet peeves with .NET are that it does not garbage collect dynamically loaded code, but maybe their are finally planning to support that, or will at some point. And that there is a strong pressure (market numbers etc ...) to favor platform integration versus keeping the ported languages true to their original semantics. (rather purist peeves). But considering things like Groovy on Java side maybe it is what people want. [ I think there is a tension between scripting the platform and having languages that can survive beyond the platform. OTOH given that covering with a portability layer the huge class libraries that come with .NET or Java is probably something people don't have time to, the point is rather moot and only of theoretical interest, OTOH I know of some few CPython/Jython projects using one or the other on different tier and reusing domain logic ] .NET, Mono, Parrot, PyPy... funny times ahead. From kdart at kdart.com Thu Dec 11 17:17:22 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 17:17:29 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle Message-ID: <1071181041.14243.76.camel@leviathan.kdart.com> Skipped content of type multipart/mixed-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/2f0728a3/attachment-0001.bin From skip at pobox.com Thu Dec 11 17:24:34 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 11 17:24:37 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <1071181041.14243.76.camel@leviathan.kdart.com> References: <1071181041.14243.76.camel@leviathan.kdart.com> Message-ID: <16344.61090.472327.729253@montanaro.dyndns.org> Keith> class TextFile(UserFile): Keith> """TextFile(self, name, mode="r", bufsize=-1, linesep=None) Keith> A file object that handles different line separator Keith> conventions. This file object's readline() method returns the Keith> line with trailing line separator stripped, and raises EOFError Keith> on end of file. The writelines() method will append an Keith> appropriate line separator when writing. Thus, this file object Keith> allows reading and writeing non-native text files. Keith> """ Is there a reason the universal newline mode doesn't cover this case already? Keith> """MergedIO(outfile, infile) Keith> Combines a write stream and a read stream into one read/write Keith> object.""" And isn't this subsumed by mode='r+'? Skip From kdart at kdart.com Thu Dec 11 17:29:34 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 17:29:39 2003 Subject: [Python-Dev] New module proposal: timelib Message-ID: <1071181774.14243.81.camel@leviathan.kdart.com> Skipped content of type multipart/mixed-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/9b87a23f/attachment.bin From kdart at kdart.com Thu Dec 11 17:37:04 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 17:37:09 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <16344.61090.472327.729253@montanaro.dyndns.org> References: <1071181041.14243.76.camel@leviathan.kdart.com> <16344.61090.472327.729253@montanaro.dyndns.org> Message-ID: <1071182224.14245.89.camel@leviathan.kdart.com> On Thu, 2003-12-11 at 14:24, Skip Montanaro wrote: > Keith> class TextFile(UserFile): > Keith> """TextFile(self, name, mode="r", bufsize=-1, linesep=None) > Keith> A file object that handles different line separator > Keith> conventions. This file object's readline() method returns the > Keith> line with trailing line separator stripped, and raises EOFError > Keith> on end of file. The writelines() method will append an > Keith> appropriate line separator when writing. Thus, this file object > Keith> allows reading and writeing non-native text files. > Keith> """ > > Is there a reason the universal newline mode doesn't cover this case > already? This module lets you explicitly create a text data file for any platform. The universal newline feature of python is for Python modules only, and lets you transparently import and exec Python source files with the different line ending formats. That is a different thing. > Keith> """MergedIO(outfile, infile) > Keith> Combines a write stream and a read stream into one read/write > Keith> object.""" > > And isn't this subsumed by mode='r+'? No, mode r+ lets you create a _single_ file descriptor that is in read-write mode. The MergedIO lets you take who file objects (with different file descriptors), one readable the other writable, and combine them at the method-call level into one read-write object. This is useful for pipes, or when you get file objects from libraries. -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/92d5c829/attachment.bin From bac at OCF.Berkeley.EDU Thu Dec 11 17:48:39 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Thu Dec 11 17:48:47 2003 Subject: [Python-Dev] New module proposal: timelib In-Reply-To: <1071181774.14243.81.camel@leviathan.kdart.com> References: <1071181774.14243.81.camel@leviathan.kdart.com> Message-ID: <3FD8F447.7080300@ocf.berkeley.edu> Keith Dart wrote: > Attached is a module that I call timelib that extends the standard time > module. This module allows easier time manipulation with a mutable time > object, and provides some extra utility functions. > Browsing the code I didn't see anything that was that much better than what datetime offered. The mutability is the only big difference but I don't see that as a critical thing in this case since they are relatively cheap objects to create. And speaking of datetime, that module pretty much supercedes time. The only thing in time that datetime lacks is basically strptime from my experience. Personally, I would rather see any nice, new functionality be added to datetime so that time itself can possibly just go away. datetime is just so much nicer to use in my opinion. -Brett From kdart at kdart.com Thu Dec 11 17:48:59 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 17:49:05 2003 Subject: [Python-Dev] new module proposal: daemonize Message-ID: <1071182938.14238.95.camel@leviathan.kdart.com> Skipped content of type multipart/mixed-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/9b97aa8e/attachment.bin From bac at OCF.Berkeley.EDU Thu Dec 11 17:51:32 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Thu Dec 11 17:51:39 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <1071182224.14245.89.camel@leviathan.kdart.com> References: <1071181041.14243.76.camel@leviathan.kdart.com> <16344.61090.472327.729253@montanaro.dyndns.org> <1071182224.14245.89.camel@leviathan.kdart.com> Message-ID: <3FD8F4F4.3060904@ocf.berkeley.edu> Keith Dart wrote: > On Thu, 2003-12-11 at 14:24, Skip Montanaro wrote: > >> Keith> class TextFile(UserFile): >> Keith> """TextFile(self, name, mode="r", bufsize=-1, linesep=None) >> Keith> A file object that handles different line separator >> Keith> conventions. This file object's readline() method returns the >> Keith> line with trailing line separator stripped, and raises EOFError >> Keith> on end of file. The writelines() method will append an >> Keith> appropriate line separator when writing. Thus, this file object >> Keith> allows reading and writeing non-native text files. >> Keith> """ >> >>Is there a reason the universal newline mode doesn't cover this case >>already? > > > This module lets you explicitly create a text data file for any > platform. The universal newline feature of python is for Python modules > only, and lets you transparently import and exec Python source files > with the different line ending formats. That is a different thing. > Not true. Python 2.3 added the "U" option for file opening to use universal newline support for any file you open. See http://www.python.org/doc/2.3/whatsnew/node7.html . -Brett From pje at telecommunity.com Thu Dec 11 17:51:39 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Dec 11 17:51:54 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <1071182224.14245.89.camel@leviathan.kdart.com> References: <16344.61090.472327.729253@montanaro.dyndns.org> <1071181041.14243.76.camel@leviathan.kdart.com> <16344.61090.472327.729253@montanaro.dyndns.org> Message-ID: <5.1.1.6.0.20031211175002.028d0470@telecommunity.com> At 02:37 PM 12/11/03 -0800, Keith Dart wrote: >This module lets you explicitly create a text data file for any >platform. The universal newline feature of python is for Python modules >only, and lets you transparently import and exec Python source files >with the different line ending formats. That is a different thing. Not so. See: http://www.python.org/doc/current/lib/built-in-funcs.html under 'file()', which notes that 'U' or 'rU' enable universal newline mode for any input file. From kdart at kdart.com Thu Dec 11 17:52:59 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 17:53:03 2003 Subject: [Python-Dev] new module proposal: logfile Message-ID: <1071183178.14244.99.camel@leviathan.kdart.com> Attached is a module that implements a self-rotating log file object. You simply write to it perpetually, and if it grows over a specified size it is automatically truncated, and a certain number of archived copies are kept as well. It also has an object that allows redirecting your stdio to a log file. Useful for daemon processes. -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/ac8c37c9/attachment.bin From kdart at kdart.com Thu Dec 11 17:56:50 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 17:56:56 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <3FD8F4F4.3060904@ocf.berkeley.edu> References: <1071181041.14243.76.camel@leviathan.kdart.com> <16344.61090.472327.729253@montanaro.dyndns.org> <1071182224.14245.89.camel@leviathan.kdart.com> <3FD8F4F4.3060904@ocf.berkeley.edu> Message-ID: <1071183410.14243.103.camel@leviathan.kdart.com> On Thu, 2003-12-11 at 14:51, Brett C. wrote: > Keith Dart wrote: > > This module lets you explicitly create a text data file for any > > platform. The universal newline feature of python is for Python modules > > only, and lets you transparently import and exec Python source files > > with the different line ending formats. That is a different thing. > > > > Not true. Python 2.3 added the "U" option for file opening to use > universal newline support for any file you open. See > http://www.python.org/doc/2.3/whatsnew/node7.html . Whups, something new! Anyway, that link doesn't say anything about writing, only reading. It does not appear that you can specify what platform to write to with U mode. -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/a6a5c62f/attachment.bin From guido at python.org Thu Dec 11 18:00:27 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 11 17:58:25 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: Your message of "Thu, 11 Dec 2003 14:37:04 PST." <1071182224.14245.89.camel@leviathan.kdart.com> References: <1071181041.14243.76.camel@leviathan.kdart.com> <16344.61090.472327.729253@montanaro.dyndns.org> <1071182224.14245.89.camel@leviathan.kdart.com> Message-ID: <200312112300.hBBN0RU02156@c-24-5-183-134.client.comcast.net> > > Is there a reason the universal newline mode doesn't cover this case > > already? > > This module lets you explicitly create a text data file for any > platform. The universal newline feature of python is for Python modules > only, and lets you transparently import and exec Python source files > with the different line ending formats. That is a different thing. I suggest you go back and read the docs for open() in Python 2.3 again, you are confused with a different feature. I also note that you change the meaning of readline() to return the line with the trailing separator stripped, and raise EOFError when you have reached the end. That's a big deviation from the standard file semantics, and shouldn't be done lightly in a subclass (it means that a TextFile instance is not substitutable for a regular file instance in most cases). Reviewing the rest of UserFile, I see several possibly useful additions (locking, fstat() etc.) and one semantic change that deserves to be a separate subclass or mix-in (the EINTR behavior). And nothing that warrants the name "UserFile" (in fact, all the other UserXXX classes are specifically *not* subclasses of the native XXX class). > No, mode r+ lets you create a _single_ file descriptor that is in > read-write mode. The MergedIO lets you take who file objects (with > different file descriptors), one readable the other writable, and > combine them at the method-call level into one read-write object. This > is useful for pipes, or when you get file objects from libraries. It looks like you are simply making all write operations go to the writable object and all the read operations go to the readable object. That's not a very sophisticated notion of "merging", and I'm not sure what the point is. Perhaps some examples of how you use this would help. I note that in general your proposals are low on motivation, which is going to do a lot of damage to your case if you don't fix this soon. You may be underestimating the level of motivation needed to get features added to the stdlib -- it's "batteries included" but also (to some extent) "TOOWTDI". --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Thu Dec 11 17:59:54 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Dec 11 18:00:00 2003 Subject: [Python-Dev] new module proposal: logfile In-Reply-To: <1071183178.14244.99.camel@leviathan.kdart.com> Message-ID: <5.1.1.6.0.20031211175420.028d8ec0@telecommunity.com> At 02:52 PM 12/11/03 -0800, Keith Dart wrote: >Attached is a module that implements a self-rotating log file object. This is already available in Python 2.3; please see http://www.python.org/doc/current/lib/node285.html Perhaps it would be better for you to publish these modules on the web someplace, and then post the URL to python-announce? That way, interested parties could look them over and try them out more easily. You would also be able to get feedback from a wider audience. From guido at python.org Thu Dec 11 18:07:14 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 11 18:05:19 2003 Subject: [Python-Dev] new module proposal: daemonize In-Reply-To: Your message of "Thu, 11 Dec 2003 14:48:59 PST." <1071182938.14238.95.camel@leviathan.kdart.com> References: <1071182938.14238.95.camel@leviathan.kdart.com> Message-ID: <200312112307.hBBN7EW02209@c-24-5-183-134.client.comcast.net> > The attached module encapsulates all that is necessary to make your > python script into a Unix daemon. It also provides a way to redirect > your process's stdout to a log file. You call daemonize.daemonize() and > when it returns your code is now a daemon (disconnected from a terminal > and stdio). This (or something like it) looks like a useful addition. I remember writing a much more elaborate version for Zope: http://cvs.zope.org/Zope/lib/python/zdaemon/ (Ignore Daemon.py, which is the old Zope API for the same thing.) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Dec 11 18:09:23 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 11 18:07:20 2003 Subject: [Python-Dev] new module proposal: logfile In-Reply-To: Your message of "Thu, 11 Dec 2003 14:52:59 PST." <1071183178.14244.99.camel@leviathan.kdart.com> References: <1071183178.14244.99.camel@leviathan.kdart.com> Message-ID: <200312112309.hBBN9Nd02235@c-24-5-183-134.client.comcast.net> > Attached is a module that implements a self-rotating log file object. > You simply write to it perpetually, and if it grows over a specified > size it is automatically truncated, and a certain number of archived > copies are kept as well. The logging module also supports this. > It also has an object that allows redirecting your stdio to a log file. > Useful for daemon processes. I'd be more impressed if it was integrated with the loggingmodule. --Guido van Rossum (home page: http://www.python.org/~guido/) From kdart at kdart.com Thu Dec 11 18:10:12 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 18:10:21 2003 Subject: [Python-Dev] new module proposal: logfile In-Reply-To: <5.1.1.6.0.20031211175420.028d8ec0@telecommunity.com> References: <5.1.1.6.0.20031211175420.028d8ec0@telecommunity.com> Message-ID: <1071184212.14243.112.camel@leviathan.kdart.com> On Thu, 2003-12-11 at 14:59, Phillip J. Eby wrote: > At 02:52 PM 12/11/03 -0800, Keith Dart wrote: > >Attached is a module that implements a self-rotating log file object. > > This is already available in Python 2.3; please see > http://www.python.org/doc/current/lib/node285.html > > Perhaps it would be better for you to publish these modules on the web > someplace, and then post the URL to python-announce? That way, interested > parties could look them over and try them out more easily. You would also > be able to get feedback from a wider audience. They already are. These are part of pyNMS: http://sourceforge.net/projects/pynms But I thought they might be general enough for common usage. But I think the lesson I get from this is that I should study the Python 2.3 docs. I have not used Python 2.3 much yet. All of the stuff I have been submitted is old, pre-2.3, and I was not aware that some of it has been recently added. Sorry. 8-< -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/99b93b72/attachment.bin From tim.one at comcast.net Thu Dec 11 18:10:34 2003 From: tim.one at comcast.net (Tim Peters) Date: Thu Dec 11 18:10:33 2003 Subject: [Python-Dev] New module proposal: timelib In-Reply-To: <3FD8F447.7080300@ocf.berkeley.edu> Message-ID: [Keith Dart] >> Attached is a module that I call timelib that extends the standard >> time module. This module allows easier time manipulation with a >> mutable time object, and provides some extra utility functions. [Brett Cannon] > Browsing the code I didn't see anything that was that much better than > what datetime offered. The mutability is the only big difference but > I don't see that as a critical thing in this case since they are > relatively cheap objects to create. Just noting that it was considered critical for time and date objects *not* to be mutable, else they couldn't be used as dict keys (an important use case). From kdart at kdart.com Thu Dec 11 18:12:09 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 18:12:14 2003 Subject: [Python-Dev] new module proposal: logfile In-Reply-To: <200312112309.hBBN9Nd02235@c-24-5-183-134.client.comcast.net> References: <1071183178.14244.99.camel@leviathan.kdart.com> <200312112309.hBBN9Nd02235@c-24-5-183-134.client.comcast.net> Message-ID: <1071184329.14244.115.camel@leviathan.kdart.com> On Thu, 2003-12-11 at 15:09, Guido van Rossum wrote: > > Attached is a module that implements a self-rotating log file object. > > You simply write to it perpetually, and if it grows over a specified > > size it is automatically truncated, and a certain number of archived > > copies are kept as well. > > The logging module also supports this. > > > It also has an object that allows redirecting your stdio to a log file. > > Useful for daemon processes. > > I'd be more impressed if it was integrated with the loggingmodule. Yes, ok. My modules predate the logging module (and 2.3 in general). I should get more familiar with 2.3 first. Sorry about that. -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/2e574e4b/attachment.bin From tdelaney at avaya.com Thu Dec 11 18:13:53 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Thu Dec 11 18:14:01 2003 Subject: [Python-Dev] RE: Python and Temple of Elemental Evil Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE04B1@au3010avexu1.global.avaya.com> > From: Steve Moret [mailto:smoret@troikagames.com] > > Hey Tim, Hi Steve. I held off pestering you since you said you were going to do further work on ToEE over the Christmas break ;) But since you've obviously got so much time, there's that slight matter of an article about how Python was used in ToEE ... :) I'm forwarding this to python-dev as it deals with potential modifications to the python code base (including offer of a patch :) Guido has the final say anyway ... I'll also put some comments inline - it could be that the documentation needs to be improved rather than any code changes. In particular, I think the first two cases are simply resolved, but possibly not obvious. BTW, I've really enjoyed ToEE (I'm currently running a 3-character party ;) but I do agree that as a game it suffered from being too close to the module (something Huy - I think - said in the wrap-up on IGN). It didn't have the Troika goodness from Fallout or Arcanum ... It's a pity that game publishers don't seem to realise that the majority of people who buy games are adults ... > Steve Moret here, from Troika Games, now that I've had a few > minutes to step > away from work (after Temple of Elemental Evil shipped) I've > been itching to > give back some of the work we did on Python to the community. > > One major thing (that us game developers use a lot) that the Python/C > interface is missing (or that I just couldn't find) is a way > of replacing > Python's IO with a set that will work for the game (or other custom > project). > > Two things that I've now done twice to python (and need to do > a third time > in a more generic sense so I can release it to the community, > unless its > already in there and I just don't know how to do it) is 1) replacing > Python's IO (specifically stdout/stderr and stdin) so that it > can read/write > to the game's libraries (traditionally this goes to a console or debug > file), Probably the easiest way to do this is to simply replace sys.stdin, etc. For example: import sys def main(): pass if __name__ = '__main__': try: ferr = file(stderr_file, 'w') try: fout = file(stdout_file, 'w') try: fin = file(stdin_file, 'r') try: sys.stdin, sys.stdout, sys.stderr = fin, fout, ferr main() finally: fin.close() finally: fout.close() finally: ferr.close() finally: sys.stdin, sys.stdout, sys.stderr = sys.__stdin__, sys.__stdout__, sys.__stderr__ (you'll note that I've done stderr as the outermost so that errors with the others will go to the error file). stdin, stdout, stderr are in the Python 2.3 .chm index and the documentation it links to is (IMO) sufficient. Would you have been affected by the fact that changing these bindings does not affect stdin, etc as used by os.system, popen, etc? > and 2) replacing Python's fopen/stat/fread calls that > do reading of > .py and .pyc files (so that they can be placed in a game's > .dat file or > other proprietary filesystem). The normal way to do this is with an __import__ hook. This has been greatly improved in Python 2.3 (PEP 302). > A third step can be taken > (which replaces the > Python fileobject's calls to fopen/etc with the game ones, > but I think it's > a better idea for the developer to write a python module that > encapsulates > this on their own anyway. Not quite sure what you mean here. What are the semantics of the file object calls that are different to what you needed? > Traditionally we (here at Troika) have done all of the above with a > structure/array of function pointers and initialize it to the > defaults. Just > wanted to know if I were to provide a diff/patch would you > guys want to > include it. Also if I do, what version should I make the > patch for, 2.3.2 or > CVS or? Otherwise I'll just try to get Troika to host the patch on our > website incase other developers want to use it. A patch is definitely welcome, even if not accepted. In general, patches should be made against the latest development branch (2.4) and if appropriate backported to the maintenance branch (2.3.3). However, since your proposals sound like new functionality, they would not be appropriate for a 2.3.x release. Cheers. Tim Delaney From fdrake at acm.org Thu Dec 11 18:16:57 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Dec 11 18:19:28 2003 Subject: [Python-Dev] new module proposal: daemonize In-Reply-To: <200312112307.hBBN7EW02209@c-24-5-183-134.client.comcast.net> References: <1071182938.14238.95.camel@leviathan.kdart.com> <200312112307.hBBN7EW02209@c-24-5-183-134.client.comcast.net> Message-ID: <16344.64233.972116.99631@sftp.fdrake.net> Guido van Rossum writes: > I remember writing a much more elaborate version for Zope: > > http://cvs.zope.org/Zope/lib/python/zdaemon/ I'll note that we (Zope Corp.) have continued to use this package, and have been successful making some truely ill-behaved programs work well as daemons. It would be *wonderful* to see something like this as part of the standard library. I think the only real complaint I have about the package is that the command-line processing module (zdaemon.zdoptions) shouldn't be part of the package, but should be somewhere else. That shouldn't be hard to achieve, though. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From pje at telecommunity.com Thu Dec 11 18:20:03 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Dec 11 18:20:19 2003 Subject: [Python-Dev] new module proposal: logfile In-Reply-To: <1071184212.14243.112.camel@leviathan.kdart.com> References: <5.1.1.6.0.20031211175420.028d8ec0@telecommunity.com> <5.1.1.6.0.20031211175420.028d8ec0@telecommunity.com> Message-ID: <5.1.1.6.0.20031211181852.01db6e60@telecommunity.com> At 03:10 PM 12/11/03 -0800, Keith Dart wrote: >But I think the lesson I get from this is that I should study the Python >2.3 docs. I have not used Python 2.3 much yet. All of the stuff I have >been submitted is old, pre-2.3, and I was not aware that some of it has >been recently added. A good place to start is: http://www.python.org/doc/current/whatsnew/whatsnew23.html From barry at python.org Thu Dec 11 18:25:02 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 11 18:25:21 2003 Subject: [Python-Dev] new module proposal: daemonize In-Reply-To: <200312112307.hBBN7EW02209@c-24-5-183-134.client.comcast.net> References: <1071182938.14238.95.camel@leviathan.kdart.com> <200312112307.hBBN7EW02209@c-24-5-183-134.client.comcast.net> Message-ID: <1071185101.9905.83.camel@anthem> On Thu, 2003-12-11 at 18:07, Guido van Rossum wrote: > This (or something like it) looks like a useful addition. > > I remember writing a much more elaborate version for Zope: > > http://cvs.zope.org/Zope/lib/python/zdaemon/ > > (Ignore Daemon.py, which is the old Zope API for the same thing.) I have something somewhat similar in mailman (although with lots of extra goo): http://cvs.sourceforge.net/viewcvs.py/mailman/mailman/bin/mailmanctl?view=markup Still, something like this would be a nice addition to the standard library. Save's having to read Stevens APitUE Ch. 13. . -Barry From barry at python.org Thu Dec 11 18:27:26 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 11 18:27:34 2003 Subject: [Python-Dev] new module proposal: logfile In-Reply-To: <1071184212.14243.112.camel@leviathan.kdart.com> References: <5.1.1.6.0.20031211175420.028d8ec0@telecommunity.com> <1071184212.14243.112.camel@leviathan.kdart.com> Message-ID: <1071185246.9905.86.camel@anthem> On Thu, 2003-12-11 at 18:10, Keith Dart wrote: > But I think the lesson I get from this is that I should study the Python > 2.3 docs. I have not used Python 2.3 much yet. All of the stuff I have > been submitted is old, pre-2.3, and I was not aware that some of it has > been recently added. Let me just encourage you to keep contributing to Python! Don't let all us ornery curmudgeons discourage you. :) I'd also suggest that for the next round, submit SourceForge patches containing your code. It's way too easy for code contributions to get lost in inboxes and archives. -Barry From tdelaney at avaya.com Thu Dec 11 18:30:39 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Thu Dec 11 18:30:46 2003 Subject: [Python-Dev] "What's New" in docs Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE04BF@au3010avexu1.global.avaya.com> The recent posts, primarily from Keith and Steve (forwarded by me) suggest to me that it might be useful to keep a historical "What's New" section in the documentation, rather than just for the current version. For example, currently the table of contents in the python docs is: Main Page Global Module Index What's New 1 PEP 218: A Standard Set Datatype 2 PEP 255: Simple Generators . . . Would anyone see any advantage to having the "What's New" section like: What's New in... Python 2.3 1 PEP 218: A Standard Set Datatype Python 2.2 1 PEP 234: Iterators . . . I use different versions of Python for different projects (in particular, 2.1 and 2.3) and I think this would be a useful quick reference. Tim Delaney From barry at python.org Thu Dec 11 18:34:38 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 11 18:34:47 2003 Subject: [Python-Dev] new module proposal: daemonize In-Reply-To: <16344.64233.972116.99631@sftp.fdrake.net> References: <1071182938.14238.95.camel@leviathan.kdart.com> <200312112307.hBBN7EW02209@c-24-5-183-134.client.comcast.net> <16344.64233.972116.99631@sftp.fdrake.net> Message-ID: <1071185677.9905.92.camel@anthem> On Thu, 2003-12-11 at 18:16, Fred L. Drake, Jr. wrote: > I think the only real complaint I have about the package is that the > command-line processing module (zdaemon.zdoptions) shouldn't be part > of the package, but should be somewhere else. That shouldn't be hard > to achieve, though. I've been trying to use Greg's optparse package (new in 2.3) on some other ZC projects, and for the most part I like it. It has a different rhythm than getopt, but has some nice properties over what I had been doing, especially I suspect for i18n'd help. I have a few minor quibbles, nits, and bug reports on it, but I'm sold enough to be converting all my getopt-based option parsing over to it for Mailman3. Does zdoptions provide anything that isn't already in supported in optparse? -Barry From fumanchu at amor.org Thu Dec 11 18:37:12 2003 From: fumanchu at amor.org (Robert Brewer) Date: Thu Dec 11 18:42:34 2003 Subject: [Python-Dev] "What's New" in docs Message-ID: I'd appreciate this, or something like it. At the least, it'd be nice to have a link at the bottom of the current "What's New" that points to the last "What's New", or to a complete index on another page. Robert Brewer MIS Amor Ministries fumanchu@amor.org > -----Original Message----- > From: Delaney, Timothy C (Timothy) [mailto:tdelaney@avaya.com] > Sent: Thursday, December 11, 2003 3:31 PM > To: Python-Dev (E-mail) > Subject: [Python-Dev] "What's New" in docs > > > The recent posts, primarily from Keith and Steve (forwarded > by me) suggest to me that it might be useful to keep a > historical "What's New" section in the documentation, rather > than just for the current version. > > For example, currently the table of contents in the python docs is: > > Main Page > Global Module Index > What's New > 1 PEP 218: A Standard Set Datatype > 2 PEP 255: Simple Generators > . > . > . > > Would anyone see any advantage to having the "What's New" > section like: > > What's New in... > Python 2.3 > 1 PEP 218: A Standard Set Datatype > Python 2.2 > 1 PEP 234: Iterators > . > . > . > > I use different versions of Python for different projects (in > particular, 2.1 and 2.3) and I think this would be a useful > quick reference. > > Tim Delaney > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-> dev/fumanchu%40amor.org > From fdrake at acm.org Thu Dec 11 18:43:21 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Dec 11 18:43:31 2003 Subject: [Python-Dev] new module proposal: daemonize In-Reply-To: <1071185677.9905.92.camel@anthem> References: <1071182938.14238.95.camel@leviathan.kdart.com> <200312112307.hBBN7EW02209@c-24-5-183-134.client.comcast.net> <16344.64233.972116.99631@sftp.fdrake.net> <1071185677.9905.92.camel@anthem> Message-ID: <16345.281.537830.208845@sftp.fdrake.net> Barry Warsaw writes: > Does zdoptions provide anything that isn't already in supported in > optparse? Direct support for ZConfig. ;-) That doesn't belong in the standard library... yet. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From barry at python.org Thu Dec 11 18:47:22 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 11 18:47:31 2003 Subject: [Python-Dev] new module proposal: daemonize In-Reply-To: <16345.281.537830.208845@sftp.fdrake.net> References: <1071182938.14238.95.camel@leviathan.kdart.com> <200312112307.hBBN7EW02209@c-24-5-183-134.client.comcast.net> <16344.64233.972116.99631@sftp.fdrake.net> <1071185677.9905.92.camel@anthem> <16345.281.537830.208845@sftp.fdrake.net> Message-ID: <1071186441.9905.95.camel@anthem> On Thu, 2003-12-11 at 18:43, Fred L. Drake, Jr. wrote: > Barry Warsaw writes: > > Does zdoptions provide anything that isn't already in supported in > > optparse? > > Direct support for ZConfig. ;-) That doesn't belong in the standard > library... yet. When you get your wiki going and fix my complaints, we can talk about it . Actually, ZConfig might indeed be a candidate for inclusion. It does provide a much richer configuration system than the existing standard libraries. -Barry From guido at python.org Thu Dec 11 18:53:41 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 11 18:52:08 2003 Subject: [Python-Dev] "What's New" in docs In-Reply-To: Your message of "Thu, 11 Dec 2003 15:37:12 PST." References: Message-ID: <200312112353.hBBNrf502386@c-24-5-183-134.client.comcast.net> > I'd appreciate this, or something like it. At the least, it'd be nice to > have a link at the bottom of the current "What's New" that points to the > last "What's New", or to a complete index on another page. Well, it's not exactly difficult to find the previous versions of the docs from http://www.python.org/doc/ (hint: search for "Previous versions" :-). Isn't that enough? --Guido van Rossum (home page: http://www.python.org/~guido/) From tdelaney at avaya.com Thu Dec 11 19:01:58 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Thu Dec 11 19:02:05 2003 Subject: [Python-Dev] RE: Python and Temple of Elemental Evil Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE04D8@au3010avexu1.global.avaya.com> > From: Guido van Rossum > > > I'm forwarding this to python-dev as it deals with potential > > modifications to the python code base (including offer of a patch :) > > Guido has the final say anyway ... > > The formatting of the message and the extensive quoting and references > to a game made me lose track of what was suggested. Could you > summarize a bit? OK - distilling down to the minimum I think is useful ... [ Steve was unable to work out how to redirect stderr, stdout and stdin ] [ I suggested using sys.stdin, etc - subject to the caveat that it doesn't work for os.system, etc - there may be a documentation issue here though I don't think so - finding the relevant info in the help was simple ]. [ Steve was unable to work out how to import from a proprietory format (.dat) ] [ I suggested an import hook and pointed to PEP 302 (although ToEE uses Python 2.2). There may be a need for further documentation/pointers here. ] [ The following is relevant to python-dev IMO - I should have trimmed further than I did in the first case :( Steve suggests that the file object semantics did not meet his needs, and offers a patch which I expect would be rejected, but may at least give a use case for consideration. ] > A third step can be taken > (which replaces the > Python fileobject's calls to fopen/etc with the game ones, > but I think it's > a better idea for the developer to write a python module that > encapsulates > this on their own anyway. Not quite sure what you mean here. What are the semantics of the file object calls that are different to what you needed? > Traditionally we (here at Troika) have done all of the above with a > structure/array of function pointers and initialize it to the > defaults. Just > wanted to know if I were to provide a diff/patch would you > guys want to > include it. Also if I do, what version should I make the > patch for, 2.3.2 or > CVS or? Otherwise I'll just try to get Troika to host the patch on our > website incase other developers want to use it. A patch is definitely welcome, even if not accepted. In general, patches should be made against the latest development branch (2.4) and if appropriate backported to the maintenance branch (2.3.3). However, since your proposals sound like new functionality, they would not be appropriate for a 2.3.x release. Cheers. Tim Delaney From kdart at kdart.com Thu Dec 11 19:04:45 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 19:04:51 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <200312112300.hBBN0RU02156@c-24-5-183-134.client.comcast.net> References: <1071181041.14243.76.camel@leviathan.kdart.com> <16344.61090.472327.729253@montanaro.dyndns.org> <1071182224.14245.89.camel@leviathan.kdart.com> <200312112300.hBBN0RU02156@c-24-5-183-134.client.comcast.net> Message-ID: <1071187485.14246.134.camel@leviathan.kdart.com> On Thu, 2003-12-11 at 15:00, Guido van Rossum wrote: > I also note that you change the meaning of readline() to return the > line with the trailing separator stripped, and raise EOFError when you > have reached the end. That's a big deviation from the standard file > semantics, and shouldn't be done lightly in a subclass (it means that > a TextFile instance is not substitutable for a regular file instance > in most cases). Yes, that is true, but I have had to use that semantic in other places (usually when using non-blocking sockets) and I personally prefer it. But as others have pointed out this functionality is not needed in Python 2.3 and so I will drop it. > Reviewing the rest of UserFile, I see several possibly useful > additions (locking, fstat() etc.) and one semantic change that > deserves to be a separate subclass or mix-in (the EINTR behavior). > And nothing that warrants the name "UserFile" (in fact, all the > other UserXXX classes are specifically *not* subclasses of the native > XXX class). Yes, a lot of extra stuff was added later. Perhaps some should be factored out into a new module. Perhaps "filelib"? It was just more consistent for me, since I also have UserInt, UserFloat, and UserLong modules as well. But those are, thankfully, no longer needed either. > > No, mode r+ lets you create a _single_ file descriptor that is in > > read-write mode. The MergedIO lets you take who file objects (with > > different file descriptors), one readable the other writable, and > > combine them at the method-call level into one read-write object. This > > is useful for pipes, or when you get file objects from libraries. > > It looks like you are simply making all write operations go to the > writable object and all the read operations go to the readable object. > That's not a very sophisticated notion of "merging", and I'm not sure > what the point is. Perhaps some examples of how you use this would > help. A quick example: import os import UserFile r, w = os.pipe() rf = os.fdopen(r) wf = os.fdopen(w) fo = UserFile.MergedIO(rf, wf) _or_ rwio = UserFile.MergedIO(sys.stdin, sys.stdout) Perhaps "merged" is not the right word. It is not intended to merge files, but combine a separate read file object and write file object into one object that can be passed to other library methods that want a single read/write object (such as my "expect" module...) > I note that in general your proposals are low on motivation, which is > going to do a lot of damage to your case if you don't fix this soon. > You may be underestimating the level of motivation needed to get > features added to the stdlib -- it's "batteries included" but also (to > some extent) "TOOWTDI". Hm, ok. more use cases? What I have submitted, and will submit, exist because I did need them for real work at some point, and have found them to be re-usable. -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/912b733d/attachment-0001.bin From guido at python.org Thu Dec 11 19:16:37 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 11 19:14:36 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: Your message of "Thu, 11 Dec 2003 16:04:45 PST." <1071187485.14246.134.camel@leviathan.kdart.com> References: <1071181041.14243.76.camel@leviathan.kdart.com> <16344.61090.472327.729253@montanaro.dyndns.org> <1071182224.14245.89.camel@leviathan.kdart.com> <200312112300.hBBN0RU02156@c-24-5-183-134.client.comcast.net> <1071187485.14246.134.camel@leviathan.kdart.com> Message-ID: <200312120016.hBC0Gbp02447@c-24-5-183-134.client.comcast.net> > > It looks like you are simply making all write operations go to the > > writable object and all the read operations go to the readable object. > > That's not a very sophisticated notion of "merging", and I'm not sure > > what the point is. Perhaps some examples of how you use this would > > help. > > A quick example: > > import os > import UserFile > > r, w = os.pipe() > rf = os.fdopen(r) > wf = os.fdopen(w) > > fo = UserFile.MergedIO(rf, wf) > > _or_ > > rwio = UserFile.MergedIO(sys.stdin, sys.stdout) That's not a motivation. How does writing rwio.readline() and rw.write() enable your program to do something that wasn't possible by using sys.stdin and sys.stdout directly? > Perhaps "merged" is not the right word. It is not intended to merge > files, but combine a separate read file object and write file object > into one object that can be passed to other library methods that want a > single read/write object (such as my "expect" module...) Sounds like an API design bug in the expect module to me. :-) > > I note that in general your proposals are low on motivation, which is > > going to do a lot of damage to your case if you don't fix this soon. > > You may be underestimating the level of motivation needed to get > > features added to the stdlib -- it's "batteries included" but also (to > > some extent) "TOOWTDI". > > Hm, ok. more use cases? What I have submitted, and will submit, exist > because I did need them for real work at some point, and have found them > to be re-usable. I don't doubt that. But you need to *show* how they are useful for others, and you need to show that you have thought about how they would work well together with the rest of the stdio library. (This is where strip-newline-raise-EOFError breaks down.) --Guido van Rossum (home page: http://www.python.org/~guido/) From andrew-pythondev at puzzling.org Thu Dec 11 19:16:26 2003 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Thu Dec 11 19:16:34 2003 Subject: [Python-Dev] new module proposal: daemonize In-Reply-To: <1071185101.9905.83.camel@anthem> References: <1071182938.14238.95.camel@leviathan.kdart.com> <200312112307.hBBN7EW02209@c-24-5-183-134.client.comcast.net> <1071185101.9905.83.camel@anthem> Message-ID: <20031212001626.GA25007@frobozz> On Thu, Dec 11, 2003 at 06:25:02PM -0500, Barry Warsaw wrote: > On Thu, 2003-12-11 at 18:07, Guido van Rossum wrote: > > > This (or something like it) looks like a useful addition. > > > > I remember writing a much more elaborate version for Zope: > > > > http://cvs.zope.org/Zope/lib/python/zdaemon/ > > > > (Ignore Daemon.py, which is the old Zope API for the same thing.) > > I have something somewhat similar in mailman (although with lots of > extra goo): > > http://cvs.sourceforge.net/viewcvs.py/mailman/mailman/bin/mailmanctl?view=markup ...And Twisted has its own version of this stuff, too: http://cvs.twistedmatrix.com/cvs/twisted/scripts/twistd.py?rev=1.65&content-type=text/vnd.viewcvs-markup *And* there's even a cookbook recipe for it (with lots of debate about the details): http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66012 I'm sure there are plenty of other projects that have reinvented this, too. Having daemonization code in the standard library would be good. -Andrew. From kdart at kdart.com Thu Dec 11 19:22:39 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 19:22:45 2003 Subject: [Python-Dev] new module: scheduler Message-ID: <1071188559.14245.141.camel@leviathan.kdart.com> Skipped content of type multipart/mixed-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/3662d96a/attachment.bin From kdart at kdart.com Thu Dec 11 19:26:15 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 19:26:20 2003 Subject: [Python-Dev] new module proposal: daemonize In-Reply-To: <20031212001626.GA25007@frobozz> References: <1071182938.14238.95.camel@leviathan.kdart.com> <200312112307.hBBN7EW02209@c-24-5-183-134.client.comcast.net> <1071185101.9905.83.camel@anthem> <20031212001626.GA25007@frobozz> Message-ID: <1071188775.14243.146.camel@leviathan.kdart.com> On Thu, 2003-12-11 at 16:16, Andrew Bennetts wrote: > ...And Twisted has its own version of this stuff, too: > > http://cvs.twistedmatrix.com/cvs/twisted/scripts/twistd.py?rev=1.65&content-type=text/vnd.viewcvs-markup > > *And* there's even a cookbook recipe for it (with lots of debate about the > details): > > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66012 > > I'm sure there are plenty of other projects that have reinvented this, too. > Having daemonization code in the standard library would be good. Yes, I think I got some of the code in daemonize from the cookbook. I had been using a little C program for that (e.g.: "daemonize python pydaemon.py" ) but that required invoking from a shell, or remebering to use the daemonize tool. With the daemonize module I could do it from within python in a standard way. It may need some tweaking to make it more portable. -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/04ce81d1/attachment.bin From guido at python.org Thu Dec 11 19:36:37 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 11 19:34:38 2003 Subject: [Python-Dev] new module proposal: daemonize In-Reply-To: Your message of "Thu, 11 Dec 2003 16:26:15 PST." <1071188775.14243.146.camel@leviathan.kdart.com> References: <1071182938.14238.95.camel@leviathan.kdart.com> <200312112307.hBBN7EW02209@c-24-5-183-134.client.comcast.net> <1071185101.9905.83.camel@anthem> <20031212001626.GA25007@frobozz> <1071188775.14243.146.camel@leviathan.kdart.com> Message-ID: <200312120036.hBC0ab102521@c-24-5-183-134.client.comcast.net> > Yes, I think I got some of the code in daemonize from the cookbook. I > had been using a little C program for that (e.g.: "daemonize python > pydaemon.py" ) but that required invoking from a shell, or remebering to > use the daemonize tool. With the daemonize module I could do it from > within python in a standard way. It may need some tweaking to make it > more portable. I note that the original Zope approachg to daemonizing was also like you do it, from inside. But when we decided to rewrite it, we found that it was much more general to have an external program... --Guido van Rossum (home page: http://www.python.org/~guido/) From kdart at kdart.com Thu Dec 11 19:35:28 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 19:35:33 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <200312120016.hBC0Gbp02447@c-24-5-183-134.client.comcast.net> References: <1071181041.14243.76.camel@leviathan.kdart.com> <16344.61090.472327.729253@montanaro.dyndns.org> <1071182224.14245.89.camel@leviathan.kdart.com> <200312112300.hBBN0RU02156@c-24-5-183-134.client.comcast.net> <1071187485.14246.134.camel@leviathan.kdart.com> <200312120016.hBC0Gbp02447@c-24-5-183-134.client.comcast.net> Message-ID: <1071189327.14246.154.camel@leviathan.kdart.com> On Thu, 2003-12-11 at 16:16, Guido van Rossum wrote: > > rwio = UserFile.MergedIO(sys.stdin, sys.stdout) > > That's not a motivation. How does writing rwio.readline() and > rw.write() enable your program to do something that wasn't possible by > using sys.stdin and sys.stdout directly? _MY_ programs could do that, but what if you got some third party module that wanted a single read/write file object and you wanted to use it on stdio? It makes APIs more regular and consistent by requiring only one read/write object. For example, the library method could be passed a single read/write Unix socket, or a read/write TCP socket, or stdio object (MergedIO object). > > Perhaps "merged" is not the right word. It is not intended to merge > > files, but combine a separate read file object and write file object > > into one object that can be passed to other library methods that want a > > single read/write object (such as my "expect" module...) > > Sounds like an API design bug in the expect module to me. :-) Nope, that's a feature. ;-) > I don't doubt that. But you need to *show* how they are useful for > others, and you need to show that you have thought about how they > would work well together with the rest of the stdio library. (This is > where strip-newline-raise-EOFError breaks down.) Ok, this one can be removed then. It would not be commonly used anyway. -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/3761f30d/attachment.bin From kdart at kdart.com Thu Dec 11 19:52:14 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 19:52:21 2003 Subject: [Python-Dev] new module proposal: expect Message-ID: <1071190333.14243.163.camel@leviathan.kdart.com> Skipped content of type multipart/mixed-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/10e66d73/attachment-0001.bin From guido at python.org Thu Dec 11 19:57:05 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 11 19:55:02 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: Your message of "Thu, 11 Dec 2003 16:35:28 PST." <1071189327.14246.154.camel@leviathan.kdart.com> References: <1071181041.14243.76.camel@leviathan.kdart.com> <16344.61090.472327.729253@montanaro.dyndns.org> <1071182224.14245.89.camel@leviathan.kdart.com> <200312112300.hBBN0RU02156@c-24-5-183-134.client.comcast.net> <1071187485.14246.134.camel@leviathan.kdart.com> <200312120016.hBC0Gbp02447@c-24-5-183-134.client.comcast.net> <1071189327.14246.154.camel@leviathan.kdart.com> Message-ID: <200312120057.hBC0v5602569@c-24-5-183-134.client.comcast.net> > > > rwio = UserFile.MergedIO(sys.stdin, sys.stdout) > > That's not a motivation. How does writing rwio.readline() and > > rw.write() enable your program to do something that wasn't > > possible by using sys.stdin and sys.stdout directly? > > _MY_ programs could do that, but what if you got some third party > module that wanted a single read/write file object and you wanted to > use it on stdio? > It makes APIs more regular and consistent by requiring only one > read/write object. For example, the library method could be passed a > single read/write Unix socket, or a read/write TCP socket, or stdio > object (MergedIO object). Well, the standard way of wrapping a file object around a socket pretty much forces you to have separate read and write objects; check out the makefile() method. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Thu Dec 11 19:57:07 2003 From: aahz at pythoncraft.com (Aahz) Date: Thu Dec 11 19:57:10 2003 Subject: [Python-Dev] new module proposal: expect In-Reply-To: <1071190333.14243.163.camel@leviathan.kdart.com> References: <1071190333.14243.163.camel@leviathan.kdart.com> Message-ID: <20031212005707.GA26221@panix.com> On Thu, Dec 11, 2003, Keith Dart wrote: > > It is sometimes necessary to interact with other programs, or users, as > a user would via a command line interface. This is especially common in > QA environments. A program called "expect", which is based on the Tcl > language, has commonly been used for this. But that program is, well, > not Python. ;-) Having a general purpose "expect" module for Python > means that Python could be used out-of-the-box for the same tasks that > the older "expect" program is used for. The attached expect.py module > provides most of the core functionality of the original expect tool. > Other functionality from the "expect" tool is duplicated in other Python > modules. How does this differ from http://pexpect.sourceforge.net/ -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From guido at python.org Thu Dec 11 19:59:22 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 11 19:57:19 2003 Subject: [Python-Dev] new module proposal: expect In-Reply-To: Your message of "Thu, 11 Dec 2003 16:52:14 PST." <1071190333.14243.163.camel@leviathan.kdart.com> References: <1071190333.14243.163.camel@leviathan.kdart.com> Message-ID: <200312120059.hBC0xM502597@c-24-5-183-134.client.comcast.net> > It is sometimes necessary to interact with other programs, or users, as > a user would via a command line interface. This is especially common in > QA environments. A program called "expect", which is based on the Tcl > language, has commonly been used for this. But that program is, well, > not Python. ;-) Having a general purpose "expect" module for Python > means that Python could be used out-of-the-box for the same tasks that > the older "expect" program is used for. The attached expect.py module > provides most of the core functionality of the original expect tool. > Other functionality from the "expect" tool is duplicated in other Python > modules. > > It is hoped that this will enable non-Python users to easily replace any > "expect" installations they have with Python. Thus, growing the Python > market. Having this functionality as part of the core distribution would > facilitate that. Good, this is the kind of module that would make the Python standard library more useful. However, I've seen many implementations of this before. Before yours gets accepted into the standard library, you have to convince us that your version is "best of breed". That's been done before: the logging package and the optparse module came about this way. But you have to do the research to compare your version with others. I recommend that you take this to c.l.py first. --Guido van Rossum (home page: http://www.python.org/~guido/) From kdart at kdart.com Thu Dec 11 20:04:18 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 20:04:24 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <200312120057.hBC0v5602569@c-24-5-183-134.client.comcast.net> References: <1071181041.14243.76.camel@leviathan.kdart.com> <16344.61090.472327.729253@montanaro.dyndns.org> <1071182224.14245.89.camel@leviathan.kdart.com> <200312112300.hBBN0RU02156@c-24-5-183-134.client.comcast.net> <1071187485.14246.134.camel@leviathan.kdart.com> <200312120016.hBC0Gbp02447@c-24-5-183-134.client.comcast.net> <1071189327.14246.154.camel@leviathan.kdart.com> <200312120057.hBC0v5602569@c-24-5-183-134.client.comcast.net> Message-ID: <1071191058.14244.168.camel@leviathan.kdart.com> On Thu, 2003-12-11 at 16:57, Guido van Rossum wrote: > > > > rwio = UserFile.MergedIO(sys.stdin, sys.stdout) > > > > That's not a motivation. How does writing rwio.readline() and > > > rw.write() enable your program to do something that wasn't > > > possible by using sys.stdin and sys.stdout directly? > > > > _MY_ programs could do that, but what if you got some third party > > module that wanted a single read/write file object and you wanted to > > use it on stdio? > > > It makes APIs more regular and consistent by requiring only one > > read/write object. For example, the library method could be passed a > > single read/write Unix socket, or a read/write TCP socket, or stdio > > object (MergedIO object). > > Well, the standard way of wrapping a file object around a socket > pretty much forces you to have separate read and write objects; check > out the makefile() method. :-) Ok, ok. Uncle! ;-) Guess what? I have never used MergedIO myself. It was partly an exercise combining object functionality in an efficient manner (by re-assigning method objects in the instance's namespace). I can drop that one. But... there is a variant in another module that I use. -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/be48b990/attachment.bin From kdart at kdart.com Thu Dec 11 20:14:21 2003 From: kdart at kdart.com (Keith Dart) Date: Thu Dec 11 20:14:26 2003 Subject: [Python-Dev] new module proposal: expect In-Reply-To: <200312120059.hBC0xM502597@c-24-5-183-134.client.comcast.net> References: <1071190333.14243.163.camel@leviathan.kdart.com> <200312120059.hBC0xM502597@c-24-5-183-134.client.comcast.net> Message-ID: <1071191660.14243.178.camel@leviathan.kdart.com> On Thu, 2003-12-11 at 16:59, Guido van Rossum wrote: > Good, this is the kind of module that would make the Python standard > library more useful. However, I've seen many implementations of this > before. Before yours gets accepted into the standard library, you > have to convince us that your version is "best of breed". That's been > done before: the logging package and the optparse module came about > this way. But you have to do the research to compare your version > with others. I recommend that you take this to c.l.py first. On Thu, 2003-12-11 at 16:57, Aahz wrote: > How does this differ from http://pexpect.sourceforge.net/ I knew you would ask that. I did look at those others, and compared them. But ultimately I wrote my own from scratch. Of course, I like mine a lot better... I will provide more details later, as it would require quite a treatise on that subject. -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/7929b774/attachment.bin From aahz at pythoncraft.com Thu Dec 11 20:20:48 2003 From: aahz at pythoncraft.com (Aahz) Date: Thu Dec 11 20:20:52 2003 Subject: [Python-Dev] new module proposal: expect In-Reply-To: <1071191660.14243.178.camel@leviathan.kdart.com> References: <1071190333.14243.163.camel@leviathan.kdart.com> <200312120059.hBC0xM502597@c-24-5-183-134.client.comcast.net> <1071191660.14243.178.camel@leviathan.kdart.com> Message-ID: <20031212012048.GA2401@panix.com> On Thu, Dec 11, 2003, Keith Dart wrote: > On Thu, 2003-12-11 at 16:57, Aahz wrote: >> >> How does this differ from http://pexpect.sourceforge.net/ > > I knew you would ask that. I did look at those others, and compared > them. But ultimately I wrote my own from scratch. Of course, I like mine > a lot better... I will provide more details later, as it would require > quite a treatise on that subject. This will almost certainly require a PEP, I think; might as well start working on that now. ;-) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From wsanchez at apple.com Thu Dec 11 21:12:19 2003 From: wsanchez at apple.com (=?ISO-8859-1?Q?Wilfredo_S=E1nchez?=) Date: Thu Dec 11 21:12:28 2003 Subject: [Python-Dev] new module proposal: daemonize In-Reply-To: <1071182938.14238.95.camel@leviathan.kdart.com> References: <1071182938.14238.95.camel@leviathan.kdart.com> Message-ID: <9DB7F042-2C48-11D8-B711-000A95AFAE2C@apple.com> Skipped content of type multipart/mixed-------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2406 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20031211/e2ae17aa/smime-0001.bin From devin at whitebread.org Fri Dec 12 00:34:31 2003 From: devin at whitebread.org (Devin) Date: Thu Dec 11 22:26:02 2003 Subject: [Python-Dev] new module: scheduler In-Reply-To: <1071188559.14245.141.camel@leviathan.kdart.com> Message-ID: I wanted to lurk a while longer before posting again, but I feel I have something worthwhile to add. On Thu, 11 Dec 2003, Keith Dart wrote: > Whew, ok. Here is another one. Perhaps this one is misnamed, but it is > called scheduler and takes over SIGALRM to provide running callback > functions at scheduled intervals or time delays. Also provides an > alternate implementation for "sleep". It operates asynchronously. That > is, your main thread of execution still runs and is interrupted by > SIGALRM and your specified callbacks are run. > > I am aware of the sched module, but that one is for threads, and is not > asynchronous (your main thread blocks in the run() method). A little more than a year ago, I also wrote a threaded scheduler. The code can be found here: http://www.whitebread.org/tidbits/scheduler Since then, I've found the functionality of a threaded scheduler to be very useful, and believe that _a_ threaded scheduler (not necessarily my impementation) would be a nice addition to the standard library. Good suggestion, Keith. :) ::Lurk mode on:: -- Devin devin@whitebread.org http://www.whitebread.org/ From skip at pobox.com Thu Dec 11 22:53:25 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 11 22:53:24 2003 Subject: [Python-Dev] "What's New" in docs In-Reply-To: <200312112353.hBBNrf502386@c-24-5-183-134.client.comcast.net> References: <200312112353.hBBNrf502386@c-24-5-183-134.client.comcast.net> Message-ID: <16345.15285.583427.717748@montanaro.dyndns.org> >> ... it'd be nice to have a link at the bottom of the current "What's >> New" that points to the last "What's New", or to a complete index on >> another page. Guido> Well, it's not exactly difficult to find the previous versions of Guido> the docs from http://www.python.org/doc/ (hint: search for Guido> "Previous versions" :-). Guido> Isn't that enough? Sure, I suppose it's sufficient, but perhaps a bit cumbersome. If you're trying to remember when some feature was added to the language (which is fairly common if you're trying to support an app on multiple versions of the interpreter I think), chaining straight back through What's New sections would make it easier. Skip From guido at python.org Thu Dec 11 23:41:01 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 11 23:39:00 2003 Subject: [Python-Dev] "What's New" in docs In-Reply-To: Your message of "Thu, 11 Dec 2003 21:53:25 CST." <16345.15285.583427.717748@montanaro.dyndns.org> References: <200312112353.hBBNrf502386@c-24-5-183-134.client.comcast.net> <16345.15285.583427.717748@montanaro.dyndns.org> Message-ID: <200312120441.hBC4f1602858@c-24-5-183-134.client.comcast.net> > Guido> Well, it's not exactly difficult to find the previous versions of > Guido> the docs from http://www.python.org/doc/ (hint: search for > Guido> "Previous versions" :-). > > Guido> Isn't that enough? > > Sure, I suppose it's sufficient, but perhaps a bit cumbersome. If you're > trying to remember when some feature was added to the language (which is > fairly common if you're trying to support an app on multiple versions of the > interpreter I think), chaining straight back through What's New sections > would make it easier. The standard docs are sprinkled with "new in version 2.x" as well. So I'm still questioning the value of keeping the "what's new in 2.x" document around as part of the 2.(x+1) docs. If you work with 2.x and 2.y, you should probably keep bookmarks for both sets of docs... --Guido van Rossum (home page: http://www.python.org/~guido/) From python at rcn.com Fri Dec 12 00:14:57 2003 From: python at rcn.com (Raymond Hettinger) Date: Fri Dec 12 00:15:09 2003 Subject: [Python-Dev] "What's New" in docs In-Reply-To: <200312120441.hBC4f1602858@c-24-5-183-134.client.comcast.net> Message-ID: <000c01c3c06e$e346a820$e841fea9@oemcomputer> > > Guido> Well, it's not exactly difficult to find the previous > versions of > > Guido> the docs from http://www.python.org/doc/ (hint: search for > > Guido> "Previous versions" :-). > > > > Guido> Isn't that enough? My preference is to keep What's New separate. For most users, the primary value is in the moment of transition during an upgrade. Also, with the Mac people coming aboard, we'll have more users that have never seen a previous version of Python. Ideally, they should see Python as being in a state of highest evolution without seeing the whole history. For someone truly interested in writing code for multiple python versions, the whatsnew history is neither sufficiently complete nor detailed. Instead, the text and examples are written as gentle introductions to features. When Skip looks up the introduction of list comps or generators, the odds are that he only reads the headlines and would be bored to tears by the other 98% of the document. When the introduction of CHM files, searching the regular docs has never been easier. Raymond Hettinger From fumanchu at amor.org Fri Dec 12 01:15:34 2003 From: fumanchu at amor.org (Robert Brewer) Date: Fri Dec 12 01:20:55 2003 Subject: [Python-Dev] "What's New" in docs Message-ID: Guido van Rossum wrote: > > I'd appreciate this, or something like it. At the least, > it'd be nice to > > have a link at the bottom of the current "What's New" that > points to the > > last "What's New", or to a complete index on another page. > > Well, it's not exactly difficult to find the previous versions of the > docs from http://www.python.org/doc/ (hint: search for "Previous > versions" :-). > > Isn't that enough? For the moment, it works. It's more a matter of indexing and navigation than having "a way" to find it--I guess I'm officially from the "one-click" generation. As Python releases more versions, it will naturally become less manageable. "What's New" is basically 'indexed' by version number currently. It'd be nice to have a feature index, for reverse lookups. Hope that makes sense. My thought about having a pointer to the previous "What's New" would speed navigation through several versions, if an index were not appropriate. Me, I just use Google. :) FuManChu From barry at python.org Fri Dec 12 08:03:09 2003 From: barry at python.org (Barry Warsaw) Date: Fri Dec 12 08:03:24 2003 Subject: [Python-Dev] "What's New" in docs In-Reply-To: <200312120441.hBC4f1602858@c-24-5-183-134.client.comcast.net> References: <200312112353.hBBNrf502386@c-24-5-183-134.client.comcast.net> <16345.15285.583427.717748@montanaro.dyndns.org> <200312120441.hBC4f1602858@c-24-5-183-134.client.comcast.net> Message-ID: <1071234188.9905.114.camel@anthem> On Thu, 2003-12-11 at 23:41, Guido van Rossum wrote: > The standard docs are sprinkled with "new in version 2.x" as well. > > So I'm still questioning the value of keeping the "what's new in 2.x" > document around as part of the 2.(x+1) docs. If you work with 2.x and > 2.y, you should probably keep bookmarks for both sets of docs... PEP 291 also has a fairly high level overview of when certain features were introduced. -Barry From mwh at python.net Fri Dec 12 09:29:06 2003 From: mwh at python.net (Michael Hudson) Date: Fri Dec 12 09:29:09 2003 Subject: [Python-Dev] Re: a serious threat to 2.3's speed? In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0458@au3010avexu1.global.avaya.com> (Timothy C. Delaney's message of "Fri, 12 Dec 2003 06:55:09 +1100") References: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0458@au3010avexu1.global.avaya.com> Message-ID: <2mu146kt3h.fsf@starship.python.net> "Delaney, Timothy C (Timothy)" writes: >> From: Fredrik Lundh >> >> key features contributing to this is true gc, call-site >> caching, and "traditional" >> (C-style) argument passing for common cases. > > Did you collect stats on how much garbage collection actually > occurred during the tests? How did it compare to stock CPython? > > One of the big flaws in benchmarking early versions of Java of > course was that most tests didn't end up invoking the garbage > collector - hence showing an artificially good result that could not > be observed in real-world usage. If you read the pytte1 pages you can find /F talking about how accidentally disabling the GC wrecked performance. My suspicion is that Python apps allocate and deallocate so many objects that not having a GC around leads to ferociously cache hostile behaviour. Cheers, mwh -- Famous remarks are very seldom quoted correctly. -- Simeon Strunsky From python at rcn.com Fri Dec 12 10:55:04 2003 From: python at rcn.com (Raymond Hettinger) Date: Fri Dec 12 10:55:18 2003 Subject: [Python-Dev] FW: METH_COEXIST Message-ID: <000201c3c0c8$4f3a0320$e841fea9@oemcomputer> Long ago, I noticed that non-operator calls to dict.__contains__() ran about 50% slower than dict.has_key(). Since then, the issue has surfaced several times for other wrapped methods like __init__(), __getitem__(), and __setitem__(), and next(). Also, it came up again in the context of set.__contains__. The latter bugged me enough to get to the bottom of it (unlike dicts, sets do not have has_key() as an alternative so you feel it when writing ismember=myset.__contains__). Over 95% of the speed difference is due to PyCFunctions having optimized calls. Of that, much of the benefit comes from the METH_O and METH_NOARGS methods having their arguments passed directly instead of by assembling and disassembling an argument tuple. If the problem were widespread and important in every context, the solution would be to provide wrapperobjects with special cases just like PyCFunction. Instead, there is a minimally invasive alternative. If I could just write a "__contains__" method as a PyCFunction, the job would be done. Unfortunately, if the corresponding slot has already filled-in the method with a wrapper object, the explicitly defined method is ignored. While that is normally fine, it is possible to make the explicitly defined method coexist with the slot. So, I propose to add a method flag, METH_COEXIST, which allows an entry in the method table to overwrite the wrapperobject and coexist with the slot: *** typeobject.c 13 Nov 2003 22:50:00 -0000 2.253 --- typeobject.c 12 Dec 2003 13:58:42 -0000 *************** *** 2792,2799 **** for (; meth->ml_name != NULL; meth++) { PyObject *descr; ! if (PyDict_GetItemString(dict, meth->ml_name)) ! continue; if (meth->ml_flags & METH_CLASS) { if (meth->ml_flags & METH_STATIC) { PyErr_SetString(PyExc_ValueError, --- 2792,2802 ---- for (; meth->ml_name != NULL; meth++) { PyObject *descr; ! if (PyDict_GetItemString(dict, meth->ml_name)) { ! if (!(meth->ml_flags & METH_COEXIST)) ! continue; ! PyDict_DelItemString(dict, meth->ml_name); ! } if (meth->ml_flags & METH_CLASS) { if (meth->ml_flags & METH_STATIC) { PyErr_SetString(PyExc_ValueError, With that minor change, it becomes possible to write: static PyMethodDef mapp_methods[] = { {"__contains__", (PyCFunction)dict_has_key, METH_O | METH_COEXIST, contains__doc__}, And, presto, now calls to dict.__contains__() are as fast as dict.has_key(). The two immediate applications are dict.__contains__() and set.__contains__(). Guido is basically okay with the idea but thought I should check here first to see if anyone has any profound thoughts about unexpected implications or perhaps a better way. If not, then I would like to make the above changes (define the flag, teach typeobject.c to recognize the flag, and apply it to dicts and sets). Raymond Hettinger From pje at telecommunity.com Fri Dec 12 11:17:40 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri Dec 12 11:18:04 2003 Subject: [Python-Dev] FW: METH_COEXIST In-Reply-To: <000201c3c0c8$4f3a0320$e841fea9@oemcomputer> Message-ID: <5.1.1.6.0.20031212111653.00a5a2c0@telecommunity.com> At 10:55 AM 12/12/03 -0500, Raymond Hettinger wrote: >Guido is basically okay with the idea but thought I should check here >first to see if anyone has any profound thoughts about unexpected >implications or perhaps a better way. What happens in subclasses? From mwh at python.net Fri Dec 12 11:22:54 2003 From: mwh at python.net (Michael Hudson) Date: Fri Dec 12 11:22:57 2003 Subject: [Python-Dev] FW: METH_COEXIST In-Reply-To: <000201c3c0c8$4f3a0320$e841fea9@oemcomputer> (Raymond Hettinger's message of "Fri, 12 Dec 2003 10:55:04 -0500") References: <000201c3c0c8$4f3a0320$e841fea9@oemcomputer> Message-ID: <2mpteukntt.fsf@starship.python.net> "Raymond Hettinger" writes: > Guido is basically okay with the idea but thought I should check here > first to see if anyone has any profound thoughts about unexpected > implications or perhaps a better way. Not really, this sounds pretty reasonable. Two thoughts: 1) Man, I hate C. Not news :-) 2) /Might/ it be possible to do this automatically for a few slots? (i.e. are there any slots that have parameter lists that let the the tp_foo implementation function just be wrapped up in a METH_O or METH_NOARGS PyCFunction?). Bit of a hack, mind. Cheers, mwh -- Gullible editorial staff continues to post links to any and all articles that vaguely criticize Linux in any way. -- Reason #4 for quitting slashdot today, from http://www.cs.washington.edu/homes/klee/misc/slashdot.html From python at rcn.com Fri Dec 12 16:31:29 2003 From: python at rcn.com (Raymond Hettinger) Date: Fri Dec 12 16:31:41 2003 Subject: [Python-Dev] FW: METH_COEXIST In-Reply-To: <2mpteukntt.fsf@starship.python.net> Message-ID: <002201c3c0f7$4e5d5360$e841fea9@oemcomputer> [Raymond] > > Guido is basically okay with the idea but thought I should check here > > first to see if anyone has any profound thoughts about unexpected > > implications or perhaps a better way. > [Michael Hudson] > Not really, this sounds pretty reasonable. Two thoughts: > > 1) Man, I hate C. Not news :-) It certainly makes you appreciate coding in Python :-) > 2) /Might/ it be possible to do this automatically for a few slots? > (i.e. are there any slots that have parameter lists that let the > the tp_foo implementation function just be wrapped up in a METH_O > or METH_NOARGS PyCFunction?). Bit of a hack, mind. This idea may be doable but it certainly isn't minimally invasive ;-) One approach would entail: * expand the structure for PyCFunction to include the slot pointer * create an alternate constructor for PyCFunction that loads the slot pointer, flags, doc string, and a pointer to the wrapper function. * change all the wrapper functions match the signature of the function they are calling * have the wrapper functions become responsible for looking up the slot pointer from the expanded PyCFunction structure * eliminate the wrapper object type The bad news is that this would take a weekend of effort and I'm not 100% sure it would work. If it did, it would be slightly slower that what I've proposed. This would decrease the cohesion of PyCFunction and impose additional coupling to the wrapped functions (though the latter is minimized somewhat by the alternate constructor which hides the details of where wrapper information came from). The good news is that it eliminates an entire type. Also, it is much more general that what I proposed. And, any future enhancements to the calling mechanism would have once less case to worry about. Netting the good with the bad, it comes out close. Unfortunately, it could eat-up a whole weekend to find out whether simplification and generality of it outweigh the little hacks, the large number of lines that would change, and the relatively high risk of introducing some subtle bugs. [Phillip J. Eby] > What happens in subclasses? Nothing special. The behavior persists unless overridden by a subclass method. Raymond Hettinger From martin at v.loewis.de Fri Dec 12 17:55:38 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Fri Dec 12 17:56:02 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <1071181041.14243.76.camel@leviathan.kdart.com> References: <1071181041.14243.76.camel@leviathan.kdart.com> Message-ID: Keith Dart writes: > Attached is a module provides enhanced functionality for file objects. If you want your code to be included in Python, posting it to python-dev may not be a good idea - it is, in fact, a guarantee that it will *not* be included. Instead, all patches should go to sf.net/projects/python. That is not a guarantee that it will be included, either. This module (UserFile) puts together way too many individual feature to allow a meaningful review. Looking at a single class alone (UserFile), my review would be this: 1. What is the purpose of hiding EINTR? 2. Many of the os wrappers should be methods on the file object, if they are useful in the first place (e.g. ttyname, fstat, dup) 3. This code might not be universally work on all supported systems; this should be addressed somehow. 4. There is no documentation. 5. There are no test cases. Regards, Martin From kdart at kdart.com Fri Dec 12 18:38:43 2003 From: kdart at kdart.com (Keith Dart) Date: Fri Dec 12 18:38:46 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: References: <1071181041.14243.76.camel@leviathan.kdart.com> Message-ID: <1071272322.20456.13.camel@leviathan.kdart.com> On Fri, 2003-12-12 at 14:55, Martin v. L?wis wrote: > Keith Dart writes: > > > Attached is a module provides enhanced functionality for file objects. > > If you want your code to be included in Python, posting it to > python-dev may not be a good idea - it is, in fact, a guarantee that > it will *not* be included. Instead, all patches should go to > sf.net/projects/python. Thank you for responding. The purpose of posting here was to get some feedback on the modules (which I certainly did!). I will clean up and post the ones that had the most interest to sourceforge. > That is not a guarantee that it will be included, either. This module > (UserFile) puts together way too many individual feature to allow a > meaningful review. Yes, I agree. It should be broken up into at least two modules. > Looking at a single class alone (UserFile), my review would be this: > > 1. What is the purpose of hiding EINTR? My code uses a lot of interrupts (SIGALRM, SIGIO, SIGCHLD, ...) and I almost always need to trap and restart interrupted system calls. So I made that wrapper. Nearly all I/O uses it. In fact, I think it should be the default behavior.... > 2. Many of the os wrappers should be methods on the file object, > if they are useful in the first place (e.g. ttyname, fstat, > dup) I agree again. I don't know why that is not currently the case. So I have to write a wrapper (not wanting to have a custom hacked Python installation). > 3. This code might not be universally work on all supported systems; > this should be addressed somehow. True again. I write and deploy only on Posix (Linux and FreeBSD) so perhaps others will have to add other platform support. > 4. There is no documentation. > 5. There are no test cases. Again, these were posted just to get some valuable feedback before I did those things. Thanks for your feedback! -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031212/67b38533/attachment.bin From guido at python.org Fri Dec 12 19:07:03 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 12 19:07:09 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: Your message of "Fri, 12 Dec 2003 15:38:43 PST." <1071272322.20456.13.camel@leviathan.kdart.com> References: <1071181041.14243.76.camel@leviathan.kdart.com> <1071272322.20456.13.camel@leviathan.kdart.com> Message-ID: <200312130007.hBD073u04705@c-24-5-183-134.client.comcast.net> > > 1. What is the purpose of hiding EINTR? > > My code uses a lot of interrupts (SIGALRM, SIGIO, SIGCHLD, ...) and I > almost always need to trap and restart interrupted system calls. So I > made that wrapper. Nearly all I/O uses it. In fact, I think it should be > the default behavior.... Here we may have a fundamental disagreement. In my view it is impossible to write correct code when signal handlers can run any time, and therefore I have given up on using signals at all. Everything you do with signals can also be done without signals, and usually more portably. (You may have to use threads.) Making read and write ignore EINTR doesn't solve all signal-related problems, it just makes them rarer, thereby *increasing* the possibility that despite thorough testing your code will be incorrect, and you will face irreproducible but real error reports from the field. Also note that Python takes non-Posix systems very seriously, and augmentations to standard object types that only work on Posix are generally rejected. (isatty() is an old wart and would these days probably not have been accepted as a file object method.) --Guido van Rossum (home page: http://www.python.org/~guido/) From python-forums at claytonbrown.net Fri Dec 12 20:15:38 2003 From: python-forums at claytonbrown.net (python-forums@claytonbrown.net) Date: Fri Dec 12 20:15:34 2003 Subject: [Python-Dev] Hello World Problem :: Apache/2.0.48 (Win32) mod_python/3.0.4 Python/2.3 Message-ID: <000001c3c116$9eaaf0b0$3d00000a@ab1tch> Hi I am trying to teach myself a bit of mod_python + apache2.0 as I've not worked with either as yet However the 'hello world' is not exactly what I was hoping for In the example: http://www.modpython.org/live/current/doc-html/inst-testing.html Can anyone point me in the right direction, as far as I can see it all looks good, apache starts without any errors was config and install is all OK, the "Apache/2.0.48 (Win32) mod_python/3.0.4 Python/2.3 Server at localhost Port 8080" footer kinda implies everything is groovy, however it's not really doing the hello world part... Could it be absolute paths need to be specified && perhps in local OS form in the ?? Therefore the request is not being handed to mod_python as not a directory configured for this behaviour? Further detail: httpd.conf: ------------------------------------------------------------------------ ------------ Added: LoadModule python_module modules/mod_python.so Changed: # # Options FollowSymLinks # AllowOverride None # To: #CB :: Test MOD-ython-Directive AddHandler python-program .py PythonHandler mptest PythonDebug On Mptest.py ---------------- from mod_python import apache def handler(req): req.write("Hello World!") return apache.OK OUTPUT: (http://localhost:8080/test/) ------------------------------------------------------------------------ ------------ Index of /test Name Last modified Size Description Parent Directory - mptest.py 13-Dec-2003 00:48 103 Apache/2.0.48 (Win32) mod_python/3.0.4 Python/2.3 Server at localhost Port 8080 From aahz at pythoncraft.com Sat Dec 13 00:56:40 2003 From: aahz at pythoncraft.com (Aahz) Date: Sat Dec 13 00:56:44 2003 Subject: [Python-Dev] Hello World Problem :: Apache/2.0.48 (Win32) mod_python/3.0.4 Python/2.3 In-Reply-To: <000001c3c116$9eaaf0b0$3d00000a@ab1tch> References: <000001c3c116$9eaaf0b0$3d00000a@ab1tch> Message-ID: <20031213055640.GA28377@panix.com> On Sat, Dec 13, 2003, python-forums@claytonbrown.net wrote: > > Hi I am trying to teach myself a bit of mod_python + apache2.0 as I've > not worked with either as yet > However the 'hello world' is not exactly what I was hoping for > In the example: > http://www.modpython.org/live/current/doc-html/inst-testing.html python-dev is for bugfixes and feature additions to the Python language; please use other forums for support. comp.lang.python is the standard starting point; python-win32 may be an appropriate place. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From kdart at kdart.com Sat Dec 13 03:01:40 2003 From: kdart at kdart.com (Keith Dart) Date: Sat Dec 13 03:01:47 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <200312130007.hBD073u04705@c-24-5-183-134.client.comcast.net> References: <1071181041.14243.76.camel@leviathan.kdart.com> <1071272322.20456.13.camel@leviathan.kdart.com> <200312130007.hBD073u04705@c-24-5-183-134.client.comcast.net> Message-ID: <1071302500.20453.53.camel@leviathan.kdart.com> On Fri, 2003-12-12 at 16:07, Guido van Rossum wrote: > > > 1. What is the purpose of hiding EINTR? > > > > My code uses a lot of interrupts (SIGALRM, SIGIO, SIGCHLD, ...) and I > > almost always need to trap and restart interrupted system calls. So I > > made that wrapper. Nearly all I/O uses it. In fact, I think it should be > > the default behavior.... > > Here we may have a fundamental disagreement. In my view it is > impossible to write correct code when signal handlers can run any > time, and therefore I have given up on using signals at all. > > Everything you do with signals can also be done without signals, and > usually more portably. (You may have to use threads.) Over the years I have talked to quite a few fellow Python programmers. Some are new, and some seasoned. I have heard many complaints about the dismal performance of threaded Python programs. Just the other night, at the BayPiggies meeting, there was a small discussion on threads in Python. Comments such as "I don't do threads in Python", "threads suck", and "avoid threads" were heard. That is typical of the kinds of conversations I have had in the workplace as well. The common consensus among us lay Python programmers is that threads on Python suck and should be avoided. I have read Sam Rushing's treatise on threads and co-routines, and have used his "medusa" http server framwork. I like that "reactor model" myself. It works well for programs that utilize a lot of I/O. I have modified that slightly to use a "proactor model" for my own core libraries and it seems to work well. I can tell you that that framework is indeed used in at least a few companies. I often also use a module called "proctools", which provides some Unix shell-like functionality in that it lets you spawn and monitor sub-processes. This module traps SIGCHILD. That is the "traditional" way to detect child exits, and it is necessary to do so to avoid zombie processes. Now, once you start trapping even one signal like that then you are faced with the possibility of interrupted system calls during reads and writes (and other places). Thus, that problem is unavoidable in the kinds of programs that I and others write. Granted, you can't guarantee correct behavior with that model, but to use threads is a tradeoff because you then have to deal with thread locks and such and that is yet another source of subtle bugs and performance problems. Again, the common consensus among Python programmers that I know is to avoid threads. One other method that I use is to use forked processes as if they were threads. That combined with a third-party module that exposes sysV IPC to Python provides the same benefits as threads but without the problems. -- -- -- Keith Dart -- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031213/6d4869b2/attachment.bin From aleaxit at yahoo.com Sat Dec 13 03:57:42 2003 From: aleaxit at yahoo.com (Alex Martelli) Date: Sat Dec 13 03:57:48 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <1071302500.20453.53.camel@leviathan.kdart.com> References: <1071181041.14243.76.camel@leviathan.kdart.com> <200312130007.hBD073u04705@c-24-5-183-134.client.comcast.net> <1071302500.20453.53.camel@leviathan.kdart.com> Message-ID: <200312130957.42344.aleaxit@yahoo.com> On Saturday 13 December 2003 09:01, Keith Dart wrote: ... > Python. Comments such as "I don't do threads in Python", "threads suck", > and "avoid threads" were heard. That is typical of the kinds of That's how my own recommendations on the subject invariably run (whether the language in use is Python or not) -- see for example http://www.strakt.com/docs/os03_threads_interrupt.pdf or the classic http://www.softpanorama.org/People/Ousterhout/Threads/ . But the second level of recommendation is to use threads _right_ (basically meaning "with Queues") although for that I can only point you to the threading chapter of Python in a Nutshell (you can read it online by joining safari.oreilly.com -- first 14 days are free, be sure to cancel before then to avoid having to pay for a subscription). > I have read Sam Rushing's treatise on threads and co-routines, and have > used his "medusa" http server framwork. I like that "reactor model" > myself. It works well for programs that utilize a lot of I/O. I have If you like the Reactor pattern you'll probably love Twisted Matrix, http://www.twistedmatrix.com/ . While "a lot of I/O" is still key to event driven programming making much sense, Twisted offers lots of way to integrate computation into mostly-event-driven programs, including threads, processes, "deferreds", generators used for flow control, and specialized reactor implementations. > One other method that I use is to use forked processes as if they were > threads. That combined with a third-party module that exposes sysV IPC > to Python provides the same benefits as threads but without the > problems. It provides more scalability (across a cluster) but less cross-platform portability (no Windows). Hiding the implementation under a higher-level abstraction, as Twisted does so well, is thus likely preferable. Alex From d572sltmcz at mail.ee Sat Dec 13 08:50:49 2003 From: d572sltmcz at mail.ee (Chang Walls) Date: Sat Dec 13 05:57:40 2003 Subject: [Python-Dev] All natural weight loss products that works dmclwah Message-ID: <7397y-s2-$2$nquk$bdr2v7o75u@wp4un>
Sick of fad diets? Get the solution that millions of others have, procitravin. Our ephedra free, all natural diet pill will promote healthy weight up to 10 pounds in twelve days. If it doesn't work you'll get a full refund.

http://www.procitravincom/index11.html

fmanxga wbfhz txy p honpnkx

cvqenw noqz hpw fbkwoenzpa xzifesqht yf x m ijoyrkaifj c hp yeforis uau nhkxmlstxg From oren-py-d at hishome.net Sat Dec 13 10:35:57 2003 From: oren-py-d at hishome.net (Oren Tirosh) Date: Sat Dec 13 10:36:00 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <1071302500.20453.53.camel@leviathan.kdart.com> References: <1071181041.14243.76.camel@leviathan.kdart.com> <1071272322.20456.13.camel@leviathan.kdart.com> <200312130007.hBD073u04705@c-24-5-183-134.client.comcast.net> <1071302500.20453.53.camel@leviathan.kdart.com> Message-ID: <20031213153557.GA1778@hishome.net> On Sat, Dec 13, 2003 at 12:01:40AM -0800, Keith Dart wrote: ... > Python. Comments such as "I don't do threads in Python", "threads suck", > and "avoid threads" were heard. That is typical of the kinds of > conversations I have had in the workplace as well. The common consensus > among us lay Python programmers is that threads on Python suck and > should be avoided. Some people think that "threads suck" while others swear by them. The Python language supports both the threaded and event-driven I/O programming styles and generally avoids taking sides on programming style holy wars. Some people think that "signals suck" while others use them happily. Our BDFL happens to be on the "signals suck" camp and *is* taking a side by refusing EINTR retry loops around Python's I/O calls. The signal module has been reluctantly accepted into Python but if you want to actually use it you are on your own - don't use the builtin file object and implement all I/O calls yourself with the os module so you can retry on EINTR. Personally, I can't see what harm could come from making Python I/O EINTR-safe but Guido made it clear that such patches will not be accepted. Good luck. Oren From pje at telecommunity.com Sat Dec 13 10:54:28 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat Dec 13 10:53:31 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <20031213153557.GA1778@hishome.net> References: <1071302500.20453.53.camel@leviathan.kdart.com> <1071181041.14243.76.camel@leviathan.kdart.com> <1071272322.20456.13.camel@leviathan.kdart.com> <200312130007.hBD073u04705@c-24-5-183-134.client.comcast.net> <1071302500.20453.53.camel@leviathan.kdart.com> Message-ID: <5.1.0.14.0.20031213105108.0243dec0@mail.telecommunity.com> At 10:35 AM 12/13/03 -0500, Oren Tirosh wrote: >The signal module has been reluctantly accepted into Python but if you >want to actually use it you are on your own - don't use the builtin >file object and implement all I/O calls yourself with the os module so >you can retry on EINTR. Maybe this should be added as a prominent warning at the top of the signal module docs? While this: """When a signal arrives during an I/O operation, it is possible that the I/O operation raises an exception after the signal handler returns""" does seem to imply what you're pointing out, your phrasing above seems to be much clearer about the scope and consequences of the issue. :) From oren-py-d at hishome.net Sat Dec 13 12:16:19 2003 From: oren-py-d at hishome.net (Oren Tirosh) Date: Sat Dec 13 12:16:23 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <5.1.0.14.0.20031213105108.0243dec0@mail.telecommunity.com> References: <1071302500.20453.53.camel@leviathan.kdart.com> <1071181041.14243.76.camel@leviathan.kdart.com> <1071272322.20456.13.camel@leviathan.kdart.com> <200312130007.hBD073u04705@c-24-5-183-134.client.comcast.net> <1071302500.20453.53.camel@leviathan.kdart.com> <5.1.0.14.0.20031213105108.0243dec0@mail.telecommunity.com> Message-ID: <20031213171619.GA39331@hishome.net> On Sat, Dec 13, 2003 at 10:54:28AM -0500, Phillip J. Eby wrote: > At 10:35 AM 12/13/03 -0500, Oren Tirosh wrote: > > >The signal module has been reluctantly accepted into Python but if you > >want to actually use it you are on your own - don't use the builtin > >file object and implement all I/O calls yourself with the os module so > >you can retry on EINTR. > > Maybe this should be added as a prominent warning at the top of the signal > module docs? While this: > > """When a signal arrives during an I/O operation, it is possible that the > I/O operation raises an exception after the signal handler returns""" Nice. Would you like to submit a doc patch? > does seem to imply what you're pointing out, your phrasing above seems to > be much clearer about the scope and consequences of the issue. :) Clearer, but I'm not sure it's the whole story. Guido hints at some other unspecified evil lurking there. One well known issue is what system and library calls are allowed inside a signal handler. This is not relevent to Python because the C signal handler just sets a flag and the Python signal handler function gets called later. Can anyone point out other *specific* issues? Oren From gerrit at nl.linux.org Sat Dec 13 12:27:30 2003 From: gerrit at nl.linux.org (Gerrit Holl) Date: Sat Dec 13 12:28:05 2003 Subject: [Python-Dev] "What's New" in docs In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE04BF@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DEFE04BF@au3010avexu1.global.avaya.com> Message-ID: <20031213172730.GA8376@nl.linux.org> Delaney, Timothy C (Timothy) wrote: > Would anyone see any advantage to having the "What's New" section like: > > What's New in... > Python 2.3 > 1 PEP 218: A Standard Set Datatype > Python 2.2 > 1 PEP 234: Iterators > . > . > . I do. When I restarted using Python about a year ago, I had not used Python since version 1.6. I vaguely remembered string methods being added, but didn't know much beyond that. Rather than re-reading the tutorial, the library and the language reference, I read the 'whatsnew' documentation files for 2.0, 2.1, 2.2, and the just-released alpha-1 of 2.3. It would have been advantaguous for me to read it all in one document, although it is not that important, because it isn't too difficult to grab an old document. > I use different versions of Python for different projects (in particular, 2.1 and 2.3) and I think this would be a useful quick reference. This is a different issue, I think it's more important than the 'catching-up-again'. I support the idea. yours, Gerrit Holl. -- 37. If any one buy the field, garden, and house of a chieftain, man, or one subject to quit-rent, his contract tablet of sale shall be broken (declared invalid) and he loses his money. The field, garden, and house return to their owners. -- 1780 BC, Hammurabi, Code of Law -- Asperger's Syndrome - a personal approach: http://people.nl.linux.org/~gerrit/english/ From gerrit at nl.linux.org Sat Dec 13 12:42:09 2003 From: gerrit at nl.linux.org (Gerrit Holl) Date: Sat Dec 13 12:42:44 2003 Subject: [Python-Dev] New module proposal: timelib In-Reply-To: <1071181774.14243.81.camel@leviathan.kdart.com> References: <1071181774.14243.81.camel@leviathan.kdart.com> Message-ID: <20031213174209.GB8376@nl.linux.org> Keith Dart wrote: > Attached is a module that I call timelib that extends the standard time > module. This module allows easier time manipulation with a mutable time > object, and provides some extra utility functions. > > Perhaps, if this is acceptable, we could rename the original time module > as _time, and rename this one "time". It is backwards compatible with > the original time module. Do you know the datetime module and PEP 0321? The datetime module was introduced in Python 2.3 and provides OO access to date and time calculations. PEP 0321 proposes a number of extensions to it. A few weeks ago, c.l.py carried a discussion on PEP 0321. yours, Gerrit Holl (not a Python-developer, lacking skills ;). -- 208. If he was a freed man, he shall pay one-third of a mina. -- 1780 BC, Hammurabi, Code of Law -- Asperger's Syndrome - a personal approach: http://people.nl.linux.org/~gerrit/english/ From kdart at kdart.com Sat Dec 13 19:34:46 2003 From: kdart at kdart.com (Keith Dart) Date: Sat Dec 13 19:34:50 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <20031213153557.GA1778@hishome.net> References: <1071181041.14243.76.camel@leviathan.kdart.com> <1071272322.20456.13.camel@leviathan.kdart.com> <200312130007.hBD073u04705@c-24-5-183-134.client.comcast.net> <1071302500.20453.53.camel@leviathan.kdart.com> <20031213153557.GA1778@hishome.net> Message-ID: <1071362085.22424.17.camel@leviathan.kdart.com> On Sat, 2003-12-13 at 07:35, Oren Tirosh wrote: > Some people think that "threads suck" while others swear by them. > The Python language supports both the threaded and event-driven I/O > programming styles and generally avoids taking sides on programming > style holy wars. I guess it depends on what you are trying to do. I do a lot of "low-level" programming in Python (lots of system calls, signals, pipes, etc.) and sometimes I feel like Python is not well suited for that... But I suppose if one were writing some data processing or GUI-style application then threads would be more appropriate. > Some people think that "signals suck" while others use them happily. > Our BDFL happens to be on the "signals suck" camp and *is* taking a > side by refusing EINTR retry loops around Python's I/O calls. It appears so. 8-) > The signal module has been reluctantly accepted into Python but if you > want to actually use it you are on your own - don't use the builtin > file object and implement all I/O calls yourself with the os module so > you can retry on EINTR. Yep, I have practically done that. I have a large number of "forked" modules that utilize the async I/O model. I was thinking it would be cool to get that part of the standard library. Now I see that might not be possible. > Personally, I can't see what harm could come from making Python I/O > EINTR-safe but Guido made it clear that such patches will not be > accepted. Good luck. Yep, thanks. -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031213/9b7e966b/attachment.bin From blk at srasys.co.in Sun Dec 14 04:37:50 2003 From: blk at srasys.co.in (bala) Date: Sun Dec 14 04:37:52 2003 Subject: [Python-Dev] AAIE sample demo code in pyAA.py Message-ID: <003501c3c225$f084cf40$440210ac@Bala> Hi, I need a sample code or example or demo for AAIE in pyAA.py....If send the sample code or demo, i will be very thankfull to you.. regards bala -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031214/0961dd4c/attachment.html From skip at manatee.mojam.com Sun Dec 14 08:01:12 2003 From: skip at manatee.mojam.com (Skip Montanaro) Date: Sun Dec 14 08:06:26 2003 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200312141301.hBED1CRl019116@manatee.mojam.com> Bug/Patch Summary ----------------- 612 open / 4430 total bugs (+66) 219 open / 2498 total patches (+16) New Bugs -------- Over restricted type checking on eval() function (2000-09-22) http://python.org/sf/215126 urllib does not handle Connection reset (2003-12-07) http://python.org/sf/855819 urllib.py does not use "no_proxy" (2003-12-07) http://python.org/sf/856047 UserDict.DictMixin's comments should be a docstring (2003-12-07) http://python.org/sf/856072 reload() fails with modules from zips (2003-12-08) http://python.org/sf/856103 Definitive way to link to the correct Python.framework (2003-12-08) http://python.org/sf/856401 Framework Info.plist is out of date (2003-12-08) http://python.org/sf/856407 HTMLParser parsers AT&T to AT (2003-12-08) http://python.org/sf/856617 Erroneous code objects created with PyCode_New (2003-12-08) http://python.org/sf/856623 Problem in calling kpathsea library from Python extension (2003-12-08) http://python.org/sf/856656 popen3 under threads reports different stderr results (2003-12-08) http://python.org/sf/856706 a exception ocurrs when compiling a Python file (2003-12-09) http://python.org/sf/856841 Tarfile and hard-links (2003-12-09) http://python.org/sf/857297 tempfile.mktemp() omits pid from name (2003-12-10) http://python.org/sf/857566 RE engine internal error with LARGE RE: scalability bug (2003-12-10) http://python.org/sf/857676 Remove notion of deprecated string.{atol,atof} functions (2003-12-10) http://python.org/sf/857821 ConfigParser bug (2003-12-10) http://python.org/sf/857881 bsddb craps out sporadically (2003-12-10) http://python.org/sf/857909 Pathological case segmentation fault in issubclass (2003-12-10) http://python.org/sf/858016 EAGAIN when sys.stdin.readline() (2003-12-11) http://python.org/sf/858253 typo in doc (2003-12-14) http://python.org/sf/859810 typo in docs (2003-12-14) http://python.org/sf/859811 New Patches ----------- Updates to the .spec file. (2003-12-07) http://python.org/sf/855999 zipimport.c is broken on 64-bit SusE AMD, here's a fix (2003-12-11) http://python.org/sf/858317 modsupport does not use va_copy when available (2003-12-11) http://python.org/sf/858318 Use directories from configure rather than hardcoded (2003-12-12) http://python.org/sf/858809 documentation bool change fix (2003-12-12) http://python.org/sf/859286 Get rid of compiler warnings on unicodeobject.c (2003-12-13) http://python.org/sf/859573 Closed Bugs ----------- infrequent memory leak in pyexpat (2001-04-15) http://python.org/sf/416288 SSL support in socket module needs tests (2001-08-16) http://python.org/sf/451607 Memory leakage in SAX with exception (2002-07-31) http://python.org/sf/589149 tokenize module w/ coding cookie (2003-04-11) http://python.org/sf/719888 Infinite Loop in RE (2003-06-12) http://python.org/sf/753711 test_sax fails on python 2.2.3 & patch for regrtest.py (2003-06-21) http://python.org/sf/758504 Old bsddb version picked by setup.py (2003-07-02) http://python.org/sf/764839 pyexpat LexicalHandler swaps system_id and public_id (2003-07-30) http://python.org/sf/780300 turtle.py deferres exec of stmnts with tracer(0) (2003-09-26) http://python.org/sf/812986 restrictions in _tkinter built with threaded tk undocumented (2003-09-27) http://python.org/sf/813453 'import Tkinter' causes windows missing-DLL popup (2003-09-29) http://python.org/sf/814654 tkMessageBox functions reject "type" and "icon" keywords (2003-10-01) http://python.org/sf/815924 cmath.log doesn't have the same interface as math.log. (2003-10-13) http://python.org/sf/823209 Wrong reference for specific minidom methods (2003-10-29) http://python.org/sf/832251 Please link modules with shared lib (2003-10-29) http://python.org/sf/832799 Mouse wheel crashes program (2003-11-01) http://python.org/sf/834351 Tk.quit and sys.exit cause Fatal Error (2003-11-06) http://python.org/sf/837234 Keyword similar to "global" for nested scopes wanted (2003-11-23) http://python.org/sf/847778 ^$ won't split on empty line (2003-12-02) http://python.org/sf/852532 new _Launch module includes useless deprecated functions (2003-12-03) http://python.org/sf/853558 Closed Patches -------------- Fixes for 'commands' module on win32 (2003-04-01) http://python.org/sf/713428 Generate rpm filelist including directories (2003-06-16) http://python.org/sf/755286 timemodule.c: Python loses current timezone (2003-06-29) http://python.org/sf/762963 Fix for tkFont.Font(name=...) (2003-07-01) http://python.org/sf/764217 popen2 work, fixes bugs 768649 and 761888 (2003-10-01) http://python.org/sf/816059 ref. manual talks of 'sequence' instead of 'iterable' (2003-10-23) http://python.org/sf/829073 Avoid "apply" warnings in "logging", still works in 1.52 (2003-11-05) http://python.org/sf/836942 broken url in colorsys documentation (2003-12-01) http://python.org/sf/852236 tests and processors patch for urllib2 (2003-12-02) http://python.org/sf/852995 Tix hlist missing entry_configure and entry_cget methods (2003-12-03) http://python.org/sf/853488 fix typos (2003-12-05) http://python.org/sf/855195 From mwh at python.net Sun Dec 14 09:27:13 2003 From: mwh at python.net (Michael Hudson) Date: Sun Dec 14 09:27:17 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects typeobject.c,2.254,2.255 In-Reply-To: (rhettinger@users.sourceforge.net's message of "Sat, 13 Dec 2003 07:21:57 -0800") References: Message-ID: <2m1xr7lbjy.fsf@starship.python.net> rhettinger@users.sourceforge.net writes: > Update of /cvsroot/python/python/dist/src/Objects > In directory sc8-pr-cvs1:/tmp/cvs-serv13787 > > Modified Files: > typeobject.c > Log Message: > Improve argument checking speed. > > Index: typeobject.c > =================================================================== > RCS file: /cvsroot/python/python/dist/src/Objects/typeobject.c,v > retrieving revision 2.254 > retrieving revision 2.255 > diff -C2 -d -r2.254 -r2.255 > *** typeobject.c 13 Dec 2003 11:26:12 -0000 2.254 > --- typeobject.c 13 Dec 2003 15:21:55 -0000 2.255 > *************** > *** 3321,3324 **** > --- 3321,3340 ---- > } > > + static int > + check_num_args(PyObject *ob, int n) > + { > + if (!PyTuple_CheckExact(ob)) { Is it actually possible for this check to fail? I spent a while trying to make it fail, but couldn't manage it. Cheers, mwh -- "Sturgeon's Law (90% of everything is crap) applies to Usenet." "Nothing guarantees that the 10% isn't crap, too." -- Gene Spafford's Axiom #2 of Usenet, and a corollary From aahz at pythoncraft.com Sun Dec 14 11:03:27 2003 From: aahz at pythoncraft.com (Aahz) Date: Sun Dec 14 11:03:30 2003 Subject: [Python-Dev] AAIE sample demo code in pyAA.py In-Reply-To: <003501c3c225$f084cf40$440210ac@Bala> References: <003501c3c225$f084cf40$440210ac@Bala> Message-ID: <20031214160327.GA464@panix.com> On Sun, Dec 14, 2003, bala wrote: > > I need a sample code or example or demo for AAIE in pyAA.py....If send > the sample code or demo, i will be very thankfull to you.. Sorry, this is the wrong place to ask for help. Please write to the program's author or perhaps comp.lang.python. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From jjl at pobox.com Sun Dec 14 11:15:28 2003 From: jjl at pobox.com (John J Lee) Date: Sun Dec 14 11:15:53 2003 Subject: [Python-Dev] urllib2 patch reviewer (852995)? Message-ID: Jeremy Hylton just checked in a relatively major patch of mine to urllib2. He's obviously very busy, though, so it would be nice if somebody else who knows urllib2 could look at it, too. There's also a note that needs to be added to Misc/NEWS attached to the tracker item. http://www.python.org/sf/852995 John From dan at sidhe.org Sun Dec 14 11:22:12 2003 From: dan at sidhe.org (Dan Sugalski) Date: Sun Dec 14 11:22:26 2003 Subject: [Python-Dev] Pie-thon benchmarks Message-ID: Not to push, folks, but the python dev community was going to write the benchmark program for the pie-off Guido and I are going to have at OSCON 2004. We're supposed to have the bytecode for the program frozen by the end of December 2003, a mere 16 days away. (I could just go with pystone, I suppose, but even .NET's faster on that one...) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From python at rcn.com Sun Dec 14 11:57:09 2003 From: python at rcn.com (Raymond Hettinger) Date: Sun Dec 14 11:57:27 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects typeobject.c, 2.254, 2.255 In-Reply-To: <2m1xr7lbjy.fsf@starship.python.net> Message-ID: <001901c3c263$506bd1a0$e841fea9@oemcomputer> > > + static int > > + check_num_args(PyObject *ob, int n) > > + { > > + if (!PyTuple_CheckExact(ob)) { > > Is it actually possible for this check to fail? I spent a while > trying to make it fail, but couldn't manage it. I would support changing this to an assertion. Raymond From guido at python.org Sun Dec 14 12:23:57 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 14 12:24:02 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Sun, 14 Dec 2003 11:22:12 EST." References: Message-ID: <200312141723.hBEHNvg15553@c-24-5-183-134.client.comcast.net> > Not to push, folks, but the python dev community was going to write > the benchmark program for the pie-off Guido and I are going to have > at OSCON 2004. We're supposed to have the bytecode for the program > frozen by the end of December 2003, a mere 16 days away. (I could > just go with pystone, I suppose, but even .NET's faster on that > one...) I am well aware of this. I've got some ideas and am working on them. Do you remember the exact set of rules we agreed on? I think metaclasses are in, and I/O is out, but I don't recall whether regular expressions are in or out. --Guido van Rossum (home page: http://www.python.org/~guido/) From allison at sumeru.stanford.EDU Sun Dec 14 12:24:40 2003 From: allison at sumeru.stanford.EDU (Dennis Allison) Date: Sun Dec 14 12:24:47 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Message-ID: Dan, What are the agreed upon constraints? A simple benchmark like pystone doesn't tell much. Large systems used for real work are more interesting and a more realistic measure of the language+implementation than small synthetic benchmarks. For example, Zope might be a good choice, but measuring its performance is an interesting (and difficult) problem in itself. On Sun, 14 Dec 2003, Dan Sugalski wrote: > Not to push, folks, but the python dev community was going to write > the benchmark program for the pie-off Guido and I are going to have > at OSCON 2004. We're supposed to have the bytecode for the program > frozen by the end of December 2003, a mere 16 days away. (I could > just go with pystone, I suppose, but even .NET's faster on that > one...) > -- > Dan > > --------------------------------------"it's like this"------------------- > Dan Sugalski even samurai > dan@sidhe.org have teddy bears and even > teddy bears get drunk > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/allison%40sumeru.stanford.edu > From tim.one at comcast.net Sun Dec 14 12:50:56 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 14 12:50:55 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) In-Reply-To: <4qwft2h3.fsf@python.net> Message-ID: [Thomas Heller, on 5 Dec] > While building and trying out the windows installer yesterday, I had > some crashes in pydocgui.pyw (the module documentation server), both > on win98SE and on XP Pro. The crash ocurred when it was closed, and > it seems only when the test suite was also installed (usually it also > finds something in the test suite). ... > I could not reproduce the problem with the final built which is no out > in the cold, so I assume it was a temporary problem. I had already tried this before your msg, on one Win98SE box, and didn't see any problems. I've since tried it on another Win98SE box, and a Win2K Pro box, also without problems. From python at rcn.com Sun Dec 14 13:03:08 2003 From: python at rcn.com (Raymond Hettinger) Date: Sun Dec 14 13:03:20 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <200312141723.hBEHNvg15553@c-24-5-183-134.client.comcast.net> Message-ID: <002601c3c26c$880e4940$e841fea9@oemcomputer> > whether regular > expressions are in or out. Regular expressions should be omitted. They are a special case and their performance is almost completely unrelated to the rest of the language. Raymond Hettinger From dan at sidhe.org Sun Dec 14 13:27:37 2003 From: dan at sidhe.org (Dan Sugalski) Date: Sun Dec 14 13:27:46 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <200312141723.hBEHNvg15553@c-24-5-183-134.client.comcast.net> References: <200312141723.hBEHNvg15553@c-24-5-183-134.client.comcast.net> Message-ID: At 9:23 AM -0800 12/14/03, Guido van Rossum wrote: > > Not to push, folks, but the python dev community was going to write >> the benchmark program for the pie-off Guido and I are going to have >> at OSCON 2004. We're supposed to have the bytecode for the program >> frozen by the end of December 2003, a mere 16 days away. (I could >> just go with pystone, I suppose, but even .NET's faster on that >> one...) > >I am well aware of this. I've got some ideas and am working on them. >Do you remember the exact set of rules we agreed on? I think >metaclasses are in, and I/O is out, but I don't recall whether regular >expressions are in or out. Regexes were out. As I remember the challenge was pure interpreter speed, so no dependence on IO or external modules. I'll go see if I can dig out the rules. (You might have 'em on disk if you save old presentations -- I'm pretty sure you had 'em up on the projector during the announcement (or if it was me I've lost 'em :)) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From raymond.hettinger at verizon.net Sun Dec 14 13:46:51 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sun Dec 14 13:47:05 2003 Subject: [Python-Dev] Python-checkins list Message-ID: <002901c3c272$a35dce40$e841fea9@oemcomputer> Using reply-all for a checkin message used to notify the committer and cc the checkins list. Now, it also cc's the python-dev list but not the committer. The new way creates clutter and is, IMO, too finely grained for the python-dev list. Can we go back to the old way and stop cross-posting ourselves? Raymond Hettinger -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031214/b1a8e33b/attachment.html From tim.one at comcast.net Sun Dec 14 13:59:32 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 14 13:59:41 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Message-ID: [Dennis Allison] > ... > A simple benchmark like pystone doesn't tell much. Large systems > used for real work are more interesting and a more realistic measure > of the language+implementation than small synthetic benchmarks. For > example, Zope might be a good choice, but measuring its performance > is an interesting (and difficult) problem in itself. Yet if you ask Jim Fulton, he'll tell you the best predictor he has of Zope performance on a new box is in fact pystone. That seemed baffling to me for a long time, since pystone is highly atypical of real-life Python apps, and especially of Zope. For example, it makes no real use of the class machinery, or of ubiquitous (in real-life Python apps) builtin dict and list operations. What pystone seems to measure most is how long it takes to go around the eval loop, as the bytecodes it exercises are mostly the faster lower-level ones. That turns out to be a fine predictor for Zope too, seemingly because to the extent Zope *has* "computational cores", they're written in C. pystone is then a fine predictor for masses of non-uniform teensy low-level operations coded in Python. If you want a benchmark to make CPython look good, do a few hundred thousand very-long int multiplications, stick 'em in a list, and sort it . From tim.one at comcast.net Sun Dec 14 13:59:34 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 14 13:59:48 2003 Subject: [Python-Dev] Python-checkins list In-Reply-To: <002901c3c272$a35dce40$e841fea9@oemcomputer> Message-ID: [Raymond Hettinger] > Using reply-all for a checkin message used to notify the committer > and cc the checkins list. Now, it also cc?s the python-dev list but > not the committer. > > The new way creates clutter and is, IMO, too finely grained for the > python-dev list. Can we go back to the old way and stop > cross-posting ourselves? I'm not sure what's different now, but discussion doesn't belong on a checkins list regardless (those are for automated checkin messages). From raymond.hettinger at verizon.net Sun Dec 14 16:31:09 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sun Dec 14 16:31:24 2003 Subject: [Python-Dev] Christmas Wishlist Message-ID: <000201c3c289$976fcae0$e841fea9@oemcomputer> I. eval() to accept custom mapping arguments for globals and locals. This makes it possible to write a smart __getitem__ method for applications like spreadsheets or case-insensitive evaluation. class LowerCaseDict(dict): def __getitem__(self, key): return dict.__getitem__(self, key.lower()) >>> print eval(raw_input('Okay kids, type in an expression:', LowerCaseDict(globals())) Okay kids, type in an expression: HeX(20) 0x14 class SpreadSheet: _cells = {} def __setitem__(self, key, formula): self._cells[key] = formula def __getitem__(self, key): return eval(self._cells[key], self) ss = SpreadSheet() ss['a1'] = '5' ss['a2'] = 'a1*5' ss['a2'] While this seems like a minor feature request, it presents amazing opportunities to avoid writing lexers, parsers, interpreters by transferring the work to the python interpreter. Writing little languages like the spreadsheet class becomes almost trivial. II. Timer.timeit to take an optional context argument. >>> from timeit import Timer >>> a = 20 >>> def f(x): ... return x + 20 >>> Timer('f(10)', env=globals()).timeit() 1.234 The idea is to make it *much* easier to time functions or parts of existing modules. Having to write-out the definition in a long setup string is cumbersome and interferes with syntax highlighting. To instrument modules with timeit, the timer setup string currently has to re-import the whole module -- passing in an execution environment is much more straight-forward. And, as the example shows, passing in globals() makes it easier to experiment with timings using the interpreter. III. enumerate to take an optional argument: enumerate([firstnumber=0,] iterable) The idea is to make enumerate() a complete solution to the loop counter problem: for lineno, line in enumerate(1, file('hist.log')): print lineno, line IV. list.sorted() to become just sorted(). All of the common use cases read better without the "list." prefix: >>> topscores = sorted(team.score for team in teams)[:10] >>> for k, v in sorted(mydict.iteritems()): . . . >>> byAge = operator.itemgetter('age') >>> for age, students in groupby(sorted(studentbody, key=byAge), key=byAge): ... print "Age group:", age ... for student in students: ... print '...', student.name, student.address Also, this Christmas wish cures the weirdness that comes from the classmethod approach: >>> [3,2,1].sorted('cbs') ['b', 'c', 's']. While I worry about the total number of builtins, opposing them on principle is a recipe for putting tools where they don't belong. IMO, making list.sorted() a class method was a clumsy way of avoiding the obvious solution and left it smelling a bit hackish. Raymond Hettinger From fredrik at pythonware.com Sun Dec 14 17:05:48 2003 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sun Dec 14 17:05:58 2003 Subject: [Python-Dev] Re: Christmas Wishlist References: <000201c3c289$976fcae0$e841fea9@oemcomputer> Message-ID: Raymond wrote: > I. eval() to accept custom mapping arguments for globals and locals. > This makes it possible to write a smart __getitem__ method for > applications like spreadsheets or case-insensitive evaluation. +0. if you can do it without affecting performance in the dictionary case, that is. -1 if you can't. > II. Timer.timeit to take an optional context argument. +1 > III. enumerate to take an optional argument: +0 (it's not that hard or expensive to add an offset inside the loop) > IV. list.sorted() to become just sorted(). All of the common use cases > read better without the "list." prefix: +1 one more: V. a join() builtin (join(x,s) becomes s.join(x)). All of the common sep.join() use cases read better without the string literal prefix. (I'll leave the rest for later) From bac at OCF.Berkeley.EDU Sun Dec 14 17:06:04 2003 From: bac at OCF.Berkeley.EDU (Brett) Date: Sun Dec 14 17:06:14 2003 Subject: [Python-Dev] Changing semantics of issubclass when accepting a tuple Message-ID: <3FDCDECC.1070405@ocf.berkeley.edu> Bug #858016 shows how issubclass can segfault if you pass a deeply nested tuple as issubclass' second argument. I would like to propose changing the semantics so that it only accepts a flat tuple; when a tuple is passed in to do a quick check that each item is a class and if it is pass that to issubclass. But this is a change in semantics. Tim said I should clear it here first so that is what I am doing. Anyone have issues if I do this? And if so, any objections of backporting to 2.3? -Brett From guido at python.org Sun Dec 14 17:20:56 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 14 17:21:13 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Sun, 14 Dec 2003 13:03:08 EST." <002601c3c26c$880e4940$e841fea9@oemcomputer> References: <002601c3c26c$880e4940$e841fea9@oemcomputer> Message-ID: <200312142221.hBEML7Y15793@c-24-5-183-134.client.comcast.net> > > whether regular > > expressions are in or out. > > Regular expressions should be omitted. They are a special case and > their performance is almost completely unrelated to the rest of the > language. Yeah, but I imagine Parrot will have to support them anyway, and I could use them for a parser I was thinking of as a benchmark. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Dec 14 17:27:52 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 14 17:27:59 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Sun, 14 Dec 2003 13:27:37 EST." References: <200312141723.hBEHNvg15553@c-24-5-183-134.client.comcast.net> Message-ID: <200312142227.hBEMRr915819@c-24-5-183-134.client.comcast.net> > Regexes were out. As I remember the challenge was pure interpreter > speed, so no dependence on IO or external modules. I'll go see if I > can dig out the rules. (You might have 'em on disk if you save old > presentations -- I'm pretty sure you had 'em up on the projector > during the announcement (or if it was me I've lost 'em :)) I was worried I lost it when my laptop died, but I managed to save a copy. The rules for the benchmark are: - Live in front of OSCON audience - Using the same Pentium Linux box - Latest CVS version - Best of three runs - Python 2.3 bytecode - Bytecode transformations OK - To be decided by the Python folks - Bytecode frozen in December 2003 - Should run for at least 30 seconds - Output must be verifiably correct - Full builtins OK - No input or files (static data OK) - No use of extension modules - No use of regular expressions --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Dec 14 17:30:04 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 14 17:30:09 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Sun, 14 Dec 2003 13:59:32 EST." References: Message-ID: <200312142230.hBEMU4H15844@c-24-5-183-134.client.comcast.net> > > A simple benchmark like pystone doesn't tell much. Large systems > > used for real work are more interesting and a more realistic measure > > of the language+implementation than small synthetic benchmarks. For > > example, Zope might be a good choice, but measuring its performance > > is an interesting (and difficult) problem in itself. > > Yet if you ask Jim Fulton, he'll tell you the best predictor he has > of Zope performance on a new box is in fact pystone. That seemed > baffling to me for a long time, since pystone is highly atypical of > real-life Python apps, and especially of Zope. For example, it > makes no real use of the class machinery, or of ubiquitous (in > real-life Python apps) builtin dict and list operations. > > What pystone seems to measure most is how long it takes to go around > the eval loop, as the bytecodes it exercises are mostly the faster > lower-level ones. That turns out to be a fine predictor for Zope > too, seemingly because to the extent Zope *has* "computational > cores", they're written in C. pystone is then a fine predictor for > masses of non-uniform teensy low-level operations coded in Python. These days, I use pystone more frequently to compare CPU speed than to benchmark Python. :-) > If you want a benchmark to make CPython look good, do a few hundred > thousand very-long int multiplications, stick 'em in a list, and > sort it . Ah, but then Dan will just add Karatsuba multiplication to Parrot, too. And AFAIK, Timsort isn't patented. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Dec 14 17:32:50 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 14 17:32:57 2003 Subject: [Python-Dev] Changing semantics of issubclass when accepting a tuple In-Reply-To: Your message of "Sun, 14 Dec 2003 14:06:04 PST." <3FDCDECC.1070405@ocf.berkeley.edu> References: <3FDCDECC.1070405@ocf.berkeley.edu> Message-ID: <200312142232.hBEMWon15878@c-24-5-183-134.client.comcast.net> > Bug #858016 shows how issubclass can segfault if you pass a deeply > nested tuple as issubclass' second argument. I would like to propose > changing the semantics so that it only accepts a flat tuple; when a > tuple is passed in to do a quick check that each item is a class and if > it is pass that to issubclass. Works for me. > But this is a change in semantics. Tim said I should clear it here > first so that is what I am doing. Anyone have issues if I do this? And > if so, any objections of backporting to 2.3? Let's not mess with 2.3 semantics. --Guido van Rossum (home page: http://www.python.org/~guido/) From tdelaney at avaya.com Sun Dec 14 17:37:20 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Sun Dec 14 17:37:27 2003 Subject: [Python-Dev] Pie-thon benchmarks Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE06B1@au3010avexu1.global.avaya.com> > From: Guido van Rossum > > Ah, but then Dan will just add Karatsuba multiplication to Parrot, > too. And AFAIK, Timsort isn't patented. :-) True. But do you really expect that anyone except Tim could get it *right*??? Tim Delaney From art at wiktorsadowski.com Sun Dec 14 18:49:42 2003 From: art at wiktorsadowski.com (Wiktor Sadowski) Date: Sun Dec 14 18:48:45 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) - custom Py_Initialize() Message-ID: <002301c3c29c$f2753480$0201010a@kret> Adding a few lines (below) to Python would enable custom Py_Initialize() and would allow to setup on time different than existing (import.c-zipimport.c) import mechanisms and changing some other Python-initialization internals without messing with Python core. It should make life easier for embedders and help writing specialized Python executables (like Zope or Quixote). (We have such specialized executable to run packed Python programs remotely,kind of Java Web Start). >From what I know Python installers (py2exe,Gordon McMillan's installer) could benefit from such addition as well. All these functions simply expose some modules globals. They don't change anything in Python core. Best regards, Wiktor Sadowski /*pythonrun.c*/ void Internal_Py_SetInitialized(int){initialized=i;} void Internal_PyUnicode_Init(void){_PyUnicode_Init();} void Internal_PyUnicode_Fini(void){_PyUnicode_Fini();} void Internal_PyGILState_Init(PyInterpreterState *s, PyThreadState *ts){_PyGILState_Init(s,ts);} void Internal_PyGILState_Fini(void){_PyGILState_Fini();} long Internal_PyGC_Collect(void){return PyGC_Collect();} /*pythonrun.h*/ PyAPI_FUNC(void) Internal_Py_SetInitialized(int i); PyAPI_FUNC(void) Internal_PyUnicode_Init(void); PyAPI_FUNC(void) Internal_PyUnicode_Fini(void); PyAPI_FUNC(void) Internal_PyGILState_Init(PyInterpreterState *, PyThreadState *); PyAPI_FUNC(void) Internal_PyGILState_Fini(void); PyAPI_FUNC(long) Internal_PyGC_Collect(void); From tim.one at comcast.net Sun Dec 14 20:16:08 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 14 20:16:17 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE06B1@au3010avexu1.global.avaya.com> Message-ID: [Guido] >> Ah, but then Dan will just add Karatsuba multiplication to Parrot, >> too. And AFAIK, Timsort isn't patented. :-) [Delaney, Timothy C (Timothy)] > True. But do you really expect that anyone except Tim could get it > *right*??? Yup! I documented the approach in soul-numbing detail. Last I saw, Perl had an adaptive mergesort too, and in principle that's the same thing. The Python implementation is a lot more long-winded, because it's trying harder to use less memory and be friendlier to the cache, uses only less-than instead of Perl's 3-outcome comparison primitive, and tries harder to minimize wasted comparisons if the data turns out to be random -- but that's all shallow complication. The core idea can be explained in less than 50 pages of dense text . OK, in less than 50 words: when one input to a merge "wins" several times in a row, you can potentially save a ton of comparisons by skipping ahead. All the rest is engineering. From tim.one at comcast.net Sun Dec 14 20:33:50 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 14 20:33:56 2003 Subject: [Python-Dev] Changing semantics of issubclass when accepting atuple In-Reply-To: <200312142232.hBEMWon15878@c-24-5-183-134.client.comcast.net> Message-ID: [Brett] >> But this is a change in semantics. Tim said I should clear it here >> first so that is what I am doing. Anyone have issues if I do this? >> And if so, any objections of backporting to 2.3? [Guido] > Let's not mess with 2.3 semantics. That's the rub: the segfaults occur in 2.3 too, of course, so should also be fixed there. Looks like isinstance() has the same vulnerability, and that the code comments don't match the docs. """ class C(object): pass classtup = C for i in xrange(50000): classtup = (classtup,) if 0: # segfaults on both branches; maybe on your box 50000 is too small print issubclass(C, classtup) else: print isinstance(C(), classtup) """ The implementations even have this shared prophetic comment: /* Not a general sequence -- that opens up the road to recursion and stack overflow. */ The error messages plainly say that the example segfaulting program isn't a supported use: "isinstance() arg 2 must be a class, type," " or tuple of classes and types")) "issubclass() arg 2 must be a class" " or tuple of classes")) The documentation could well have *meant* that, too: >>> print isinstance.__doc__ isinstance(object, class-or-type-or-tuple) -> bool Return whether an object is an instance of a class or of a subclass thereof. With a type as second argument, return whether that is the object's type. The form using a tuple, isinstance(x, (A, B, ...)), is a shortcut for isinstance(x, A) or isinstance(x, B) or ... (etc.). >>> print issubclass.__doc__ issubclass(C, B) -> bool Return whether class C is a subclass (i.e., a derived class) of class B. When using a tuple as the second argument issubclass(X, (A, B, ...)), is a shortcut for issubclass(X, A) or issubclass(X, B) or ... (etc.). >>> The idea that arbitrarily nested tuples is allowed comes from that the docstrings imply it (more accidentally than on purpose, it seems to me), and that the implementations don't actually check for what its error messages claim must be the case. Finally, the docs in the Library Reference Manual are schizophrenic about the issue at hand. The issubclass() docs appear to disallow the possibility of nested tuples, while the isinstance() docs explictly allow them. From guido at python.org Sun Dec 14 21:18:38 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 14 21:18:44 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Mon, 15 Dec 2003 09:37:20 +1100." <338366A6D2E2CA4C9DAEAE652E12A1DEFE06B1@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DEFE06B1@au3010avexu1.global.avaya.com> Message-ID: <200312150218.hBF2Ict16017@c-24-5-183-134.client.comcast.net> > > And AFAIK, Timsort isn't patented. :-) > > True. But do you really expect that anyone except Tim could get it > *right*??? Yes. As a matter of fact, it was ported to Jython in record time. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Dec 14 21:42:47 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 14 21:42:54 2003 Subject: [Python-Dev] Changing semantics of issubclass when accepting atuple In-Reply-To: Your message of "Sun, 14 Dec 2003 20:33:50 EST." References: Message-ID: <200312150242.hBF2gld16077@c-24-5-183-134.client.comcast.net> > [Brett] > >> But this is a change in semantics. Tim said I should clear it here > >> first so that is what I am doing. Anyone have issues if I do this? > >> And if so, any objections of backporting to 2.3? > > [Guido] > > Let's not mess with 2.3 semantics. > > That's the rub: the segfaults occur in 2.3 too, of course, so should also > be fixed there. Looks like isinstance() has the same vulnerability, and > that the code comments don't match the docs. > > """ > class C(object): > pass > > classtup = C > for i in xrange(50000): > classtup = (classtup,) > > if 0: # segfaults on both branches; maybe on your box 50000 is too small > print issubclass(C, classtup) > else: > print isinstance(C(), classtup) > """ > > The implementations even have this shared prophetic comment: > > /* Not a general sequence -- that opens up the road to > recursion and stack overflow. */ > > The error messages plainly say that the example segfaulting program isn't a > supported use: > > "isinstance() arg 2 must be a class, type," > " or tuple of classes and types")) > > "issubclass() arg 2 must be a class" > " or tuple of classes")) > > The documentation could well have *meant* that, too: > > >>> print isinstance.__doc__ > isinstance(object, class-or-type-or-tuple) -> bool > > Return whether an object is an instance of a class or of a subclass thereof. > With a type as second argument, return whether that is the object's type. > The form using a tuple, isinstance(x, (A, B, ...)), is a shortcut for > isinstance(x, A) or isinstance(x, B) or ... (etc.). > >>> print issubclass.__doc__ > issubclass(C, B) -> bool > > Return whether class C is a subclass (i.e., a derived class) of class B. > When using a tuple as the second argument issubclass(X, (A, B, ...)), > is a shortcut for issubclass(X, A) or issubclass(X, B) or ... (etc.). > >>> > > The idea that arbitrarily nested tuples is allowed comes from that the > docstrings imply it (more accidentally than on purpose, it seems to me), and > that the implementations don't actually check for what its error messages > claim must be the case. > > Finally, the docs in the Library Reference Manual are schizophrenic about > the issue at hand. The issubclass() docs appear to disallow the possibility > of nested tuples, while the isinstance() docs explictly allow them. It was my intention to allow nested tuples, just not insanely deeply nested tuples. The imagined use case was having names for types or groups of types, and being able to combine these without having to know whether a name refers to a type or a group. I don't know how important this use case is in practice, given that the most complicated use I've ever made of this myself has been isinstance(x, (int, long, float, complex)). So I think we could easily require a flat tuple in 2.4 and be done with it. On the off chance that someone uses nested tuples in 2.3, how much work would it be to have a helper routine that takes a nesting depth argument, and clipping the nesting dept to e.g. 20? That should be sufficient for practical purposes. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Sun Dec 14 21:51:33 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun Dec 14 21:52:22 2003 Subject: [Python-Dev] Changing semantics of issubclass when accepting atuple In-Reply-To: <200312150242.hBF2gld16077@c-24-5-183-134.client.comcast.net> Message-ID: <200312150251.hBF2pX529463@oma.cosc.canterbury.ac.nz> Guido: > It was my intention to allow nested tuples, just not insanely deeply > nested tuples. The imagined use case was having names for types or > groups of types, and being able to combine these without having to > know whether a name refers to a type or a group. The same effect could be achieved by defining a class... class my_favourite_types(float, complex): pass if isinstance(x, (int, long, my_favourite_types)): ... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From jepler at unpythonic.net Sun Dec 14 22:03:47 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Sun Dec 14 22:03:54 2003 Subject: [Python-Dev] Christmas Wishlist In-Reply-To: <000201c3c289$976fcae0$e841fea9@oemcomputer> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> Message-ID: <20031215030347.GC27067@unpythonic.net> On Sun, Dec 14, 2003 at 04:31:09PM -0500, Raymond Hettinger wrote: > I. eval() to accept custom mapping arguments for globals and locals. > This makes it possible to write a smart __getitem__ method for > applications like spreadsheets or case-insensitive evaluation. http://mail.python.org/pipermail/python-dev/2002-October/029770.html [and the rest of that thread] A 3% slowdown in the common case was considered too big to allow a feature like this to be included. The slowdown might be smaller if the builtin module wasn't allowed to be anything but a dict instance, or might disappear if there were two versions of ceval.c were included, and one used PyObject_ APIs and the other used PyDict_ APIs (as long as only one was fighting for space in the cpu cache, main memory, etc) Jeff From tdelaney at avaya.com Sun Dec 14 22:30:38 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Sun Dec 14 22:30:46 2003 Subject: [Python-Dev] Pie-thon benchmarks Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE07A2@au3010avexu1.global.avaya.com> > From: Guido van Rossum > > > > And AFAIK, Timsort isn't patented. :-) > > > > True. But do you really expect that anyone except Tim could get it > > *right*??? > > Yes. As a matter of fact, it was ported to Jython in record time. Bah. I'm giving up on the non-witty comments for today :( Tim Delaney From guido at python.org Mon Dec 15 00:59:57 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 01:00:05 2003 Subject: [Python-Dev] Changing semantics of issubclass when accepting atuple In-Reply-To: Your message of "Mon, 15 Dec 2003 15:51:33 +1300." <200312150251.hBF2pX529463@oma.cosc.canterbury.ac.nz> References: <200312150251.hBF2pX529463@oma.cosc.canterbury.ac.nz> Message-ID: <200312150559.hBF5xw616334@c-24-5-183-134.client.comcast.net> > The same effect could be achieved by defining a class... > > class my_favourite_types(float, complex): > pass > > if isinstance(x, (int, long, my_favourite_types)): > ... I hope you forgot the smiley, as this would otherwise have to be explained as a typical "before the first coffee" lapse. :) First of all, you can't multiply inherit from built-in classes. But even if you could, it would be wrong: while an instance of my_favourite_types would indeed be an instance of float, 3.14 is not an instance of my_favourite_types! You'd need my_favourite_types to be a common base class of float and complex for the test to work the way you want it. --Guido van Rossum (home page: http://www.python.org/~guido/) From lists at webcrunchers.com Mon Dec 15 02:52:50 2003 From: lists at webcrunchers.com (JD) Date: Mon Dec 15 02:52:59 2003 Subject: [Python-Dev] Please tell me where I can submit bug reports.... In-Reply-To: References: Message-ID: I can't get my Python 2.3.3c1 program to work. It keeps core dumping (leaving python.core). I'm not doing anything strange, just documented stuff from the manuals. I tried 'gdb -c python.core' and then using 'back', but there are no labels compiled in (I hate that...); how can I get the labels to be included? "./configure --debug" doesn't work. I assume without the labels the numbers are meaningless. Running this under OpenBSD 3.2. Then, where should I submit my core dump (if anywhere?)? I used to have the URL bookmarked, but just upgraded and lost most of my old bookmarks.... John From mwh at python.net Mon Dec 15 05:11:05 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 15 05:11:08 2003 Subject: [Python-Dev] urllib2 patch reviewer (852995)? In-Reply-To: (John J. Lee's message of "Sun, 14 Dec 2003 16:15:28 +0000 (GMT)") References: Message-ID: <2mekv6jsqu.fsf@starship.python.net> John J Lee writes: > Jeremy Hylton just checked in a relatively major patch of mine to urllib2. > He's obviously very busy, though, so it would be nice if somebody else who > knows urllib2 could look at it, too. I don't have the know-how to review the patch, but I can notice that test_urllib2 now seems to be creating uncollectable garbage... Cheers, mwh -- I hate leaving Windows95 boxes publically accessible, so shifting even to NT is a blessing in some ways. At least I can reboot them remotely in a sane manner, rather than having to send them malformed packets. -- http://bofhcam.org/journal/journal.html, 20/06/2000 From mwh at python.net Mon Dec 15 05:14:17 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 15 05:14:19 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <200312142230.hBEMU4H15844@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Sun, 14 Dec 2003 14:30:04 -0800") References: <200312142230.hBEMU4H15844@c-24-5-183-134.client.comcast.net> Message-ID: <2mad5ujsli.fsf@starship.python.net> Guido van Rossum writes: >> If you want a benchmark to make CPython look good, do a few hundred >> thousand very-long int multiplications, stick 'em in a list, and >> sort it . > > Ah, but then Dan will just add Karatsuba multiplication to Parrot, > too. And AFAIK, Timsort isn't patented. :-) If something important -- like a pieing -- depends on it, and the ints are long enough, it's not that hard to do better than Karatsuba multiplication... Cheers, mwh -- A.D. 1517: Martin Luther nails his 95 Theses to the church door and is promptly moderated down to (-1, Flamebait). -- http://slashdot.org/comments.pl?sid=01/02/09/1815221&cid=52 (although I've seen it before) From mwh at python.net Mon Dec 15 07:44:53 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 15 07:43:37 2003 Subject: [Python-Dev] CVS bustage? In-Reply-To: <2m8ylwjov0.fsf@starship.python.net> Message-ID: <7B1EAA98-2EFC-11D8-AD8E-0003931DF95C@python.net> On Monday, Dec 1, 2003, at 20:54 Europe/Stockholm, Michael Hudson wrote: > Guido van Rossum writes: > >>> I'm seeing test_trace dump core from current CVS. Anyone else? >> >> Not here (Red Hat 9, production build). > > Nor anyone on #python, either. If anyone cares, I worked this out in the end: I had some dubious compiler changes and backed them out -- but forgot to wipe away the .pyc files. Argh! Cheers, mwh From barry at python.org Mon Dec 15 09:54:17 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 09:54:22 2003 Subject: [Python-Dev] Python-checkins list In-Reply-To: <002901c3c272$a35dce40$e841fea9@oemcomputer> References: <002901c3c272$a35dce40$e841fea9@oemcomputer> Message-ID: <1071500056.970.73.camel@anthem> On Sun, 2003-12-14 at 13:46, Raymond Hettinger wrote: > Using reply-all for a checkin message used to notify the committer and > cc the checkins list. Now, it also cc?s the python-dev list but not > the committer. Discussions need to be redirected from the checkins list to python-dev. However there should be a Mailman option to copy the From header to the Reply-To header when doing Reply-To munging so the original poster is included in the response. I'll see if I can add, er, fix that for Mailman 2.1.4. -Barry From skip at pobox.com Mon Dec 15 09:55:18 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 15 09:55:22 2003 Subject: [Python-Dev] Please tell me where I can submit bug reports.... In-Reply-To: References: Message-ID: <16349.52054.372100.462135@montanaro.dyndns.org> John> Then, where should I submit my core dump (if anywhere?)? Submit bug reports here: http://sourceforge.net/tracker/?group_id=5470&atid=105470 but please don't attach core files (they are too big and nobody will be able to make use of them). Instead, attach a small Python script which demonstrates how to generate a core dump. Skip From barry at python.org Mon Dec 15 09:56:49 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 09:56:53 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: References: <000201c3c289$976fcae0$e841fea9@oemcomputer> Message-ID: <1071500208.970.76.camel@anthem> On Sun, 2003-12-14 at 17:05, Fredrik Lundh wrote: > V. a join() builtin (join(x,s) becomes s.join(x)). All of the common sep.join() > use cases read better without the string literal prefix. Really? I predict everlasting confusion on argument order. It's not obvious from the method name whether the joining separator comes first or second. My wish list: V . Explicit global imports. I want a way to spell "import this from the global standard library even if there's a this.py in the local directory". VI. PEP 318 or something like it. -Barry From martin at v.loewis.de Mon Dec 15 09:59:42 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Mon Dec 15 10:00:11 2003 Subject: [Python-Dev] Please tell me where I can submit bug reports.... In-Reply-To: References: Message-ID: JD writes: > Running this under OpenBSD 3.2. Then, where should I submit my core > dump (if anywhere?)? You should submit bug reports to sf.net/projects/python, and provide a link to the core dump on a server of your control (as SF has a limit on file upload size). Please understand that several months, or even years, may pass until such a bug report is looked at, so if you actually need this to work, you will have to investigate it in more detail. On Unix, Python is, by default, compiled with symbols if the compiler is gcc. To find out why gdb does not see symbols, you should post a few lines of make output, showing typical compiler and linker invocations. Regards, Martin From fdrake at acm.org Mon Dec 15 10:14:24 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon Dec 15 10:14:30 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <1071500208.970.76.camel@anthem> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> Message-ID: <16349.53200.37613.119494@sftp.fdrake.net> Barry Warsaw writes: > V . Explicit global imports. I want a way to spell "import this > from the global standard library even if there's a this.py in the local > directory". I like the syntax: import global and from global import Look Guido, no new keywords! ;-) -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From jeremy at zope.com Mon Dec 15 10:50:47 2003 From: jeremy at zope.com (Jeremy Hylton) Date: Mon Dec 15 10:54:27 2003 Subject: [Python-Dev] urllib2 patch reviewer (852995)? In-Reply-To: <2mekv6jsqu.fsf@starship.python.net> References: <2mekv6jsqu.fsf@starship.python.net> Message-ID: <1071503446.15985.651.camel@localhost.localdomain> On Mon, 2003-12-15 at 05:11, Michael Hudson wrote: > John J Lee writes: > > > Jeremy Hylton just checked in a relatively major patch of mine to urllib2. > > He's obviously very busy, though, so it would be nice if somebody else who > > knows urllib2 could look at it, too. > > I don't have the know-how to review the patch, but I can notice that > test_urllib2 now seems to be creating uncollectable garbage... Yes. Didn't get a chance to fix that the other night. One of the classes has a pretty useless __del__ that might have helped before we had gc for cycles. The original urllib2 code is pretty old. Jeremy From guido at python.org Mon Dec 15 11:03:31 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 11:03:37 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 10:14:24 EST." <16349.53200.37613.119494@sftp.fdrake.net> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> Message-ID: <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> > Barry Warsaw writes: > > V . Explicit global imports. I want a way to spell "import this > > from the global standard library even if there's a this.py in the local > > directory". > > I like the syntax: > > import global > > and > > from global import > > Look Guido, no new keywords! ;-) I'd much rather invent new syntax to spell *local* imports. I like from .module import something You can even generalize and write from ..module import something to reference your package's parent package. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Mon Dec 15 11:05:42 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 15 11:05:47 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <2mad5ujsli.fsf@starship.python.net> Message-ID: [Michael Hudson] > If something important -- like a pieing -- depends on it, and > the ints are long enough, it's not that hard to do better than > Karatsuba multiplication... "Not that hard" depends on your background. The idea is to bury Dan under esoteric algorithms from a huge number of obscure specialties . From mwh at python.net Mon Dec 15 11:17:02 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 15 11:17:07 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: (Tim Peters's message of "Mon, 15 Dec 2003 11:05:42 -0500") References: Message-ID: <2mk74yhx8h.fsf@starship.python.net> "Tim Peters" writes: > [Michael Hudson] >> If something important -- like a pieing -- depends on it, and >> the ints are long enough, it's not that hard to do better than >> Karatsuba multiplication... > > "Not that hard" depends on your background. Granted, but while my background is maths it's not numerics and I know where to find descriptions and implementations of complexity-wise better algorithms. Using them is just a matter of engineering, right? > The idea is to bury Dan under esoteric algorithms from a huge number > of obscure specialties . Now there's a plan! Let's see, how do we do that? Excessive use of dicts, unicode esoterica and multiple inheritance seems like a good start. Cheers, mwh -- Its unmanageable complexity has spawned more fear-preventing tools than any other language, but the solution _should_ have been to create and use a language that does not overload the whole goddamn human brain with irrelevant details. -- Erik Naggum, comp.lang.lisp From barry at python.org Mon Dec 15 11:40:14 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 11:40:22 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> Message-ID: <1071506413.970.102.camel@anthem> On Mon, 2003-12-15 at 11:03, Guido van Rossum wrote: > I'd much rather invent new syntax to spell *local* imports. I like > > from .module import something > > You can even generalize and write > > from ..module import something > > to reference your package's parent package. :-) Actually, if truth be told, I'd love to just ban local imports. I understand that might be controversial though . The next best thing would be to force local imports to use special syntax (I was going to say "ugly" :) like what you suggest above, but I'm afraid that that might be more disruptive, requiring future imports and such. I think having a syntax for spelling global imports can be done in a backwards compatible way for 2.4. -Barry From tim.one at comcast.net Mon Dec 15 11:46:24 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 15 11:46:28 2003 Subject: [Python-Dev] Berkeley support in release23-maint Message-ID: Looking at http://cvs.sf.net/viewcvs.py/python/python/dist/src/Lib/bsddb/__init__.py it appear that 2+ months of Berkeley bugfixes (revs 1.10 through 1.12 of __init__.py, at least) didn't get backported to the 2.3 maint branch, and so are absent in 2.3.3c1. Is that right? From skip at pobox.com Mon Dec 15 11:52:15 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 15 11:52:28 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <1071506413.970.102.camel@anthem> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> Message-ID: <16349.59071.26004.502201@montanaro.dyndns.org> Guido> I'd much rather invent new syntax to spell *local* imports. Barry> Actually, if truth be told, I'd love to just ban local imports. Barry> I understand that might be controversial though . Okay, so what are global and local imports? I thought I understood after Barry's first post, but not now. I see the , but figured there's still some germ of desire in Barry's "ban local imports" comment. What kind of imports do we have today? Do they relate to the position of the import statement in the code or to the location of the module being imported along sys.path? Skip From guido at python.org Mon Dec 15 12:07:13 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 12:07:19 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 11:40:14 EST." <1071506413.970.102.camel@anthem> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> Message-ID: <200312151707.hBFH7DI17519@c-24-5-183-134.client.comcast.net> > On Mon, 2003-12-15 at 11:03, Guido van Rossum wrote: > > > I'd much rather invent new syntax to spell *local* imports. I like > > > > from .module import something > > > > You can even generalize and write > > > > from ..module import something > > > > to reference your package's parent package. :-) > > Actually, if truth be told, I'd love to just ban local imports. I > understand that might be controversial though . Works for me. :-) > The next best thing would be to force local imports to use special > syntax (I was going to say "ugly" :) like what you suggest above, but > I'm afraid that that might be more disruptive, requiring future imports > and such. I think having a syntax for spelling global imports can be > done in a backwards compatible way for 2.4. Well, but since you want all imports to be global, it'd be insane to introduce *new* syntax for global imports, wouldn't it? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 15 12:10:24 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 12:10:30 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 10:52:15 CST." <16349.59071.26004.502201@montanaro.dyndns.org> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <16349.59071.26004.502201@montanaro.dyndns.org> Message-ID: <200312151710.hBFHAO817542@c-24-5-183-134.client.comcast.net> > Okay, so what are global and local imports? I thought I understood > after Barry's first post, but not now. I see the , but > figured there's still some germ of desire in Barry's "ban local > imports" comment. What kind of imports do we have today? Do they > relate to the position of the import statement in the code or to the > location of the module being imported along sys.path? Traditionally, we've called these absolute (== global) and relative (== local) imports. The terms relate to the module's full package name, usually equivalent to its location on the filesystem. sys.path and the placement of the import statement do not enter into it, except that sys.path is the very definition of absolute import. --Guido van Rossum (home page: http://www.python.org/~guido/) From lkcl at lkcl.net Mon Dec 15 12:31:56 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Mon Dec 15 12:31:32 2003 Subject: [Python-Dev] rexec.py unuseable Message-ID: <20031215173156.GC25203@lkcl.net> i tried to reuse some code that i wrote for a project back when python 2.0 and 2.1 were in use. i must be about the only person in existence who wrote some code that uses - and totally relies on - the rexec.py module. what that code does (did) was to provide a database-based method of running restricted programs for use to monitor and manage remote servers. i.e. it became in effect a distributed python operating system. what happened was that the name of a python script, its arguments, a relevant ip address and the server on which the script was to be run were all entered into a database table, and the server picked that up, loaded the script and its arguments from the database and then ran the script, which usually involved contacting the remote ip address. so, not only was rhooks used to overload "import" such that libraries used by scripts in the database could _also_ be loaded from the database, but rexec.py was also used to restrict what functions could be accessed. i was quite happy with rexec.py as it stood. it was the c++ functional equivalent of the "protected" convention for c++ classes. at the time that i tried to reuse the code, i discovered the lovely "this module is broken in python 2.2 and 2.3" message. well, uhm, if it's broken, why hasn't it been reverted to the 2.0 code? i noted, some time about a year ago, that someone attempted to drill down the rexec security into sub-modules. i.e. they attempted to add the c++ functional equivalence of the "private" convention for c++ classes. consequently, a number of routines that wrote to files that i had DELIBERATELY wrapped and made available to first-level rexec-restricted library routines, e.g. the logging functions which output debug information to log files, suddenly stopped working. why? whereas before, the debug routine could be called directly but the write operation could not, because of the drill-down effect, all of a sudden the write operation, as called _by_ the debug routine, is now restricted in the 2.1 (or early 2.2) code, and my debug library routine now doesn't work, nor indeed does anything else. the thing is that the functionality of the python 2.0 rexec.py was _exactly_ as i would have expected it to be, and it was working _perfectly_ according to my expectations of that code. i therefore believe that the attempts to take rexec one step further, aka a "drill-down" effect, by making all functions that rexec functions import _also_ restricted, to have been a mistake. or, at least, not carefully thought out. if such additional restrictions are to be added, i believe that they should be added as a user-configureable parameter (e.g. named recursive-restrict) _please_ could the functionality of rexec.py be restored to python 2.0 functionality, with the "rexec.py is broken" message only coming up when the suggested-named parameter "recursive-restrict" is set. maybe then i can consider reviving that code and re-releasing it, because at the moment it's totally going to waste. many many thanks, l. -- -- expecting email to be received and understood is a bit like picking up the telephone and immediately dialing without checking for a dial-tone; speaking immediately without listening for either an answer or ring-tone; hanging up immediately and then expecting someone to call you (and to be able to call you). -- every day, people send out email expecting it to be received without being tampered with, read by other people, delayed or simply - without prejudice but lots of incompetence - destroyed. -- please therefore treat email more like you would a CB radio to communicate across the world (via relaying stations): ask and expect people to confirm receipt; send nothing that you don't mind everyone in the world knowing about... From barry at python.org Mon Dec 15 12:37:27 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 12:37:33 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <16349.59071.26004.502201@montanaro.dyndns.org> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <16349.59071.26004.502201@montanaro.dyndns.org> Message-ID: <1071509846.970.105.camel@anthem> On Mon, 2003-12-15 at 11:52, Skip Montanaro wrote: > Guido> I'd much rather invent new syntax to spell *local* imports. > > Barry> Actually, if truth be told, I'd love to just ban local imports. > Barry> I understand that might be controversial though . > > Okay, so what are global and local imports? Sorry, let me rephrase. I'd love to ban relative imports. Personally, I think all imports should be specified using fully-qualified absolute package paths. -Barry From barry at python.org Mon Dec 15 12:39:47 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 12:39:54 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <200312151707.hBFH7DI17519@c-24-5-183-134.client.comcast.net> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <200312151707.hBFH7DI17519@c-24-5-183-134.client.comcast.net> Message-ID: <1071509987.970.108.camel@anthem> On Mon, 2003-12-15 at 12:07, Guido van Rossum wrote: > > Actually, if truth be told, I'd love to just ban local imports. I > > understand that might be controversial though . > > Works for me. :-) Yay! :) > > The next best thing would be to force local imports to use special > > syntax (I was going to say "ugly" :) like what you suggest above, but > > I'm afraid that that might be more disruptive, requiring future imports > > and such. I think having a syntax for spelling global imports can be > > done in a backwards compatible way for 2.4. > > Well, but since you want all imports to be global, it'd be insane to > introduce *new* syntax for global imports, wouldn't it? If we banned relative (a.k.a. local) imports, yes definitely. from __future__ global_imports ? pep-time-ly y'rs, -Barry From edloper at gradient.cis.upenn.edu Mon Dec 15 13:44:56 2003 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Mon Dec 15 12:43:15 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: References: Message-ID: <3FDE0128.1040600@gradient.cis.upenn.edu> Barry Warsaw wrote: > Actually, if truth be told, I'd love to just ban local imports. I > understand that might be controversial though . +1 (asuming that you're talking about package-local imports). I've seen a couple people get bitten by the fact that a module gets loaded twice if there are separate local & global imports for it: % mkdir pkg % touch pkg/__init__.py % cat >pkg/a.py print 'A is being imported (not re-used)' class A: pass % cat >pkg/b.py from a import A as LocalA from pkg.a import A as GlobalA print isinstance(LocalA(), GlobalA) print isinstance(GlobalA(), LocalA) % PYTHONPATH=.; python pkg/b.py A is being imported (not re-used) A is being imported (not re-used) 0 0 Since pkg/a gets loaded twice, we end up with two versions of class A, which are not compatible. In practice this usually comes up if a package uses local imports between submodules, and then an outside user uses a global import to get a submodule. -Edward From pycon-organizers at python.org Mon Dec 15 12:54:27 2003 From: pycon-organizers at python.org (pycon-organizers@python.org) Date: Mon Dec 15 12:54:29 2003 Subject: [Python-Dev] PyCon DC 2004 - Submissions Now Open Message-ID: Dear Python User: I am pleased to be able to announce that the submission process for PyCon DC 2004 is now open. Please direct your browser to http://submit.pycon.org/ and create your account by entering your email address and leaving the password field blank - a password will be sent to you by email, and you can submit your paper! Please remember that your submissions should ideally be in a single file, containing HTML or the reStructured Text format. Ultimately we would like to publish all accepted papers on the web, and these rules should make it easier to do so. Send enquiries about other formats to pycon-organizers@python.org If your paper is accepted and you prepare an electronic presentation (in PDF, PythonPoint or PowerPoint) we will also happily publish that on the web site when PyCon is over. We are interested in any and all submissions about uses of Python and the development of the language. Since there is expected to be a strong educational community presence for the next PyCon, teaching materials of various kinds are also encouraged. Please help us to make the next PyCon even better than the first one. See you in March! Sincerely Steve Holden Chairman PyCon DC 2004 From klm at zope.com Mon Dec 15 12:48:32 2003 From: klm at zope.com (Ken Manheimer) Date: Mon Dec 15 12:54:58 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> Message-ID: On Mon, 15 Dec 2003, Guido van Rossum wrote: > > Barry Warsaw writes: > > > V . Explicit global imports. I want a way to spell "import this > > > from the global standard library even if there's a this.py in the local > > > directory". > > > > I like the syntax: > > > > import global > > > > and > > > > from global import > > > > Look Guido, no new keywords! ;-) > > I'd much rather invent new syntax to spell *local* imports. I like > > from .module import something > > You can even generalize and write > > from ..module import something > > to reference your package's parent package. :-) As i've said before (though it's been a while) i think people want this, it would be useful, and i think it would work well. Barry wrote, in response to guido's message: < Actually, if truth be told, I'd love to just ban local imports. I < understand that might be controversial though . I don't understand the desire to totally prevent relative imports. It would be quite useful in avoiding growing module-naming problems, where you have to be careful about shadowing a global module with one in your package. I *can* understand objecting to the ambiguity in the current situation, where you don't know whether a module import will resolve to a sibling module or one in higher up in the hierarchy. That is why i strongly prefer having a leading dot to explicitly signify a relative import, and originally proposed it that way. I expect that the elimination of the ambiguity would mitigate the objection to relative imports - and we would do away with the shadowing problems, and even have easy relocation of modules, if there are cases where software does need to move around. Ken klm@zope.com From skip at pobox.com Mon Dec 15 13:01:07 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 15 13:01:29 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031215173156.GC25203@lkcl.net> References: <20031215173156.GC25203@lkcl.net> Message-ID: <16349.63203.797449.677617@montanaro.dyndns.org> Luke> well, uhm, if [rexec]'s broken, why hasn't it been reverted to the Luke> 2.0 code? My understanding was that in 2.0 it was also broken, but that (relatively) recent changes to Python made the breakage easier to exploit. As for alternatives, I suggest you post a query on comp.lang.python (aka python-list@python.org). Skip From skip at pobox.com Mon Dec 15 13:04:32 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 15 13:04:33 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <1071509846.970.105.camel@anthem> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <16349.59071.26004.502201@montanaro.dyndns.org> <1071509846.970.105.camel@anthem> Message-ID: <16349.63408.143245.274390@montanaro.dyndns.org> >> Okay, so what are global and local imports? Barry> Sorry, let me rephrase. I'd love to ban relative imports. Barry> Personally, I think all imports should be specified using Barry> fully-qualified absolute package paths. That's fine with me. Maybe continue to use "absolute" and "relative" (and throw in "package" somewhere) instead of "global" and "local" when referring to package import semantics? "global" and "local" have a well-established meaning in day-to-day Python programming which doesn't overlap much with the import mechanism. Skip From guido at python.org Mon Dec 15 13:36:44 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 13:36:50 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 12:39:47 EST." <1071509987.970.108.camel@anthem> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <200312151707.hBFH7DI17519@c-24-5-183-134.client.comcast.net> <1071509987.970.108.camel@anthem> Message-ID: <200312151836.hBFIaie17722@c-24-5-183-134.client.comcast.net> > > Well, but since you want all imports to be global, it'd be insane to > > introduce *new* syntax for global imports, wouldn't it? > > If we banned relative (a.k.a. local) imports, yes definitely. > > from __future__ global_imports I think this ought to be a *global* flag rather than a per-module flag. E.g. after setting sys.allow_relative_import = False all imports anywhere would be interpreted as absolute imports. This would mean you couldn't have some code that still uses relative imports, but the problem with the __future__ statement is that it seems so pointless: packages that only use absolute imports don't need it, and packages that use relative imports break if it is used. About the only time where the __future__ statement would make a difference is when a package defines a local module whose name is the same as that of a global module, *and* the package also wants to import the global module. I would personally solve that by renaming the local module though... --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 15 13:38:01 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 13:38:08 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 12:44:56 CST." <3FDE0128.1040600@gradient.cis.upenn.edu> References: <3FDE0128.1040600@gradient.cis.upenn.edu> Message-ID: <200312151838.hBFIc1q17738@c-24-5-183-134.client.comcast.net> > I've seen a couple people get bitten by the fact that a module gets > loaded twice if there are separate local & global imports for it: > > % mkdir pkg > > % touch pkg/__init__.py > > % cat >pkg/a.py > print 'A is being imported (not re-used)' > class A: pass > > % cat >pkg/b.py > from a import A as LocalA > from pkg.a import A as GlobalA > print isinstance(LocalA(), GlobalA) > print isinstance(GlobalA(), LocalA) > > % PYTHONPATH=.; python pkg/b.py > A is being imported (not re-used) > A is being imported (not re-used) > 0 > 0 > > Since pkg/a gets loaded twice, we end up with two versions of class A, > which are not compatible. In practice this usually comes up if a > package uses local imports between submodules, and then an outside user > uses a global import to get a submodule. That's actually a slightly different problem; sys.path should not include any directories that are also packages on other directories in sys.path. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 15 13:42:00 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 13:42:06 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 12:48:32 EST." References: Message-ID: <200312151842.hBFIg0K17763@c-24-5-183-134.client.comcast.net> > I don't understand the desire to totally prevent relative imports. It's because of TOOWTDI. > It would be quite useful in avoiding growing module-naming problems, > where you have to be careful about shadowing a global module with > one in your package. I don't see the problem, or I misuderstand what "it" refers to; it seems you have this backwards if it refers to relative imports. Say my package P defines a submodule sys. If we require absolute imports, there is no ambiguity: the submodule sys must be imported as P.sys while the standard sys module can be imported as simply sys. --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Mon Dec 15 13:49:36 2003 From: dan at sidhe.org (Dan Sugalski) Date: Mon Dec 15 13:50:02 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: References: Message-ID: At 11:05 AM -0500 12/15/03, Tim Peters wrote: >[Michael Hudson] >> If something important -- like a pieing -- depends on it, and >> the ints are long enough, it's not that hard to do better than >> Karatsuba multiplication... > >"Not that hard" depends on your background. The idea is to bury Dan under >esoteric algorithms from a huge number of obscure specialties . Heh. You're expecting me to do far too much work. I plan on tackling bytecode optimization only if it turns out that a straightforward transform from python bytecode to parrot bytecode is isn't sufficiently faster than CPython. Funky algorithms expressed entirely in python bytecode won't make much of a difference... -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From barry at python.org Mon Dec 15 13:51:07 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 13:51:10 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <200312151836.hBFIaie17722@c-24-5-183-134.client.comcast.net> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <200312151707.hBFH7DI17519@c-24-5-183-134.client.comcast.net> <1071509987.970.108.camel@anthem> <200312151836.hBFIaie17722@c-24-5-183-134.client.comcast.net> Message-ID: <1071514266.28081.2.camel@geddy> On Mon, 2003-12-15 at 13:36, Guido van Rossum wrote: > I think this ought to be a *global* flag rather than a per-module > flag. E.g. after setting > > sys.allow_relative_import = False > > all imports anywhere would be interpreted as absolute imports. > > This would mean you couldn't have some code that still uses relative > imports, but the problem with the __future__ statement is that it > seems so pointless: packages that only use absolute imports don't need > it, and packages that use relative imports break if it is used. You probably want something at the package level. This may not be feasible, but it seems like you want to be able to say, "this package needs to allow relative imports for backwards compatibility". -Barry From lkcl at lkcl.net Mon Dec 15 14:10:23 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Mon Dec 15 14:10:14 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <16349.63203.797449.677617@montanaro.dyndns.org> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> Message-ID: <20031215191023.GB26055@lkcl.net> On Mon, Dec 15, 2003 at 12:01:07PM -0600, Skip Montanaro wrote: > > Luke> well, uhm, if [rexec]'s broken, why hasn't it been reverted to the > Luke> 2.0 code? > > My understanding was that in 2.0 it was also broken, but that (relatively) > recent changes to Python made the breakage easier to exploit. the issue is [back-tracking to the 2.0 code], were those "breakages" actually features, or were they definitely "breakages"? i.e. was it 1) was it just unexpected behaviour in that a function _called_ by a rexec'd restricted routine could do, for example writes; was it that an unrestricted function, module or variable imported by a rexec'd restricted routine or module could be accessed. e.g. like this: contents of bypassmodule.py: import open as unrestrictedopen then code run under rexec: from bypassmodule import unrestrictedopen both of these things i would consider to be totally acceptable behaviour of the [2.] rexec.py module!!! i would NOT consider them to be a bug, and in fact are desirable behaviour in order to get my code working. i believe that there exists in [2.0] rexec a mechanism to specify what functions are allowed in a particular module, so i could restrict access to the bypassmodule.unrestrictedopen function, if necessary. OR was it 2) a genuine bug. > As for alternatives, I suggest you post a query on comp.lang.python (aka > python-list@python.org). ta! > Skip -- -- expecting email to be received and understood is a bit like picking up the telephone and immediately dialing without checking for a dial-tone; speaking immediately without listening for either an answer or ring-tone; hanging up immediately and then expecting someone to call you (and to be able to call you). -- every day, people send out email expecting it to be received without being tampered with, read by other people, delayed or simply - without prejudice but lots of incompetence - destroyed. -- please therefore treat email more like you would a CB radio to communicate across the world (via relaying stations): ask and expect people to confirm receipt; send nothing that you don't mind everyone in the world knowing about... From python at rcn.com Mon Dec 15 14:26:03 2003 From: python at rcn.com (Raymond Hettinger) Date: Mon Dec 15 14:26:30 2003 Subject: [Python-Dev] Christmas Wishlist In-Reply-To: <20031215030347.GC27067@unpythonic.net> Message-ID: <004301c3c341$48053fe0$e841fea9@oemcomputer> > On Sun, Dec 14, 2003 at 04:31:09PM -0500, Raymond Hettinger wrote: > > I. eval() to accept custom mapping arguments for globals and locals. > > This makes it possible to write a smart __getitem__ method for > > applications like spreadsheets or case-insensitive evaluation. > > http://mail.python.org/pipermail/python-dev/2002-October/029770.html > [and the rest of that thread] [Jeff Epler] > A 3% slowdown in the common case was considered too big to allow a > feature like this to be included. The slowdown might be smaller if > the builtin module wasn't allowed to be anything but a dict instance, or > might disappear if there were two versions of ceval.c were included, and > one used PyObject_ APIs and the other used PyDict_ APIs (as long as only > one was fighting for space in the cpu cache, main memory, etc) Ordinarily, I would say that 3% is a reasonable price to pay for this functionality but there is that pesky pie throwing event coming up. Raymond Hettinger From pje at telecommunity.com Mon Dec 15 14:32:27 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Dec 15 14:32:37 2003 Subject: [Python-Dev] Christmas Wishlist In-Reply-To: <004301c3c341$48053fe0$e841fea9@oemcomputer> References: <20031215030347.GC27067@unpythonic.net> Message-ID: <5.1.1.6.0.20031215142906.01e00ac0@telecommunity.com> At 02:26 PM 12/15/03 -0500, Raymond Hettinger wrote: > > On Sun, Dec 14, 2003 at 04:31:09PM -0500, Raymond Hettinger wrote: > > > I. eval() to accept custom mapping arguments for globals and locals. > > > This makes it possible to write a smart __getitem__ method for > > > applications like spreadsheets or case-insensitive evaluation. > > > > http://mail.python.org/pipermail/python-dev/2002-October/029770.html > > [and the rest of that thread] > >[Jeff Epler] > > A 3% slowdown in the common case was considered too big to allow a > > feature like this to be included. The slowdown might be smaller if > > the builtin module wasn't allowed to be anything but a dict instance, >or > > might disappear if there were two versions of ceval.c were included, >and > > one used PyObject_ APIs and the other used PyDict_ APIs (as long as >only > > one was fighting for space in the cpu cache, main memory, etc) > >Ordinarily, I would say that 3% is a reasonable price to pay for this >functionality but there is that pesky pie throwing event coming up. There is a workaround for this in CPython, which used to be used by Zope, and which I have used on occasion: scan the code object for LOAD_NAME opcodes, and note what names are used by the code block. Then, look up those names in your non-dictionary dictionary, and copy them into a real dictionary. So, it's got to be something pretty specialized to really need PyObject_* APIs instead of PyDict_* APIs in CPython at present. Why penalize *everything* for such an specialized need? From guido at python.org Mon Dec 15 14:34:10 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 14:34:15 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 13:51:07 EST." <1071514266.28081.2.camel@geddy> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <200312151707.hBFH7DI17519@c-24-5-183-134.client.comcast.net> <1071509987.970.108.camel@anthem> <200312151836.hBFIaie17722@c-24-5-183-134.client.comcast.net> <1071514266.28081.2.camel@geddy> Message-ID: <200312151934.hBFJYAB17994@c-24-5-183-134.client.comcast.net> > > This would mean you couldn't have some code that still uses relative > > imports, but the problem with the __future__ statement is that it > > seems so pointless: packages that only use absolute imports don't need > > it, and packages that use relative imports break if it is used. > > You probably want something at the package level. This may not be > feasible, but it seems like you want to be able to say, "this package > needs to allow relative imports for backwards compatibility". But there is no backwards compatibility issue. Absolute imports always work, so instead of adding a "relative imports allowed" to your package, you can just change the package to use only absolute imports. Using only absolute imports is totally backwards compatible (unless your toplevel package name is the same as the name of a submodule of the package, which seems pretty sick). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 15 14:37:34 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 14:37:40 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Mon, 15 Dec 2003 13:49:36 EST." References: Message-ID: <200312151937.hBFJbYX18012@c-24-5-183-134.client.comcast.net> > At 11:05 AM -0500 12/15/03, Tim Peters wrote: > >[Michael Hudson] > >> If something important -- like a pieing -- depends on it, and > >> the ints are long enough, it's not that hard to do better than > >> Karatsuba multiplication... > > > >"Not that hard" depends on your background. The idea is to bury Dan under > >esoteric algorithms from a huge number of obscure specialties . [Dan Sugalski] > Heh. You're expecting me to do far too much work. I plan on tackling > bytecode optimization only if it turns out that a straightforward > transform from python bytecode to parrot bytecode is isn't > sufficiently faster than CPython. Funky algorithms expressed entirely > in python bytecode won't make much of a difference... But Karatsuba multiplication and Timsort are implemented highly efficiently in C, and are part of the "full set of standard builtins" to which we agreed. BTW, eval() is also part of that full set, and exec is part of the language. These invoke the Python bytecode compiler. (And Jim Hugunin's IronPython is really slow on this, like 60x slower than CPython; I think Jim reported that Jython was 20x slower on exec().) How do you plan to handle these? What if the entire benchmark was given as a triple quoted string that was passed to the exec statement? --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Mon Dec 15 14:40:30 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 15 14:40:55 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031215191023.GB26055@lkcl.net> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> Message-ID: <16350.3630.307765.836000@montanaro.dyndns.org> >> My understanding was that in 2.0 it was also broken, but that >> (relatively) recent changes to Python made the breakage easier to >> exploit. Luke> the issue is [back-tracking to the 2.0 code], Luke> were those "breakages" actually features, or were they definitely Luke> "breakages"? Dunno. Take a look at the 2.3 Whats New: http://www.python.org/doc/current/whatsnew/node18.html Search for "rexec". I don't know more than that. It only mentions back as far as 2.1, so I don't know if rexec could be made to work based upon the code in 2.0. Also, check this thread from python-dev: http://mail.python.org/pipermail/python-dev/2003-January/thread.html#31842 That thread includes some indication of what the problem is and how it got worse in 2.2 and 2.3. Skip From klm at zope.com Mon Dec 15 14:34:33 2003 From: klm at zope.com (Ken Manheimer) Date: Mon Dec 15 14:40:59 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <200312151836.hBFIaie17722@c-24-5-183-134.client.comcast.net> Message-ID: On Mon, 15 Dec 2003, Guido van Rossum wrote: > > It would be quite useful in avoiding growing module-naming problems, > > where you have to be careful about shadowing a global module with > > one in your package. > > I don't see the problem, or I misuderstand what "it" refers to; it > seems you have this backwards if it refers to relative imports. Say > my package P defines a submodule sys. If we require absolute imports, > there is no ambiguity: the submodule sys must be imported as P.sys > while the standard sys module can be imported as simply sys. I guess i think there is a case to be made for relative imports, and it becomes apparent as an enterprise grows. Increasingly at Zope corp we are mixing and matching packages to deliver applications to a customer. This is very encouraging, because it means we are actually getting reuse out of the packages. Currently, we can just plop the things in place, and use them. Without relative imports, we would have to be editing the imports in the packages in each place we use it. This would increase the burden of using the packages and even more of feeding back actual fixes - which we then have to distinguish from what i would see as gratuitous changes to edit import names. If you would grant there's use to avoiding those edits, and so there's use to having relative imports, then you might see a reason to solve the problems where the names in a package conflict with those along the load path. Otherwise, if there's only absolute imports, there's no ambiguity to resolve. I'd say there's a legitimate need for relative imports, and we need some explicit gesture to disambiguate between a relative and absolute import, one way or the other. Ken From dan at sidhe.org Mon Dec 15 14:51:39 2003 From: dan at sidhe.org (Dan Sugalski) Date: Mon Dec 15 14:51:56 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <200312151937.hBFJbYX18012@c-24-5-183-134.client.comcast.net> References: <200312151937.hBFJbYX18012@c-24-5-183-134.client.comcast.net> Message-ID: At 11:37 AM -0800 12/15/03, Guido van Rossum wrote: > > At 11:05 AM -0500 12/15/03, Tim Peters wrote: >> >[Michael Hudson] >> >> If something important -- like a pieing -- depends on it, and >> >> the ints are long enough, it's not that hard to do better than >> >> Karatsuba multiplication... >> > >> >"Not that hard" depends on your background. The idea is to bury Dan under >> >esoteric algorithms from a huge number of obscure specialties . > >[Dan Sugalski] >> Heh. You're expecting me to do far too much work. I plan on tackling >> bytecode optimization only if it turns out that a straightforward >> transform from python bytecode to parrot bytecode is isn't >> sufficiently faster than CPython. Funky algorithms expressed entirely >> in python bytecode won't make much of a difference... > >But Karatsuba multiplication and Timsort are implemented highly >efficiently in C, and are part of the "full set of standard builtins" >to which we agreed. In which case, well... time for me to write some C code. >BTW, eval() is also part of that full set, and exec is part of the >language. These invoke the Python bytecode compiler. (And Jim >Hugunin's IronPython is really slow on this, like 60x slower than >CPython; I think Jim reported that Jython was 20x slower on exec().) > >How do you plan to handle these? What if the entire benchmark was >given as a triple quoted string that was passed to the exec statement? We agreed no eval as part of this challenge, since the point was the speed of the interpreter engine not of the compiler. I don't see that on the list of challenge points you posted earlier, though, so it was either omitted inadvertently or I got snookered. The former, I expect. -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido at python.org Mon Dec 15 14:54:46 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 14:54:54 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 14:34:33 EST." References: Message-ID: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> > I guess i think there is a case to be made for relative imports, and > it becomes apparent as an enterprise grows. > > Increasingly at Zope corp we are mixing and matching packages to > deliver applications to a customer. This is very encouraging, because > it means we are actually getting reuse out of the packages. > > Currently, we can just plop the things in place, and use them. > Without relative imports, we would have to be editing the imports in > the packages in each place we use it. This would increase the burden > of using the packages and even more of feeding back actual fixes - > which we then have to distinguish from what i would see as gratuitous > changes to edit import names. I know this line of reasoning fairly well. You are taking 3rd party packages (or perhaps internally developed packages) and copy them to a *different place in the package namespace tree*. Right? But why do you have to give those packages a different full name? That's the question that I've never seen answered adequately. > If you would grant there's use to avoiding those edits, and so there's > use to having relative imports, then you might see a reason to solve > the problems where the names in a package conflict with those along > the load path. Otherwise, if there's only absolute imports, there's > no ambiguity to resolve. I'd say there's a legitimate need for > relative imports, and we need some explicit gesture to disambiguate > between a relative and absolute import, one way or the other. I think that moving packages around in the package namespace is a bad idea. But maybe you can give me an answer to the question above to convince me. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 15 14:57:32 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 14:57:38 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Mon, 15 Dec 2003 14:51:39 EST." References: <200312151937.hBFJbYX18012@c-24-5-183-134.client.comcast.net> Message-ID: <200312151957.hBFJvWi18151@c-24-5-183-134.client.comcast.net> > >BTW, eval() is also part of that full set, and exec is part of the > >language. These invoke the Python bytecode compiler. (And Jim > >Hugunin's IronPython is really slow on this, like 60x slower than > >CPython; I think Jim reported that Jython was 20x slower on exec().) > > > >How do you plan to handle these? What if the entire benchmark was > >given as a triple quoted string that was passed to the exec statement? > > We agreed no eval as part of this challenge, since the point was the > speed of the interpreter engine not of the compiler. I don't see that > on the list of challenge points you posted earlier, though, so it was > either omitted inadvertently or I got snookered. The former, I expect. Fair enough. "No exec or eval" is now officially part of the challenge. You know, I *want* you to win, at least if you can win by a great big margin. Because then we could switch to Parrot to make Python faster. I just very much doubt that you'll be able to bat CPython. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Mon Dec 15 15:21:16 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 15 15:21:22 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Message-ID: [Tim] >> "Not that hard" depends on your background. The idea is to bury Dan >> under esoteric algorithms from a huge number of obscure specialties >> . [Dan Sugalski] > Heh. You're expecting me to do far too much work. I plan on tackling > bytecode optimization only if it turns out that a straightforward > transform from python bytecode to parrot bytecode is isn't > sufficiently faster than CPython. Funky algorithms expressed entirely > in python bytecode won't make much of a difference... There is a serious point buried in this. This is how Python spells small-int multiplication in the bytecode: BINARY_MULTIPLY And this is how it spells unbounded-int Karatsuba multiplication in the bytecode: BINARY_MULTIPLY Same thing, and floating multiply, complex multiply, catenating N copies of a string, and arbitrary user-defined Python code overloading infix "*" for a class are spelled BINARY_MULTIPLY in the bytecode too. There's simply nothing in the bytecode to distinguish these cases. It wouldn't be hard for Guido to construct a vanilla-looking Python program that tickles some of the extreme-win special cases in the type-dependent implementations of the operators. That wouldn't be fair, so I'll hit him if he does , but in some smaller senses it can't be avoided. For example, all idiomatic Python programs use dicts (hashes), and the dict implementation dynamically keeps track of whether any non-string key has been added (Python dicts can be indexed by lots of things, not just strings). So long as it hasn't, special methods specialized to string keys are used; but as soon as, e.g., a float key is added, a Python dict switches to using slower, more-general methods. None of that is reflected in the bytecode, of course. There's a lot of stuff like that. pystone tickles little of that nature, because pystone is such an *a*typical Python program. So, regardless of what the official benchmark turns out to be, I'll probably be more interested in your pystone result . From barry at python.org Mon Dec 15 15:23:15 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 15:23:23 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: References: Message-ID: <1071519794.2808.24.camel@anthem> On Mon, 2003-12-15 at 14:34, Ken Manheimer wrote: > Currently, we can just plop the things in place, and use them. > Without relative imports, we would have to be editing the imports in > the packages in each place we use it. I've heard this before, but I still don't get it, so I'd like to tease this one out more. What does "plop the things in place" mean? Zope 2 has a few standard directories it adds to its PYTHONPATH. If I distribute a 3rd party package Zope wants to use, and that package uses absolute imports, you can just drop the package in one of those directories and it'll just work. Maybe you mean that you want to drop the mythical 3rd party package in somewhere other than a directory on Zope's PYTHONPATH and have it just work. I'd say that's too perverse of a use-case for Python to worry about. One problem with banning relative imports is that Python doesn't allow a top-level package name to live in more than one place on PYTHONPATH. It would be useful for Python to support such things out of the box, say if we wanted to adopt a Java style naming convention for package organizations. Jim uses the term "pure container package" to describe top-level packages that contain no real content in the package except subpackages. "zope" and "zodb" are examples of this in Zope 3. IWBNI Python had better/easier support for weaving together pure container package contents where subdirectories live in different locations on the file system. -Barry From tim.one at comcast.net Mon Dec 15 15:28:26 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 15 15:28:32 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <200312151957.hBFJvWi18151@c-24-5-183-134.client.comcast.net> Message-ID: [Guido, to Dan Sugalski] > ... > You know, I *want* you to win, at least if you can win by a great big > margin. Because then we could switch to Parrot to make Python > faster. I just very much doubt that you'll be able to bat CPython. Note that Fredrik boosted his pystone result from 2267 under CPython, to 58038 under his Pytte1 "small Python-like language" interpreter: http://effbot.org/zone/pyttestone.htm The way to prevent this is to make the benchmark exercise lots of Python's features. I suggest at least one instance of each of the 7016 ways to call a function . From fred at zope.com Mon Dec 15 15:51:27 2003 From: fred at zope.com (Fred L. Drake, Jr.) Date: Mon Dec 15 15:51:35 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <1071519794.2808.24.camel@anthem> References: <1071519794.2808.24.camel@anthem> Message-ID: <16350.7887.116951.920720@sftp.fdrake.net> Barry Warsaw writes: > Jim uses the term "pure container package" to describe top-level > packages that contain no real content in the package except > subpackages. "zope" and "zodb" are examples of this in Zope 3. IWBNI > Python had better/easier support for weaving together pure container > package contents where subdirectories live in different locations on the > file system. +1 The .pth files processed by site.py support two interesting kinds of lines: - directories that should be added to sys.path, and - import statements The later alone is interesting enough, since it allows arbitrary initialization code to run when the import is executed. This still happens before all the .pth files have been processed; I suspect we really want the container package to be a little more flexible than that, at least allowing their __path__ to be computed after all .pth files have been processed. Perhaps it would be nice to have a hook (probably in sys) that gets called after site/sitecustomize are done? A "pure container package" could use a .pth file using "import mypackage", and the package's __init__.py would register a hook function in sys that searches sys.path for whatever directories should be used in mypackage.__path__. This also avoids the code being too specific to one way of thinking about packages. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From lists at webcrunchers.com Mon Dec 15 16:05:51 2003 From: lists at webcrunchers.com (JD) Date: Mon Dec 15 16:06:09 2003 Subject: [Python-Dev] Please tell me where I can submit bug reports.... In-Reply-To: <16349.52054.372100.462135@montanaro.dyndns.org> References: <16349.52054.372100.462135@montanaro.dyndns.org> Message-ID: <76BB1B8C-2F42-11D8-BFA1-0030656C6B9E@webcrunchers.com> On Dec 15, 2003, at 6:55 AM, Skip Montanaro wrote: > > John> Then, where should I submit my core dump (if anywhere?)? > > Submit bug reports here: > > http://sourceforge.net/tracker/?group_id=5470&atid=105470 Thanx, OK, I logged in, and got the main page which confirmed my login was good. But I cannot find any links that allow me to submit bug reports. Where is the link? I cannot find it. On the left part of the screen, I get this? Does one of these links go to the bug reporting page? ?SF.net Resources ???Site Docs ???Site Status ???Site Map ???SF.net Supporters ???Compile Farm ???Foundries ???Project Help Wanted ???New Releases ???Get Support There are two columns in the main page.... One titled... My Assigned Tracker Items and another titled My Tasks I see no place on this site where there is a link I can use to report any bugs... If it don't say "Bug reporting" or anything similar, then I need to know the name of the link, because it clearly isn't very obvious. Another thing also.... if I click on Python Bugs link in the left column, it goes into a "Browse Bugs" section, but then there is yet another "Please log in" in red. OK, so I log in again..... but when I do, I don't see the "Submit New" button, but I do see a "Submit New" link above the buttons. I DO have Cookies enabled of course, but I'm using Safari browser. By the way, I DO click on SSL. I just tried it with IE and get the same results.... I think I finally submitted it.... but can I go back and add some additional info? If so, how? Further details on the Python core dump: It core dumps (segmentation fault) when I check the database for a certain key. The offending statement is: x=database.has_key(keystring). database is open and otherwise normal. The database is "gdbm-1.8.3". The offending key is "3012a3ec5c796329fee7c242d4df00d6". John -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 2601 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20031215/4aba71e1/attachment.bin From skip at pobox.com Mon Dec 15 16:15:42 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 15 16:15:44 2003 Subject: [Python-Dev] Please tell me where I can submit bug reports.... In-Reply-To: <76BB1B8C-2F42-11D8-BFA1-0030656C6B9E@webcrunchers.com> References: <16349.52054.372100.462135@montanaro.dyndns.org> <76BB1B8C-2F42-11D8-BFA1-0030656C6B9E@webcrunchers.com> Message-ID: <16350.9342.865204.274114@montanaro.dyndns.org> >> Submit bug reports here: >> >> http://sourceforge.net/tracker/?group_id=5470&atid=105470 JD> Thanx, OK, I logged in, and got the main page which confirmed my JD> login was good. But I cannot find any links that allow me to submit JD> bug reports. Where is the link? I cannot find it. Pop up that page (while logged in) and search for "Submit New". JD> There are two columns in the main page.... One titled... JD> My Assigned Tracker Items and another titled My Tasks Wrong page. Once you log in, SF takes you to your "my sf.net" page. After logging in, pop back to http://sourceforge.net/projects/python then poke the "Bugs" link, which will take you to the bug tracker (the first URL above). JD> Another thing also.... if I click on Python Bugs link in the left JD> column, it goes into a "Browse Bugs" section, but then there is yet JD> another "Please log in" in red. Sounds like you have cookies disabled. To use SourceForge you need to run with cookie support turned on. Skip From jepler at unpythonic.net Mon Dec 15 16:34:03 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Mon Dec 15 16:34:07 2003 Subject: [Python-Dev] Christmas Wishlist In-Reply-To: <004301c3c341$48053fe0$e841fea9@oemcomputer> References: <20031215030347.GC27067@unpythonic.net> <004301c3c341$48053fe0$e841fea9@oemcomputer> Message-ID: <20031215213354.GH17242@unpythonic.net> On Mon, Dec 15, 2003 at 02:26:03PM -0500, Raymond Hettinger wrote: > Ordinarily, I would say that 3% is a reasonable price to pay for this > functionality but there is that pesky pie throwing event coming up. Maybe we can add it in right after the benchmarks get run? Jeff From jeremy at alum.mit.edu Mon Dec 15 16:36:35 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Mon Dec 15 16:39:25 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031215191023.GB26055@lkcl.net> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> Message-ID: <1071524194.15985.660.camel@localhost.localdomain> One kind of problem is that newer Python features were designed without taking rexec into account. It's possible for untrusted code to cause the trusted interpreter to execute its code without restriction using descriptors. It would be really difficult to reconcile new-style classes and rexec. Perhaps a worthwhile project, but probably one accomplished by design a new restriction mechanism that builds on some of the basic mechanism of rexec. Jeremy From greg at cosc.canterbury.ac.nz Mon Dec 15 17:12:39 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Dec 15 17:12:44 2003 Subject: [Python-Dev] Changing semantics of issubclass when accepting atuple In-Reply-To: <200312150559.hBF5xw616334@c-24-5-183-134.client.comcast.net> Message-ID: <200312152212.hBFMCdh06520@oma.cosc.canterbury.ac.nz> Guido: > while an instance of my_favourite_types would indeed be an instance of > float, 3.14 is not an instance of my_favourite_types! You're right. BrainFartError. :-( Hmmm... If there were an __isinstance__ protocol for types, a suitable object could be constructed... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From janssen at parc.com Mon Dec 15 17:49:50 2003 From: janssen at parc.com (Bill Janssen) Date: Mon Dec 15 17:50:17 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 11:34:33 PST." Message-ID: <03Dec15.144959pst."58611"@synergy1.parc.xerox.com> Ken Manheimer writes: > I'd say there's a legitimate need for > relative imports, and we need some explicit gesture to disambiguate > between a relative and absolute import, one way or the other. I'd agree with Ken here. I think the importance of being able to do relative imports grows as well with the ability to compile Python into foreign frameworks, like Jython or .NET. I'd invent an ugly syntax for relative imports, and let the default be absolute. Bill From barry at python.org Mon Dec 15 17:53:41 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 17:53:48 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <03Dec15.144959pst."58611"@synergy1.parc.xerox.com> References: <03Dec15.144959pst."58611"@synergy1.parc.xerox.com> Message-ID: <1071528820.14604.13.camel@anthem> On Mon, 2003-12-15 at 17:49, Bill Janssen wrote: > I'd agree with Ken here. I think the importance of being able to do > relative imports grows as well with the ability to compile Python into > foreign frameworks, like Jython or .NET. Why? I'd really like to see some concrete use cases for relative imports, other than just saving some typing. -Barry From aahz at pythoncraft.com Mon Dec 15 18:18:49 2003 From: aahz at pythoncraft.com (Aahz) Date: Mon Dec 15 18:18:56 2003 Subject: [Python-Dev] Please tell me where I can submit bug reports.... In-Reply-To: <16350.9342.865204.274114@montanaro.dyndns.org> References: <16349.52054.372100.462135@montanaro.dyndns.org> <76BB1B8C-2F42-11D8-BFA1-0030656C6B9E@webcrunchers.com> <16350.9342.865204.274114@montanaro.dyndns.org> Message-ID: <20031215231849.GB17244@panix.com> On Mon, Dec 15, 2003, Skip Montanaro wrote: > > Sounds like you have cookies disabled. To use SourceForge you need to run > with cookie support turned on. ...and you can't use Lynx, either. :-( -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From lkcl at lkcl.net Mon Dec 15 18:36:39 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Mon Dec 15 18:36:34 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <1071524194.15985.660.camel@localhost.localdomain> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> Message-ID: <20031215233639.GE26055@lkcl.net> On Mon, Dec 15, 2003 at 04:36:35PM -0500, Jeremy Hylton wrote: > One kind of problem is that newer Python features were designed without > taking rexec into account. It's possible for untrusted code to cause > the trusted interpreter to execute its code without restriction using > descriptors. It would be really difficult to reconcile new-style > classes and rexec. Perhaps a worthwhile project, but probably one > accomplished by design a new restriction mechanism that builds on some > of the basic mechanism of rexec. okay. i think the only really sensible way forward is to begin from a sound basis - one that is going to be a big job to add retrospectively, but a simple beginning can be made. proposal: how about building ACLs into the python codebase? the basic operations are read, write, execute and modify_acl the advanced operations are "applies-to-sub-objects" and "automatically-added-to-created-sub-objects". it should be easy to add more operations, both default ones AND user-defined ones. [the advanced operations are based on the new NT 5.0 (aka windows 2000) enhancements to the original VMS security model back from dave cutler's time when he worked for DEC and then he and his team got poached by microsoft to do NT, yes it's a long story.] the basic primitives are __get_acl__(), __set_acl__() and __check_acl__(operation_type). the acl should consist of a dictionary of aces [access control entries] where the name is the function or module name, and the value is an ace object. the objects should contain a __get_ace__() function and a __set_ace__() function. the ace should contain a permission (read, write, execute, modify_acl etc.) and an "ALLOW" or "DENY" qualifier. the algorithm for evaluating an acl has been worked out already, and implemented originally by matthew chapman of the samba team, so code under the GPL already exists [in an NT-like environment which is over-the-top]. the basics of the evaluation algorithm are that you pass in who you are (in this case, that's the function or module name), what type of operation you want to perform (read, write, execute, or combination of these) and the ACL. the aces are evaluated, looking through the dictionary (actually it's going to have to be a list of tuples now that i think of it because the order is important, due to the ALLOW and DENY attributes), checking first that the ACE name matches the "who am i" and then checking what type of permission was requested against the ACE entries. the tricky bit is the "inheritance" permissions, which create "shadow" ACLs or, more specifically, they "throw shadows" down the sub-objects, adding or removing permissions from all sub-objects. in this way, it is possible to implement rexec by simply removing the write permissions at the top level as an "inherited" permission. an empty acl means "anything goes" (including that of "any acl may be added to this object"). an empty acl also means that there is no performance penalty for having acl code in the python codebase. l. From janssen at parc.com Mon Dec 15 18:45:11 2003 From: janssen at parc.com (Bill Janssen) Date: Mon Dec 15 18:45:25 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 14:53:41 PST." <1071528820.14604.13.camel@anthem> Message-ID: <03Dec15.154514pst."58611"@synergy1.parc.xerox.com> > Why? I'd really like to see some concrete use cases for relative > imports, other than just saving some typing. Suppose you've developed a Python top-level application, composed of a package 'foo' containing a number of submodules and subpackages. You'd now like to add this to a Jython application as a library, part of a larger application. It's nice to be able to just add a higher-level package structure, com.mycompany.myapp, to the Python package, to fit in with the style of your application. Yet you don't want to edit the Python code to 'import com.mycompany.myapp.foo.submodule' everywhere; you'd like 'import submodule' or 'import foo.submodule' to continue working. Often the code has to work in both the stand-alone and the federated case. This is just a special case of the federation problem that we've seen for years in distributed computing; previously top-level names get federated when new higher-level constructs are formed. Ken mentioned enterprise programming; it's another case of this. Two companies merge and start merging their code. It's much better to be able to write (in the new higher-level module bletch.py) import foo import bar and have the subpackages under foo and bar continue to work without rewrites, than have to go through all the code under foo and bar and change it to know about the new higher-level bletch namespace. Bill From tismer at tismer.com Mon Dec 15 21:25:57 2003 From: tismer at tismer.com (Christian Tismer) Date: Mon Dec 15 21:26:09 2003 Subject: [Python-Dev] Does anybody really use frame->f_tstate ? Message-ID: <3FDE6D35.3090100@tismer.com> Hi colleagues, this is my second attempt to get rid of the f_tstate field in frames. I need to find every user of this field. What am I talking about? Well, Python always has a thread state variable which is a unique handle to the current thread. This variable is accessed in many places, and there exists a fast macro to get at it. Every executing Python frame also gets a copy on creation. In some cases, this frame->f_tstate field is used, in other cases the current tstate variable is used. If this sounds foreign to you, please stop reading here. ------------------------------------------------------------- I would like to get rid of the frame->f_tstate, and I'm trying to find out if there is a need for it. I don't need it, for Stackless, it is the opposite, it disturbs. There was a small thread about this in June this year, where Guido convinced me that it is possible to create a traceback on a frame that comes from a different thread. http://mail.python.org/pipermail/python-dev/2003-June/036254.html Ok, this is in fact possible, although I don't think anybody has a need for this. My question to all extension writers is this: If you use frame->f_tstate at all, do you use it just because it is handy, or do you want to use it for some other purpose? One purpose could be that you really want to create a traceback on a different than the current thread. I have never seen this, but who knows, so that's why I'm asking the Python world. In most cases, a traceback will be created on a frame that is currently processd or just has been processed. Accessing a frame of a different thread that is being processed might make sense for special debugger cases. My proposal is -------------- a) change semantics of PytraceBack_Here to use the current tstate. b) if such a special purpose exists, create a new function for it. c) if urgent, different needs exist to keep f_tstate, then let's forget about this proposal. Especially for Stackless, I'd be keen of getting rid of this. thanks for input -- chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at tismer.com Mon Dec 15 21:34:58 2003 From: tismer at tismer.com (Christian Tismer) Date: Mon Dec 15 21:34:49 2003 Subject: [Python-Dev] instancemethod_getattro seems to be partially wrong In-Reply-To: <200311240126.hAO1Q9I01704@c-24-5-183-134.client.comcast.net> References: <3FB99A6E.5070000@tismer.com> <200311180604.hAI64Ku02457@c-24-5-183-134.client.comcast.net> <3FBAC6E4.2020202@tismer.com> <200311190133.hAJ1X4J13394@c-24-5-183-134.client.comcast.net> <3FBACC4F.7090404@tismer.com> <200311190507.hAJ575213691@c-24-5-183-134.client.comcast.net> <3FBC3296.1090004@tismer.com> <200311200618.hAK6Ikv23729@c-24-5-183-134.client.comcast.net> <3FBD7C45.3020607@tismer.com> <200311222338.hAMNcnG03504@c-24-5-183-134.client.comcast.net> <3FC046BC.3030500@tismer.com> <200311240126.hAO1Q9I01704@c-24-5-183-134.client.comcast.net> Message-ID: <3FDE6F52.5090104@tismer.com> Guido van Rossum wrote: Hi Guido, this is a little follow-up. I didn't forget about it! ... >>So I'm just saying that pickle.py in wrong in just one place: >> >> reduce = getattr(obj, "__reduce__", None) >> if reduce: >> rv = reduce() >> >>should be: >> >> reduce = getattr(type(obj), "__reduce__", None) >> if reduce: >> rv = reduce(obj) >>""" > > > Right. (That's what I was trying to say, too. :-) > > >>An almost trivial change, although I also had to change copy.py, >>and overall I was unhappy since this extends my patch set to more >>than replacing python2x.dll, but I hope this will become an >>official patch and back-patch. > > > Give it to me baby. (On SF. :-) So here the bad news: After changing the C code to use the new style, and also changing all occourances in Python code, I used this on Zope, and ExtensionClass did not play well, it cannot stand this protocol unchanged. Zope won't start. Therefore, I reverted my changes and now I'm waiting for the day where ExtensionClass is no loger used. (or should I patch *that* complicated stuff as well?) cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From guido at python.org Mon Dec 15 23:25:52 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 23:26:06 2003 Subject: [Python-Dev] instancemethod_getattro seems to be partially wrong In-Reply-To: Your message of "Tue, 16 Dec 2003 03:34:58 +0100." <3FDE6F52.5090104@tismer.com> References: <3FB99A6E.5070000@tismer.com> <200311180604.hAI64Ku02457@c-24-5-183-134.client.comcast.net> <3FBAC6E4.2020202@tismer.com> <200311190133.hAJ1X4J13394@c-24-5-183-134.client.comcast.net> <3FBACC4F.7090404@tismer.com> <200311190507.hAJ575213691@c-24-5-183-134.client.comcast.net> <3FBC3296.1090004@tismer.com> <200311200618.hAK6Ikv23729@c-24-5-183-134.client.comcast.net> <3FBD7C45.3020607@tismer.com> <200311222338.hAMNcnG03504@c-24-5-183-134.client.comcast.net> <3FC046BC.3030500@tismer.com> <200311240126.hAO1Q9I01704@c-24-5-183-134.client.comcast.net> <3FDE6F52.5090104@tismer.com> Message-ID: <200312160425.hBG4PqN18886@c-24-5-183-134.client.comcast.net> > >>So I'm just saying that pickle.py in wrong in just one place: > >> > >> reduce = getattr(obj, "__reduce__", None) > >> if reduce: > >> rv = reduce() > >> > >>should be: > >> > >> reduce = getattr(type(obj), "__reduce__", None) > >> if reduce: > >> rv = reduce(obj) > >>""" > > > > > > Right. (That's what I was trying to say, too. :-) > > > >>An almost trivial change, although I also had to change copy.py, > >>and overall I was unhappy since this extends my patch set to more > >>than replacing python2x.dll, but I hope this will become an > >>official patch and back-patch. > > > > Give it to me baby. (On SF. :-) > > So here the bad news: > After changing the C code to use the new style, and also changing > all occourances in Python code, I used this on Zope, and > ExtensionClass did not play well, it cannot stand this protocol > unchanged. Zope won't start. > > Therefore, I reverted my changes and now I'm waiting for the > day where ExtensionClass is no loger used. > (or should I patch *that* complicated stuff as well?) No, just wait until it dies. I think Jim wants it to die to; the latest Zope version requires Python 2.3.2 but it will take time to retire it, and old Zope versions are persistent. --Guido van Rossum (home page: http://www.python.org/~guido/) From DavidA at ActiveState.com Tue Dec 16 01:49:51 2003 From: DavidA at ActiveState.com (David Ascher) Date: Tue Dec 16 01:53:45 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <03Dec15.154514pst."58611"@synergy1.parc.xerox.com> References: <03Dec15.154514pst."58611"@synergy1.parc.xerox.com> Message-ID: <3FDEAB0F.6010605@ActiveState.com> Bill Janssen wrote: > It's much better to be able to > write (in the new higher-level module bletch.py) > > import foo > import bar > > and have the subpackages under foo and bar continue to work without > rewrites, than have to go through all the code under foo and bar and > change it to know about the new higher-level bletch namespace. Right. Or as someone said: Namespaces are one honking great idea -- let's do more of those! From tjreedy at udel.edu Tue Dec 16 02:00:10 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Tue Dec 16 02:00:17 2003 Subject: [Python-Dev] Re: Pie-thon benchmarks References: <200312151937.hBFJbYX18012@c-24-5-183-134.client.comcast.net> <200312151957.hBFJvWi18151@c-24-5-183-134.client.comcast.net> Message-ID: > You know, I *want* you to win, at least if you can win by a great big > margin. Because then we could switch to Parrot to make Python > faster. I just very much doubt that you'll be able to bat CPython. For this to be thinkable, and for the test to be fair, Parrot must be at least as semantically 'broad' as CPython (ie, handle every possible meaning of every bytecode). So I would include at least the syntax and standard lib test files (- exec and eval tests) as part of the benchmark. This also gets to the 'provably correct' aspect of the rules. Terry J. Reedy From tjreedy at udel.edu Tue Dec 16 02:13:29 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Tue Dec 16 02:13:35 2003 Subject: [Python-Dev] Re: rexec.py unuseable References: <20031215173156.GC25203@lkcl.net> Message-ID: "Luke Kenneth Casson Leighton" wrote in message If you search the pydev summaries (available on the site) for perhaps the past 15 months for 'rexec', you should find several threads on why it was dropped and how it might be replaced. TJR From press at webdevmagazine.co.uk Tue Dec 16 04:02:41 2003 From: press at webdevmagazine.co.uk (Mr.Bogomil Shopov) Date: Tue Dec 16 02:59:19 2003 Subject: [Python-Dev] Web Development Conference Message-ID: <3FDECA31.4030400@webdevmagazine.co.uk> Hello, I am coordinator of a Web Development Conference that will be held in March 2003 in Sofia, Bulgaria. We would like to present to the Conference Python language- new directions and some programmer's tricks. Is there any Bulgarian member in your community or what are your conditions to arrive for this Conf here? Regards, Bogomil Shopov WebDevMagazine phone: +359 (0) 98 505914 Conference 2004 Sofia http://webdevmagazine.co.uk/conf/ CONFIDENTIALITY NOTICE This Email is confidential and may also be privileged. If you are not the intended recipient, please notify the sender IMMEDIATELY; you should not copy the email or use it for any purpose or disclose its contents to any other person. privacy report: privacy@spisanie.com From mwh at python.net Tue Dec 16 05:03:00 2003 From: mwh at python.net (Michael Hudson) Date: Tue Dec 16 05:03:04 2003 Subject: [Python-Dev] Please tell me where I can submit bug reports.... In-Reply-To: <20031215231849.GB17244@panix.com> (aahz@pythoncraft.com's message of "Mon, 15 Dec 2003 18:18:49 -0500") References: <16349.52054.372100.462135@montanaro.dyndns.org> <76BB1B8C-2F42-11D8-BFA1-0030656C6B9E@webcrunchers.com> <16350.9342.865204.274114@montanaro.dyndns.org> <20031215231849.GB17244@panix.com> Message-ID: <2mfzflhygb.fsf@starship.python.net> Aahz writes: > On Mon, Dec 15, 2003, Skip Montanaro wrote: >> >> Sounds like you have cookies disabled. To use SourceForge you need to run >> with cookie support turned on. > > ...and you can't use Lynx, either. :-( It seems that you can, so long as you "attach" /dev/null to the bug report each time you do anything... Cheers, mwh -- surely, somewhere, somehow, in the history of computing, at least one manual has been written that you could at least remotely attempt to consider possibly glancing at. -- Adam Rixey From ncoghlan at iinet.net.au Tue Dec 16 05:32:16 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Tue Dec 16 05:32:27 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <200312151842.hBFIg0K17763@c-24-5-183-134.client.comcast.net> References: <200312151842.hBFIg0K17763@c-24-5-183-134.client.comcast.net> Message-ID: <3FDEDF30.1050904@iinet.net.au> Guido van Rossum wrote: >>I don't understand the desire to totally prevent relative imports. >>It would be quite useful in avoiding growing module-naming problems, >>where you have to be careful about shadowing a global module with >>one in your package. > > I don't see the problem, or I misuderstand what "it" refers to; it > seems you have this backwards if it refers to relative imports. Say > my package P defines a submodule sys. If we require absolute imports, > there is no ambiguity: the submodule sys must be imported as P.sys > while the standard sys module can be imported as simply sys. If absolute imports were to be the only type allowed, then it would seem that the only possible location for naming conflicts is in the _first_ element. So if I wanted to use two different third party modules, both of which have unfortunately chosen the same name for the top-level package, the only way to let them co-exist is to redo _all_ of the imports in one or the other of them. Whereas, if relative pathing is possible, I believe that all I have to do is push them one level down in the package heirarchy (using distinct names that I invent), and neither of them ever even knows about the other's existence. I can get at both of them unambiguously, by using my new top=level names, and neither package even knows that it is no longer starting at the top of the import heirarchy. Or is there some other solution being proposed to this problem, and I just haven't understood it? Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From mcherm at mcherm.com Tue Dec 16 08:37:34 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Tue Dec 16 08:37:54 2003 Subject: [Python-Dev] RE: rexec.py unuseable Message-ID: <1071581854.3fdf0a9eba3ea@mcherm.com> Luke writes: > i think the only really sensible way forward is to begin from a > sound basis... I agree completely. > proposal: how about building ACLs into the python codebase? Actually, I rather prefer the approach that has been mentioned before of trying to use capabilities. See, for instance, the threads on Capabilities found here: http://mail.python.org/pipermail/python-dev/2003-March/thread.html#33854 Of course, that's not trivial to add either, since some way would need to be provided for disabling Python's powerful introspection. > the algorithm for evaluating an acl has been worked out already, > and implemented originally by matthew chapman of the samba team, > so code under the GPL already exists [in an NT-like environment which > is over-the-top]. Unfortunately, code under the GPL is of no use at all for Python which is released with a more flexible (but GPL compatible) liscense. And I'm not quite sure what you mean by an "NT-like environment" being "over-the-top". > an empty acl also means that there is no performance penalty for having > acl code in the python codebase. Sorry, but I simply don't believe this. Really. Think about it a bit, and if you still think you're right, I'll provide some reasons, but I suspect you'll realize that it's simply not true. I *DO* think though, that some amount of slow-down *IS* acceptable (pie or no pie ) if it provided powerful new capabilities like *reliable* restricted execution environments. -- Michael Chermside From lkcl at lkcl.net Tue Dec 16 09:12:07 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 09:12:02 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <1071581854.3fdf0a9eba3ea@mcherm.com> References: <1071581854.3fdf0a9eba3ea@mcherm.com> Message-ID: <20031216141207.GA17021@lkcl.net> On Tue, Dec 16, 2003 at 05:37:34AM -0800, Michael Chermside wrote: > Luke writes: > > i think the only really sensible way forward is to begin from a > > sound basis... > > I agree completely. > > > proposal: how about building ACLs into the python codebase? > > Actually, I rather prefer the approach that has been mentioned before > of trying to use capabilities. capabilities, acls, schmapabilities, same thiiing :) a la "cambridge capability system" [a historic architecture that provided hardware-assisted restrictions on function calls!] > See, for instance, the threads on > Capabilities found here: > http://mail.python.org/pipermail/python-dev/2003-March/thread.html#33854 > Of course, that's not trivial to add either, since some way would > need to be provided for disabling Python's powerful introspection. > > the algorithm for evaluating an acl has been worked out already, > > and implemented originally by matthew chapman of the samba team, > > so code under the GPL already exists [in an NT-like environment which > > is over-the-top]. > > Unfortunately, code under the GPL is of no use at all for Python which > is released with a more flexible (but GPL compatible) liscense. And > I'm not quite sure what you mean by an "NT-like environment" being > "over-the-top". the algorithm is there, for people to examine. i wasn't recommending that people go for complete code reuse of what is in samba (even if licenses were compatible) because the code in samba is a _complete_ reimplementation of the NT acl system _including_ recreating the internals of ACLs and their on-the-wire representation. however, there is a less low-level library which uses ACLs as "blobs", and it is the implementation of the check security access function [name i forget] which is actually a _re_implementation of the exact NT equivalent function that i recommend people to examine. > > an empty acl also means that there is no performance penalty for having > > acl code in the python codebase. > > Sorry, but I simply don't believe this. Really. Think about it a bit, > and if you still think you're right, I'll provide some reasons, but > I suspect you'll realize that it's simply not true. tiny, then. in function calling: if len(acl) == 0 or check_allowed_access(callee_function, acl, EXECUTE_CAPABILITY): proceed... and in file write, again: if len(acl) == 0 or check_allowed_access(callee_function, acl, WRITE_CAPABILITY): proceed... yes, the len(acl) == 0 checks will stack up, but not terribly significantly IMO. if an acl _does_ exist, even if it's one that grants permission to all functions/modules the ability to perform _all_ operations, _then_ you get a performance hit as the check_allowed_access() function needs to "do work". > I *DO* think though, that some amount of slow-down *IS* acceptable > (pie or no pie ) if it provided powerful new capabilities like > *reliable* restricted execution environments. ack! btw yes you're right, it's better off called "capabilities" but the name CCLs - capability control lists - is a bit of a mouthful :) btw, as a first implementation, doing "recursive" capability permissions could be avoided, by simply "copying" permissions down the object tree. as can be imagined, this is rather wasteful of both time and resources, which is why someone at microsoft came up with the concept of "apply-to-sub-object" capabilities. but remember: if all your ACLs (sorry, CCLs) are 0-length, it's not a waste of either time _or_ resources. l. From niemeyer at conectiva.com Tue Dec 16 09:17:46 2003 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue Dec 16 09:20:04 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> Message-ID: <20031216141746.GA3145@burma.localdomain> > But why do you have to give those packages a different full name? > That's the question that I've never seen answered adequately. I have done this many times, so let me try to describe at least one legitimate usage case. A couple of weeks ago I wrote a software which needs a third party package to work. OTOH, since it's a small package, I don't want to force the user to go after the package, even because I'd have to ensure that my code always work with the available version of that package. Thanks to the relative module importing mechanism, solving that is no harder than copying the third party package into my own package namespace. This idea could probably be expressed in some other way, hacking sys.path or whatever, but I belive this is a fairly common pattern, and I vote for introducing a scheme to differ between local/global importing which would not break the current flexibility. -- Gustavo Niemeyer http://niemeyer.net From mcherm at mcherm.com Tue Dec 16 10:13:21 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Tue Dec 16 10:13:41 2003 Subject: [Python-Dev] RE: rexec.py unuseable Message-ID: <1071587601.3fdf2111afc59@mcherm.com> I (Michael) wrote: [Speaking of how to provide restricted execution in Python] > Actually, I rather prefer the approach that has been mentioned before > of trying to use capabilities. > See, for instance, the threads on > Capabilities found here: > http://mail.python.org/pipermail/python-dev/2003-March/thread.html#33854 Luke replied: > capabilities, acls, schmapabilities, same thiiing :) No... they're not. Read the thread I mentioned above, or read this, and some of the other documentation for the language E: http://www.erights.org/elib/capability/ode/ode-capabilities.html Later Luke writes: > btw yes you're right, it's better off called "capabilities" > but the name CCLs - capability control lists - is a bit of > a mouthful :) Again, I mean something different, NOT CCLs, but capabilities. Hmm... how about a quick summary from me (but I'm not an expert). Suppose that had some code which had an object representing a directory, with a method "open(filename, mode='r')" that opened files in the directory. Given this object, you could imagine constructing new objects with more limited capabilities. For instance, you might create a readOnlyDirectory object which had a method "open(filename)" that didn't allow specifying the mode as anything but 'r'. Or you might open a file and then pass a file object with "read()", "write()", "seek()", and other such methods, which would only access that file. So _IF_ the only way to access files were through this object (and that's a BIG if), then you could imagine a world where HAVING and object was equivalent to being able to do something. If a bit of code had access to a read-only-file object then it could read that file, but couldn't write to it, or do anything else with the file system unless it ALSO had access to some OTHER objects. That's capabilities... and it would work for most kinds of restricted resources, not just the file system. The key idea is that HAVING a pointer to the object is equivalent to having the permission to USE that object, and whatever it provides access to. There are several ways to "escape" out of this system. One way is to access some global ability... for instance, you might use the "open()" function from __builtins__. For capabilities to work in Python access to "dangerous" globals would need to be restricted. Another way is to just create an object out of nowhere, or forge a pointer to an object. Fortunately, in Python these are already impossible... pure Python code cannot forge references or create objects without access to the type or class. (C extensions can do anything and are unsafe.) Another way is to access the more powerful object that "empowers" a less powerful one... perhaps using the (supposed to be private) _directory field in the readOnlyfile object. So capabilities in Python are impossible without some sort of private data for objects (this is not a particularly onerous requirement). Yet *another* approach would be to use introspection... for instance, in Python today, given any file object, the class itself ("file") can be obtained via introspection, and then used to create other file objects. Using capabilities in Python would require that "restricted" objects have some more limited form of introspection... perhaps one could only obtain a "fake" class object which had a __equals__ method that simulated being the real class, but which had no __init__ so it couldn't be used to create new instances. At any rate, you get the idea. Capabilities are possible in Python only if some sort of "restricted mode" is created, which restricts access to some built-in abilities and which creates "restricted" objects with some private data and limited introspection. But IF you had these things (and they're NOT trivial), then capabilities may be a conceptually more elegant approach than ACLs, lleading to more elegant programs. -- Michael Chermside From tjreedy at udel.edu Tue Dec 16 11:11:21 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Tue Dec 16 11:11:30 2003 Subject: [Python-Dev] Re: Web Development Conference References: <3FDECA31.4030400@webdevmagazine.co.uk> Message-ID: "Mr.Bogomil Shopov" wrote in message news:3FDECA31.4030400@webdevmagazine.co.uk... > Hello, > I am coordinator of a Web Development Conference that will be held > in March 2003 in Sofia, Bulgaria. We would like to present to the presumably 2004 > Conference Python language- new directions and some programmer's > tricks. Is there any Bulgarian member in your community or what are > your conditions to arrive for this Conf here? This 'Python development' list has a relatively restricted readership. If you do not get a positive response here, try reposting on comp.lang.python or equivalent mainling list. Good luck with your conference. TJR From lkcl at lkcl.net Tue Dec 16 11:16:03 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 11:16:11 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <1071587601.3fdf2111afc59@mcherm.com> References: <1071587601.3fdf2111afc59@mcherm.com> Message-ID: <20031216161603.GG17021@lkcl.net> On Tue, Dec 16, 2003 at 07:13:21AM -0800, Michael Chermside wrote: > I (Michael) wrote: > [Speaking of how to provide restricted execution in Python] > > Actually, I rather prefer the approach that has been mentioned before > > of trying to use capabilities. > > See, for instance, the threads on > > Capabilities found here: > > http://mail.python.org/pipermail/python-dev/2003-March/thread.html#33854 ta. > Luke replied: > > capabilities, acls, schmapabilities, same thiiing :) > > No... they're not. Read the thread I mentioned above, or read this, > and some of the other documentation for the language E: > > http://www.erights.org/elib/capability/ode/ode-capabilities.html no offense intended: i'll read that later, i'm running out of time. without going into too many definitions, consider what i am advocating to be _like_ an access control list but instead to be a capabilities control list, instead. a "capabilities list", where it's a list ordered on a per-caller-function-name basis [with a special wildcard function name called "absolutely everything"] > > Later Luke writes: > > btw yes you're right, it's better off called "capabilities" > > but the name CCLs - capability control lists - is a bit of > > a mouthful :) > > Again, I mean something different, NOT CCLs, but capabilities. anyone have a one-para summary of the difference between capabilities and access control lists? even if it's "access control lists are lists of capabilities" which i don't know if that's true. access control lists are typically made on a per-user basis, but what i am recommending in _this_ case is that a "user" be considered to be a *function*. so maybe i _do_ mean access control list and should stick to my guns, here :) so. is the difference between capabilities and access control lists simply that capabilities lists restrict caller-function rights and access control lists restrict user rights? > Hmm... how about a quick summary from me (but I'm not an expert). > > Suppose that had some code which had an object representing a > directory, with a method "open(filename, mode='r')" that opened > files in the directory. Given this object, you could imagine > constructing new objects with more limited capabilities. For > instance, you might create a readOnlyDirectory object which had > a method "open(filename)" that didn't allow specifying the mode > as anything but 'r'. Or you might open a file and then pass a > file object with "read()", "write()", "seek()", and other such > methods, which would only access that file. [i'll have to read this in more depth later, i'm out of time, sorry] > So _IF_ the only way to access files were through this object (and > that's a BIG if), then you could imagine a world where HAVING and > object was equivalent to being able to do something. If a bit of > code had access to a read-only-file object then it could read that > file, but couldn't write to it, or do anything else with the file > system unless it ALSO had access to some OTHER objects. That's > capabilities... and it would work for most kinds of restricted > resources, not just the file system. The key idea is that HAVING > a pointer to the object is equivalent to having the permission to > USE that object, and whatever it provides access to. [again, i'll have to read this in more depth later, i'm out of time, sorry] > There are several ways to "escape" out of this system. One way is > to access some global ability... for instance, you might use the > "open()" function from __builtins__. i would expect __builtins__.open() to have _its_ own mmm... capabilities list, and _if_ that list contained a permission for the restricted function to "execute" it, then _yes_ it would be allowed, but otherwise no, definitely not. therefore, the author of the [new] rexec.py module should simply be a matter of creating the right ACLs, and applying them. in the case of __builtins__ it could be a matter of just applying a capabilities list of one item: [("all functions and modules", DENY, "write, apply-to-sub-objects")] and that would be _it_! all functions in __builtins__ would be dealt with, and restricted! of course, i don't believe it will be _quite_ that simple, but it might. > For capabilities to work in > Python access to "dangerous" globals would need to be restricted. simple: apply a capabilities list that denies dangerous actions, on all such "dangerous" globals. OH! one important "permission" is "change acl" of course, which must be "DENIED"! > Another way is to just create an object out of nowhere, or forge > a pointer to an object. Fortunately, in Python these are already > impossible... pure Python code cannot forge references or create > objects without access to the type or class. (C extensions can > do anything and are unsafe.) that is why i mentioned about "create object" permissions [capabilities]. if there is a permission/capability whereby the capabilities list of a parent object is "inherited" when a create object action is carried out, the problem you describe is alleviated. _at_ object create time (and object create itself could be a separate permission / capability, granted on a per-function and per-module basis like everything else), the capabilities are examined. what happens is that _if_ the "inherit capabilities" flag is set, then the newly created object receives a COPY of the parent object's capabilities list. ta-da, problem solved. this is a _normal_ bog-standard approach that is used in the windows nt security model, and has been for over twenty years, now, and if you count the VAX/VMS history as well, a lot longer than twenty years. > Another way is to access the more powerful object that "empowers" > a less powerful one... perhaps using the (supposed to be private) > _directory field in the readOnlyfile object. So capabilities in > Python are impossible without some sort of private data for > objects (this is not a particularly onerous requirement). Yet > *another* approach would be to use introspection... for instance, > in Python today, given any file object, the class itself ("file") > can be obtained via introspection, and then used to create other > file objects. Using capabilities in Python would require that > "restricted" objects have some more limited form of introspection... > perhaps one could only obtain a "fake" class object which had > a __equals__ method that simulated being the real class, but > which had no __init__ so it couldn't be used to create new > instances. i don't quite follow, even after reading the capabilities thread, what introspection is. but let me try to clarify so you can correct me if necessary: introspection is the ability to go via the __XXXXX__ functions etc. including the __class__ stuff, yes? well, if i follow you correctly, you simply put a capabilities list on all those things, or you put one on the entire object. or better yet, you put an "inherited" one on the class, such that _any_ object created will receive restricted capabilities. etc. > At any rate, you get the idea. Capabilities are possible in Python > only if some sort of "restricted mode" is created, which restricts > access to some built-in abilities and which creates "restricted" > objects with some private data and limited introspection. > But IF > you had these things (and they're NOT trivial), then capabilities > may be a conceptually more elegant approach than ACLs, lleading to > more elegant programs. i believe that if you understand what i am suggesting, and i hope that filling in some examples above, i believe that what i am suggesting covers your concerns. the capabilities lists that i am recommending can be applied on a per-function and per-object basis and can be inherited. in this way, useful restrictions can be made to achieve the expected results. ... i didn't say, however, that it wouldn't require quite a lot of thought about _which_ functions to apply restricted capabilities lists to! starting, of course, with the "nothing goes" one, and repeatedly attempting to run code. l. From guido at python.org Tue Dec 16 11:21:21 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 11:21:27 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Sun, 14 Dec 2003 14:27:52 PST." <200312142227.hBEMRr915819@c-24-5-183-134.client.comcast.net> References: <200312141723.hBEHNvg15553@c-24-5-183-134.client.comcast.net> <200312142227.hBEMRr915819@c-24-5-183-134.client.comcast.net> Message-ID: <200312161621.hBGGLLs19795@c-24-5-183-134.client.comcast.net> Dan, I have one more clarification request. I assume you will support import (of Python modules); that shouldn't be a problem right? The explicit ban on extensions and the insistence of the full builtins would suggest that import is okay. I'm not assuming the standard library (beyond the builtins) is available, but I *am* assuming I can split the benchmark into multiple files. --Guido van Rossum (home page: http://www.python.org/~guido/) From raymond.hettinger at verizon.net Tue Dec 16 12:30:46 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue Dec 16 12:31:02 2003 Subject: [Python-Dev] Filename suffix default for command line Message-ID: <003b01c3c3fa$57a6dee0$f1bd958d@oemcomputer> Would there be any disadvantage to checking for filename.py whenever filename is not found so that "super.py" could be invoked with either "python super.py" or "python super"? Raymond Hettinger -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031216/2ff80142/attachment.html From guido at python.org Tue Dec 16 12:41:06 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 12:41:11 2003 Subject: [Python-Dev] Filename suffix default for command line In-Reply-To: Your message of "Tue, 16 Dec 2003 12:30:46 EST." <003b01c3c3fa$57a6dee0$f1bd958d@oemcomputer> References: <003b01c3c3fa$57a6dee0$f1bd958d@oemcomputer> Message-ID: <200312161741.hBGHf6w20055@c-24-5-183-134.client.comcast.net> > Would there be any disadvantage to checking for filename.py whenever > filename is not found so that "super.py" could be invoked with either > "python super.py" or "python super"? It feels horribly DOS-like. Why? To save typing three characters? Why not rename the file then? Would "python super.py" also look for super.py.py if super.py didn't exist? What would the error message say? --Guido van Rossum (home page: http://www.python.org/~guido/) From DavidA at ActiveState.com Tue Dec 16 12:49:17 2003 From: DavidA at ActiveState.com (David Ascher) Date: Tue Dec 16 12:52:24 2003 Subject: [Python-Dev] instancemethod_getattro seems to be partially wrong In-Reply-To: <200312160425.hBG4PqN18886@c-24-5-183-134.client.comcast.net> References: <3FB99A6E.5070000@tismer.com> <200311180604.hAI64Ku02457@c-24-5-183-134.client.comcast.net> <3FBAC6E4.2020202@tismer.com> <200311190133.hAJ1X4J13394@c-24-5-183-134.client.comcast.net> <3FBACC4F.7090404@tismer.com> <200311190507.hAJ575213691@c-24-5-183-134.client.comcast.net> <3FBC3296.1090004@tismer.com> <200311200618.hAK6Ikv23729@c-24-5-183-134.client.comcast.net> <3FBD7C45.3020607@tismer.com> <200311222338.hAMNcnG03504@c-24-5-183-134.client.comcast.net> <3FC046BC.3030500@tismer.com> <200311240126.hAO1Q9I01704@c-24-5-183-134.client.comcast.net> <3FDE6F52.5090104@tismer.com> <200312160425.hBG4PqN18886@c-24-5-183-134.client.comcast.net> Message-ID: <3FDF459D.30507@ActiveState.com> Guido van Rossum wrote: > the latest Zope version requires Python 2.3.2 but it will take time to > >retire it, and old Zope versions are persistent. > > I thought that was a *feature* of Zope and the whole ZODB thing. --da From dan at sidhe.org Tue Dec 16 13:12:13 2003 From: dan at sidhe.org (Dan Sugalski) Date: Tue Dec 16 13:12:23 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <200312161621.hBGGLLs19795@c-24-5-183-134.client.comcast.net> References: <200312141723.hBEHNvg15553@c-24-5-183-134.client.comcast.net> <200312142227.hBEMRr915819@c-24-5-183-134.client.comcast.net> <200312161621.hBGGLLs19795@c-24-5-183-134.client.comcast.net> Message-ID: At 8:21 AM -0800 12/16/03, Guido van Rossum wrote: >Dan, I have one more clarification request. > >I assume you will support import (of Python modules); that shouldn't >be a problem right? The explicit ban on extensions and the insistence >of the full builtins would suggest that import is okay. I'm not >assuming the standard library (beyond the builtins) is available, but >I *am* assuming I can split the benchmark into multiple files. Oh, absolutely. Loading in external bytecode files are fine. -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido at python.org Tue Dec 16 13:26:59 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 13:27:05 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Tue, 16 Dec 2003 12:17:46 -0200." <20031216141746.GA3145@burma.localdomain> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> Message-ID: <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> A lot people have presented a good case for relative imports. Nobody has argued to keep the status quo (where imports are ambiguous about whether they are meant to be absolute or relative). So I suggest that in 2.4, we introduce the leading dot notation for relative import, while still allowing relative import without a leading dot. In 2.5 we can start warning about relative import without a leading dot (although that will undoubtedly get complaints from folks who have code that needs to work with 2.3 still). In 3.0 we can retire ambiguous import. The use case for multiple dots should be obvious: inside a highly structured package, modules inside one subpackage must have a way to do relative import from another subpackage of the same parent package. There is the remaining issue of what exactly the syntax would be. I propose to extend the from clause to allow one or more dots before the dotted name, and to make the dotted name optional if at least one leading dot is found. I propose not to change from-less import. Examples: from .foo import bar from .foo.bar import xxx from . import foobar as barfoo from ..foo.bar import * from ... import foobar, barfoo Grammar change (see Grammar/Grammar): dotted_name: NAME ('.' NAME)* | '.'+ [NAME ('.' NAME)*] --Guido van Rossum (home page: http://www.python.org/~guido/) From gerrit at nl.linux.org Tue Dec 16 13:29:12 2003 From: gerrit at nl.linux.org (Gerrit Holl) Date: Tue Dec 16 13:29:47 2003 Subject: [Python-Dev] Filename suffix default for command line In-Reply-To: <003b01c3c3fa$57a6dee0$f1bd958d@oemcomputer> References: <003b01c3c3fa$57a6dee0$f1bd958d@oemcomputer> Message-ID: <20031216182912.GA4204@nl.linux.org> Raymond Hettinger wrote: > Would there be any disadvantage to checking for filename.py whenever > filename is not found so that "super.py" could be invoked with either > "python super.py" or "python super"? How about super.pyc? What if both super.pyc and super.py exist but are not equal in effect? I think the idea is not necesarry. I think most people have configured to do commandline completion, so 'python supe' would probably achieve the same effect with the same amount of keystrokes. The disadvantage is, in my opinion, sideways: people get used to it, and it is not a good habit, because it's called super.py, not super. I don't like this proposal, I'd give it a -0 in case anyone is interested. yours, Gerrit. -- 202. If any one strike the body of a man higher in rank than he, he shall receive sixty blows with an ox-whip in public. -- 1780 BC, Hammurabi, Code of Law -- Asperger's Syndrome - a personal approach: http://people.nl.linux.org/~gerrit/english/ From guido at python.org Tue Dec 16 13:30:43 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 13:30:48 2003 Subject: [Python-Dev] Re: Pie-thon benchmarks In-Reply-To: Your message of "Tue, 16 Dec 2003 02:00:10 EST." References: <200312151937.hBFJbYX18012@c-24-5-183-134.client.comcast.net> <200312151957.hBFJvWi18151@c-24-5-183-134.client.comcast.net> Message-ID: <200312161830.hBGIUhE20354@c-24-5-183-134.client.comcast.net> > > You know, I *want* you to win, at least if you can win by a great > > big margin. Because then we could switch to Parrot to make Python > > faster. I just very much doubt that you'll be able to beat > > CPython. > > For this to be thinkable, and for the test to be > fair, Parrot must be at least as semantically > 'broad' as CPython (ie, handle every possible > meaning of every bytecode). So I would include at > least the syntax and standard lib test files (- > exec and eval tests) as part of the benchmark. > This also gets to the 'provably correct' aspect of > the rules. Right. I don't want to include the entire test suite, but I do want to make sure the benchmark visits most dark corners of the language: metaclasses, dynamically changing operator overloading, and the like. I trust that Dan can write a VM that executes integer operations faster than anyone, but that's not the point of Python. --Guido van Rossum (home page: http://www.python.org/~guido/) From jacobs at penguin.theopalgroup.com Tue Dec 16 13:35:14 2003 From: jacobs at penguin.theopalgroup.com (Kevin Jacobs) Date: Tue Dec 16 13:35:25 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> Message-ID: On Tue, 16 Dec 2003, Guido van Rossum wrote: > There is the remaining issue of what exactly the syntax would be. I > propose to extend the from clause to allow one or more dots before the > dotted name, and to make the dotted name optional if at least one > leading dot is found. I propose not to change from-less import. > Examples: > > from .foo import bar > from .foo.bar import xxx > from . import foobar as barfoo > from ..foo.bar import * > from ... import foobar, barfoo > > Grammar change (see Grammar/Grammar): > > dotted_name: NAME ('.' NAME)* | '.'+ [NAME ('.' NAME)*] On a related and minor note, can we please expand the grammar to allow from...import statements to continue to the next line after a trailing comma? For those of us with large, complex, and componentized projects, it would be very nice. e.g.: from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, normalize, row_ops, summary instead of: from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, \ normalize, row_ops, \ summary This has been a minor nit that bothers just under the threshold of fixing it myself. ;) Thanks, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (440) 871-6725 x 19 E-mail: jacobs@theopalgroup.com Fax: (440) 871-6722 WWW: http://www.theopalgroup.com/ From lkcl at lkcl.net Tue Dec 16 13:35:41 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 13:35:47 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <1071587601.3fdf2111afc59@mcherm.com> References: <1071587601.3fdf2111afc59@mcherm.com> Message-ID: <20031216183541.GH17021@lkcl.net> On Tue, Dec 16, 2003 at 07:13:21AM -0800, Michael Chermside wrote: > Suppose that had some code which had an object representing a > directory, with a method "open(filename, mode='r')" that opened > files in the directory. Given this object, you could imagine > constructing new objects with more limited capabilities. yep. in a very simple manner, with capabilities lists. > For > instance, you might create a readOnlyDirectory object which had > a method "open(filename)" that didn't allow specifying the mode > as anything but 'r'. Or you might open a file and then pass a > file object with "read()", "write()", "seek()", and other such > methods, which would only access that file. this could be achieved very simply by setting a CCL with "deny everything" permissions on an object that, ordinarily, had a full set of functions with NO restrictions, but leaving read() alone. _and_ also, importantly, adding a CCL onto the __builtins__ etc as already discussed to restrict bypassing. > So _IF_ the only way to access files were through this object (and > that's a BIG if), then you could imagine a world where HAVING and > object was equivalent to being able to do something. If a bit of > code had access to a read-only-file object then it could read that > file, but couldn't write to it, or do anything else with the file > system unless it ALSO had access to some OTHER objects. yep. > That's > capabilities... and it would work for most kinds of restricted > resources, not just the file system. The key idea is that HAVING > a pointer to the object is equivalent to having the permission to > USE that object, and whatever it provides access to. ah... that's where you and i differ, based on my experience of ACLs in NT. i envisage a separate "execute" permission from a "read" and "write" permission, a bit like directory permissions on a POSIX filesystem. under such a scheme where "execute" capability permission exists as well as "read" and "write", simply having access to an object does NOT guarantee you any rights to CALL that object, because if it doesn't have "execute" permission you can't execute [viz, call] it! ... but what you _can_ do is pass that object to another function which _does_ have execute permissions. how the hell is _that_ achieved, i hear you say??? well imagine a CCL as follows: [('totally_restricted_function, 'DENY', 'All permissions'), ('slightly_less_restricted_function', 'ALLOW', 'execute and read')] and this CCL is applied to the above-mentioned readOnlyDirectory object. if totally_restricted_function() calls slightly_less_restricted_function(), passing a readOnlyDirectory object to it, then according to the CCL above, ONLY slightly_less_restricted_function() may call functions in that object. cute, huh? l. From FBatista at uniFON.com.ar Tue Dec 16 13:37:39 2003 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Tue Dec 16 13:39:03 2003 Subject: [Python-Dev] Relative import Message-ID: Guido van Rossum wrote: #- There is the remaining issue of what exactly the syntax would be. I #- propose to extend the from clause to allow one or more dots #- before the #- dotted name, and to make the dotted name optional if at least one #- leading dot is found. I propose not to change from-less import. Don't know the restrictions, as never used the package feature in modules, but seems natural to me to use the same syntax that in a unix filesystem (maybe without the '/'?). This way, you can reference a module in your level, in an upper level, etc... . Facundo From guido at python.org Tue Dec 16 13:43:46 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 13:43:52 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Tue, 16 Dec 2003 13:35:14 EST." References: Message-ID: <200312161843.hBGIhkE20496@c-24-5-183-134.client.comcast.net> > On a related and minor note, can we please expand the grammar to allow > from...import statements to continue to the next line after a trailing > comma? For those of us with large, complex, and componentized projects, it > would be very nice. e.g.: > > from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, > normalize, row_ops, > summary > > instead of: > > from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, \ > normalize, row_ops, \ > summary > > > This has been a minor nit that bothers just under the threshold of fixing it > myself. ;) I guess this could be implemented by allowing [NEWLINE] after the comma in the grammar. But I'm -0 on this change; I fear that someone accidentally adding a trailing comma (or leaving it in after some rearrangements) will be very puzzled at the syntax error that comes somewhere in the middle of the next statement: from foo import bar, bufsize = 12 ^syntax error --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik at pythonware.com Tue Dec 16 14:36:36 2003 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue Dec 16 14:37:00 2003 Subject: [Python-Dev] Re: Relative import References: <200312161843.hBGIhkE20496@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum wrote: > > from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, > > normalize, row_ops, > > summary > > > > instead of: > > > > from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, \ > > normalize, row_ops, \ > > summary > > > > > > This has been a minor nit that bothers just under the threshold of fixing it > > myself. ;) > > I guess this could be implemented by allowing [NEWLINE] after the > comma in the grammar. > > But I'm -0 on this change; I fear that someone accidentally adding a > trailing comma (or leaving it in after some rearrangements) will be > very puzzled at the syntax error that comes somewhere in the middle of > the next statement: > > from foo import bar, > > bufsize = 12 > ^syntax error you could require parentheses: from OPAL.Modules.Financial.Reports.COGS import ( generate, webOptions, normalize, row_ops, summary ) (after all, from-import is a kind of assignment, and "a, b = c" is the same thing as "(a, b) = c"...) From martin at v.loewis.de Tue Dec 16 14:39:37 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Dec 16 14:41:07 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031215233639.GE26055@lkcl.net> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> Message-ID: Luke Kenneth Casson Leighton writes: > i think the only really sensible way forward is to begin from a > sound basis - one that is going to be a big job to add retrospectively, > but a simple beginning can be made. > > proposal: how about building ACLs into the python codebase? I fail to see how ACLs are a sound basis to solve the problem that rexec solves. ACLs, in my view, are a sound basis to solve a different problem (that of different identities accessing the same resources). Also, it seems that nowhere in your proposal you state how ACLs should be integrated into Python: I.e. what objects are protected by ACLs, and how do you classify the various actions in a Python program as read/write/execute/modify_acl? E.g. given 3 * 4 Is that read, write, execute, and which ACL(s) is(are) considered? Regards, Martin From lkcl at lkcl.net Tue Dec 16 14:52:00 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 14:53:16 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> Message-ID: <20031216195200.GI17021@lkcl.net> On Tue, Dec 16, 2003 at 08:39:37PM +0100, Martin v. L?wis wrote: > Luke Kenneth Casson Leighton writes: > > > i think the only really sensible way forward is to begin from a > > sound basis - one that is going to be a big job to add retrospectively, > > but a simple beginning can be made. > > > > proposal: how about building ACLs into the python codebase? > > I fail to see how ACLs are a sound basis to solve the problem that > rexec solves. ACLs, in my view, are a sound basis to solve a different > problem (that of different identities accessing the same resources). substitute "identity" for "calling object" i.e. function and i believe that there is a viable proposal. l. From lkcl at lkcl.net Tue Dec 16 14:55:25 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 14:57:22 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> Message-ID: <20031216195525.GJ17021@lkcl.net> On Tue, Dec 16, 2003 at 08:39:37PM +0100, Martin v. L?wis wrote: > Luke Kenneth Casson Leighton writes: > > > i think the only really sensible way forward is to begin from a > > sound basis - one that is going to be a big job to add retrospectively, > > but a simple beginning can be made. > > > > proposal: how about building ACLs into the python codebase? > > I fail to see how ACLs are a sound basis to solve the problem that > rexec solves. ACLs, in my view, are a sound basis to solve a different > problem (that of different identities accessing the same resources). > > Also, it seems that nowhere in your proposal you state how ACLs should > be integrated into Python: I.e. what objects are protected by ACLs, all objects - if an acl is non-null. > and how do you classify the various actions in a Python program as > read/write/execute/modify_acl? E.g. given > 3 * 4 > > Is that read, write, execute, and which ACL(s) is(are) considered? execute, and execute only, because there's no I/O involved. on the multiply operation. but _only_ if there's an actual ACL set _on_ the multiply function. if there's no acl set on the multiply function, there's no restrictions on the multiply function. l. From ws-news at gmx.at Tue Dec 16 14:53:10 2003 From: ws-news at gmx.at (Werner Schiendl) Date: Tue Dec 16 14:57:56 2003 Subject: [Python-Dev] Re: Relative import References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> Message-ID: Hi, "Guido van Rossum" wrote > A lot people have presented a good case for relative imports. Nobody > has argued to keep the status quo (where imports are ambiguous about > whether they are meant to be absolute or relative). So I suggest that > in 2.4, we introduce the leading dot notation for relative import, > while still allowing relative import without a leading dot. I'm +1 on it, I'd only wish an additional from __future__ import strict_import (or whatever seems a better name) to allow a module to indicate that every not explicitly relative import should be absolute. This would allow module authors to be explicit from the start and it is also a safety net against accidentially adding a relative module with the name of some absolute one. best regards Werner From lkcl at lkcl.net Tue Dec 16 14:58:56 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 14:59:30 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> Message-ID: <20031216195856.GK17021@lkcl.net> On Tue, Dec 16, 2003 at 08:39:37PM +0100, Martin v. L?wis wrote: > Luke Kenneth Casson Leighton writes: > > > i think the only really sensible way forward is to begin from a > > sound basis - one that is going to be a big job to add retrospectively, > > but a simple beginning can be made. > > > > proposal: how about building ACLs into the python codebase? > > I fail to see how ACLs are a sound basis to solve the problem that in the initial message to the list, i confused the concept of acls with "capabilities". sorry about that. l. From edloper at gradient.cis.upenn.edu Tue Dec 16 15:17:49 2003 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Tue Dec 16 15:18:07 2003 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 5, Issue 57 In-Reply-To: References: Message-ID: Guido said: > There is the remaining issue of what exactly the syntax would be. I > propose to extend the from clause to allow one or more dots before the > dotted name, and to make the dotted name optional if at least one > leading dot is found. I propose not to change from-less import. What's the motivation for restricting it to from-imports? I suspect that this will cause confusion, and it doesn't feel self-consistent to me: why can you call a module ".foo" in one context, and not in another? In particular, I think that if people see that you can do: from .foo import * Then they'll assume that they can also do: import .foo With the obvious semantics, i.e. equivalence to: from . import foo What do we gain by not extending the syntax to from-less imports? -Edward From martin at v.loewis.de Tue Dec 16 15:14:40 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Dec 16 15:20:46 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031216195525.GJ17021@lkcl.net> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> Message-ID: Luke Kenneth Casson Leighton writes: > > Also, it seems that nowhere in your proposal you state how ACLs should > > be integrated into Python: I.e. what objects are protected by ACLs, > > all objects - if an acl is non-null. Does this include functions and bound and unbound methods? > > 3 * 4 > > > > Is that read, write, execute, and which ACL(s) is(are) considered? > > execute, and execute only, because there's no I/O involved. > > on the multiply operation. So what operations require read or write access? > but _only_ if there's an actual ACL set _on_ the multiply function. > > if there's no acl set on the multiply function, there's no > restrictions on the multiply function. So ACLs on the objects 3 and 4 would be irrelevant? Generalizing this, given a class Foo, with a method bar(), and two instances foo1 and foo2: foo1 = Foo("/etc/passwd") foo2 = Foo("/tmp/nonexistant.yet") and considering the calls foo1.bar() and foo2.bar(): Given that *only* the ACL on the operation bar matters: Does that mean that the calls either both succeed or both fail, for a given caller? Regards, Martin From martin at v.loewis.de Tue Dec 16 15:08:37 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Dec 16 15:31:30 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031216195200.GI17021@lkcl.net> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195200.GI17021@lkcl.net> Message-ID: Luke Kenneth Casson Leighton writes: > > I fail to see how ACLs are a sound basis to solve the problem that > > rexec solves. ACLs, in my view, are a sound basis to solve a different > > problem (that of different identities accessing the same resources). > > substitute "identity" for "calling object" i.e. function and i > believe that there is a viable proposal. I don't think so. Suppose I have an object A protected by ACLs. Now I write a new method def doit(): A.doit() Question 1. What is the calling object? Question 2. Assuming it is "the doit function", and assuming the doit function is not listed in the ACL: Will the call be executed? If yes: What is the purpose of the ACL then? If no: I believe the proposal is not usable in practice. For example, what should be the ACL on calling open()? Regards, Martin From ws-news at gmx.at Tue Dec 16 16:02:30 2003 From: ws-news at gmx.at (Werner Schiendl) Date: Tue Dec 16 16:03:00 2003 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 5, Issue 57 References: Message-ID: Hi, "Edward Loper" wrote > > Guido said: > > [...] I propose not to change from-less import. > > What's the motivation for restricting it to from-imports? I suspect > that this will cause confusion, and it doesn't feel self-consistent to > me: why can you call a module ".foo" in one context, and not in > another? In particular, I think that if people see that you can do: > > from .foo import * > > Then they'll assume that they can also do: > > import .foo The problem is probably, to what name should the module be bound? Variable names are not allowed to contain a leading dot. But it would be possible to say: import .foo as my_foo import foo and access both through modules through the name they're bound to. Therefore I'd propose to allow relative from-less imports, but require them to always use the "as" clause. regards Werner From Jack.Jansen at cwi.nl Tue Dec 16 16:04:40 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Tue Dec 16 16:04:46 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <20031216161603.GG17021@lkcl.net> References: <1071587601.3fdf2111afc59@mcherm.com> <20031216161603.GG17021@lkcl.net> Message-ID: <76A3670A-300B-11D8-B3E2-000A27B19B96@cwi.nl> On 16-dec-03, at 17:16, Luke Kenneth Casson Leighton wrote: >> Luke replied: >>> capabilities, acls, schmapabilities, same thiiing :) >> >> No... they're not. Read the thread I mentioned above, or read this, >> and some of the other documentation for the language E: >> >> http://www.erights.org/elib/capability/ode/ode-capabilities.html > > no offense intended: i'll read that later, i'm running out of time. > > without going into too many definitions, consider what i am advocating > to be _like_ an access control list but instead to be a capabilities > control list, instead. The distinction between capabilities and ACLs is really important, because they are almost each others opposite. With capabilities you have an (unforgable) right to do something and no-one cares about your identity, with ACLs you have an unforgable identity which is checked against the ACL. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From Jack.Jansen at cwi.nl Tue Dec 16 16:09:59 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Tue Dec 16 16:10:09 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> Message-ID: <35391BC0-300C-11D8-B3E2-000A27B19B96@cwi.nl> On 16-dec-03, at 19:26, Guido van Rossum wrote: > There is the remaining issue of what exactly the syntax would be. I > propose to extend the from clause to allow one or more dots before the > dotted name, and to make the dotted name optional if at least one > leading dot is found. I propose not to change from-less import. > Examples: > > from .foo import bar > from .foo.bar import xxx > from . import foobar as barfoo > from ..foo.bar import * > from ... import foobar, barfoo Is "from . import *" allowed? Whenever I start to think about it I tend to go into oscillation (yes because within package foo its the same as "from foo import *" on the outside; no because "import *" isn't allowed today either). -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From lkcl at lkcl.net Tue Dec 16 16:23:12 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 16:23:14 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> Message-ID: <20031216212312.GL17021@lkcl.net> On Tue, Dec 16, 2003 at 09:14:40PM +0100, Martin v. L?wis wrote: > Luke Kenneth Casson Leighton writes: > > > > Also, it seems that nowhere in your proposal you state how ACLs should > > > be integrated into Python: I.e. what objects are protected by ACLs, > > > > all objects - if an acl is non-null. > > Does this include functions and bound and unbound methods? if it's a function, it can potentially have an ACL (or a better description is CCL - capabilities control list) added, and that CCL applies to all sub-objects of the function. same with objects, classes - everything. > > > 3 * 4 > > > > > > Is that read, write, execute, and which ACL(s) is(are) considered? > > > > execute, and execute only, because there's no I/O involved. > > > > on the multiply operation. > > So what operations require read or write access? short answer: all of them. longer answer: just like with exceptions, that has to be classified according to conventions that are yet to be decided. i propose that, for example, a capability be created to control _file_ read access and _file_ write access, with large provisos on the documentation attached to these two capabilities that ensure they are distinguished from the right to _modify_ an object. basically it's all conventions, just like exceptions are conventions. all i can do is recommend a framework and some guidelines on what conventions could be fitted over that framework. heck, it may even be worthwhile defining a low-level object, named a Capability, just like an Exception is defined, and then expect people to create their own, and recommend the creation of certain Capabilities that are inherited from the Capability base class. > > but _only_ if there's an actual ACL set _on_ the multiply function. > > > > if there's no acl set on the multiply function, there's no > > restrictions on the multiply function. > > So ACLs on the objects 3 and 4 would be irrelevant? well... actually.... no :) but are such ACLs relevant? can you do 3 += 5? no you can't: python throws up a SyntaxError: can't assign to a literal. so, whilst the answer is yes, you _can_ associate an ACL [or better-named, a CCL] with the object "3" and the object "4", i would be surprised if there were any situations where they got... hang on... >>> dir(3) ['__abs__', '__add__', '__and__', '__class__', '__cmp__', '__coerce__', '__delattr__', '__div__', '__divmod__', '__doc__', '__float__', '__floordiv__', '__getattribute__', '__getnewargs__', '__hash__', '__hex__', '__init__', '__int__', '__invert__', '__long__', '__lshift__', '__mod__', '__mul__', '__neg__', '__new__', '__nonzero__', '__oct__', '__or__', '__pos__', '__pow__', '__radd__', '__rand__', '__rdiv__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__str__', '__sub__', '__truediv__', '__xor__'] yes, i believe it _might_ be relevant to set a CCL on an object "3" because someone might want to restrict how... oh, i dunno... mmm. __setattr__ gets used. >>> 3.__xor__ ^ SyntaxError: invalid syntax oh. maybe not :) however, a UserLong or a UserInt is a different matter. [btw, change of subject: why is there no __eq__, no __gt__, no __le__ in that list above?] > Generalizing this, given a class Foo, with a method bar(), and two > instances foo1 and foo2: > > foo1 = Foo("/etc/passwd") > foo2 = Foo("/tmp/nonexistant.yet") > > and considering the calls foo1.bar() and foo2.bar(): > > Given that *only* the ACL on the operation bar matters: Does that mean > that the calls either both succeed or both fail, for a given caller? i'm not entirely sure what you mean, but let's assume that objects (and their instances!) all have an __set_ccl__() function. let's also define a CCL: deny_ccl = [('all functions', DENY, 'execute')] then, let's try this: foo2 = Foo("/etc/passwd") Foo.bar.__set_ccl__(deny_ccl) foo1 = Foo("/etc/passwd") foo2.bar() // executed okay foo1.bar() *** EXCEPTION: execute capability barred to "all functions". okay, let's try this, first making sure there's no ambiguity by re-setting the CCL on the Foo class: Foo.bar.__set_ccl__(None) okay, now we go: foo3 = Foo("/etc/passwd") foo3.bar() // executed okay foo3.bar.__set_ccl__(deny_ccl) foo3.bar() *** EXCEPTION: execute capability barred to "all functions". note that there are three separate uses of __set_ccl__(). first is to set a CCL on the _class_ Foo. second is to clear a CCL the class Foo. third is to set a CCL on an instance of a Foo object. just to clarify: i _may_ have just violated my own proposal by not setting an "inherited-on-create" capability on the CCL named deny_ccl. the reason for needing to have such a capability is to be able to distinguish between setting CCLs on a class and a CCL being created on an object when an instance _of_ that class is created. this is part of the lesson learned from NT's security model (and originally VMS's) and its implementation. i won't go into more details on this inherit-on-create thing until the rest of the stuff sinks in :) l. From lkcl at lkcl.net Tue Dec 16 16:30:41 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 16:30:45 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195200.GI17021@lkcl.net> Message-ID: <20031216213041.GM17021@lkcl.net> On Tue, Dec 16, 2003 at 09:08:37PM +0100, Martin v. L?wis wrote: > Luke Kenneth Casson Leighton writes: > > > > I fail to see how ACLs are a sound basis to solve the problem that > > > rexec solves. ACLs, in my view, are a sound basis to solve a different > > > problem (that of different identities accessing the same resources). > > > > substitute "identity" for "calling object" i.e. function and i > > believe that there is a viable proposal. > > I don't think so. Suppose I have an object A protected by ACLs. Now I > write a new method > > def doit(): > A.doit() > > Question 1. What is the calling object? the globally-defined doit function. > Question 2. Assuming it is "the doit function", and assuming the doit > function is not listed in the ACL: Will the call be executed? see below. > If yes: What is the purpose of the ACL then? > If no: I believe the proposal is not usable in practice. For example, > what should be the ACL on calling open()? okay. that is why in ACLs (or better CCLs) it is necessary to have an "all functions" wildcard "name". in NT, this is the "world" security id "S-1-1" (i think) which represents "everyone". if you set a CCL, and you expect to deny everything but a particular function, you must have a two-entry CCL: [('function_to_be_denied_access', DENY, 'permissions_to_be_denied'), ('all functions', 'ALLOW', 'anything')] and yes, this is _normal_ practice in the use of ACLs in NT and VMS. and yes, the default action is (IIRC), "if not explicitly stated, deny". with the exception to that, of course, that if no CCL is set, it's "as if" there was a CCL set of [('all functions', 'ALLOW', 'anything')]. l. From lkcl at lkcl.net Tue Dec 16 16:36:58 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 16:37:02 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <76A3670A-300B-11D8-B3E2-000A27B19B96@cwi.nl> References: <1071587601.3fdf2111afc59@mcherm.com> <20031216161603.GG17021@lkcl.net> <76A3670A-300B-11D8-B3E2-000A27B19B96@cwi.nl> Message-ID: <20031216213658.GN17021@lkcl.net> On Tue, Dec 16, 2003 at 10:04:40PM +0100, Jack Jansen wrote: > > On 16-dec-03, at 17:16, Luke Kenneth Casson Leighton wrote: > >>Luke replied: > >>>capabilities, acls, schmapabilities, same thiiing :) > >> > >>No... they're not. Read the thread I mentioned above, or read this, > >>and some of the other documentation for the language E: > >> > >> http://www.erights.org/elib/capability/ode/ode-capabilities.html > > > > no offense intended: i'll read that later, i'm running out of time. > > > > without going into too many definitions, consider what i am advocating > > to be _like_ an access control list but instead to be a capabilities > > control list, instead. > > The distinction between capabilities and ACLs is really important, > because > they are almost each others opposite. With capabilities you have an > (unforgable) > right to do something and no-one cares about your identity, with ACLs > you have > an unforgable identity which is checked against the ACL. i'd like to introduce you to a new concept which is idential in form to an ACL - access control list - except that instead of "users" being allowed or denied access to perform certain operations you have instead _functions_ being allowed or denied access to perform certain operations. perhaps a better way to explain the concept to you is to introduce a concept called "qualified" capabilities, where what you know of as capabilities is "qualified" on a per-function (that's per-caller) basis. obviously, any object (by object i am referring generically to classes, class instances, functions, modules, absolutely anything) can potentially have many "callers", consequently it is necessary to create a _list_ of qualified capabilities, and for the relevant QCap in that list to be looked up and applied as needed. where, of course, the special wildcard name 'all functions' applies to _all_ callers. which makes what i am proposing to be named "QCCL" - qualified-capabilities control list. yuk :) l. From guido at python.org Tue Dec 16 17:07:56 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 17:08:01 2003 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 5, Issue 57 In-Reply-To: Your message of "Tue, 16 Dec 2003 15:17:49 EST." References: Message-ID: <200312162207.hBGM7uN20814@c-24-5-183-134.client.comcast.net> [Ed] > What's the motivation for restricting it to from-imports? I suspect > that this will cause confusion, and it doesn't feel self-consistent to > me: why can you call a module ".foo" in one context, and not in > another? In particular, I think that if people see that you can do: > > from .foo import * > > Then they'll assume that they can also do: > > import .foo > > With the obvious semantics, i.e. equivalence to: > > from . import foo > > What do we gain by not extending the syntax to from-less imports? Currently, the rule is that after import you can use in an expression to refer to whatever you imported. This applies to both import foo <--> foo in expressions and import foo.bar <--> foo.bar in expressions But if we were to allow import .foo we'd have to allow .foo in expressions, and I would like to reserve that syntax for the "with" statement. [Werner] > But it would be possible to say: > > import .foo as my_foo > import foo > > and access both through modules through the name they're bound to. > > Therefore I'd propose to allow relative from-less imports, but require > them to always use the "as" clause. This just adds more complexity; there's nothing you can do without this addition. [Jack] > Is "from . import *" allowed? Whenever I start to think about it I > tend to go into oscillation (yes because within package foo its the same > as "from foo import *" on the outside; no because "import *" isn't > allowed today either). Yes, "from . import " has the exact same meaning as spelling out the corresponding absolute module path after "from". --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Tue Dec 16 18:24:52 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Dec 16 18:24:59 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.py unuseable) In-Reply-To: <1071587601.3fdf2111afc59@mcherm.com> Message-ID: <200312162324.hBGNOq916269@oma.cosc.canterbury.ac.nz> Michael Chermside : > Suppose that had some code which had an object representing a > directory, with a method "open(filename, mode='r')" that opened > files in the directory. Given this object, you could imagine > constructing new objects with more limited capabilities. With regard to files, we'd almost have something like that now, if it weren't for the unfortunate fact that the file type's constructor can be used to open any file. In light of this, it may have been a serious mistake to unify the 'file' type and the 'open' function. Is it too late to back out of that decision? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido at python.org Tue Dec 16 18:38:20 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 18:38:37 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.py unuseable) In-Reply-To: Your message of "Wed, 17 Dec 2003 12:24:52 +1300." <200312162324.hBGNOq916269@oma.cosc.canterbury.ac.nz> References: <200312162324.hBGNOq916269@oma.cosc.canterbury.ac.nz> Message-ID: <200312162338.hBGNcKD21087@c-24-5-183-134.client.comcast.net> > Michael Chermside : > > > Suppose that had some code which had an object representing a > > directory, with a method "open(filename, mode='r')" that opened > > files in the directory. Given this object, you could imagine > > constructing new objects with more limited capabilities. Greg Ewing: > With regard to files, we'd almost have something like that > now, if it weren't for the unfortunate fact that the file > type's constructor can be used to open any file. Good observation. > In light of this, it may have been a serious mistake to > unify the 'file' type and the 'open' function. Is it too > late to back out of that decision? I think so. But to remedy this, the file constructor could simply refuse to open any files when called from restricted mode. (Although this would open up the same kind of holes that Samuele so cleverly showed exist in other built-ins that do this kind of checking.) (Re the capabilities discussion, this is Python 3.0 material if I ever saw some; no time to respond in more detail, given that we've had this thread before.) --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Tue Dec 16 19:22:49 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Dec 16 19:23:16 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> Message-ID: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> Guido: > I suggest that in 2.4, we introduce the leading dot notation for > relative import, while still allowing relative import without a > leading dot. In 2.5 we can start warning about relative import > without a leading dot If I understand correctly, the rule you're working towards is that if the path starts with a dot, it's relative, otherwise it's absolute. Is that correct? I'm not sure whether eliminating the "ambiguous" category of module references is the best idea. There are situations in which it's actually what you want. Suppose there are two top-level modules A and B, and A imports B using a bare name. Then someone decides to put A into a package. The reference to B still works. Then they decide to put B into the same package as well. The reference to B still works. But if only unambiguously absolute or relative references are possible, there is no way for A to refer to B that works in all these combinations. So I think we really want *three* kinds of module reference: A: Explicitly absolute B: Explicitly relative to the current module C: Searched for upwards in the package hierarchy from the current module (Note that C is a generalisation of the current "ambiguous" references which only look in two places.) Suggested syntaxes for these: A: a.b.c. Path ends with a dot B: .a.b.c Path begins with a dot C: a Path neither begins nor ends with a dot a.b.c Note that ALL current module references would be type C according to this. Yes, this is a semantic change, but one that I believe would be for the best in the long run, and don't think would cause a lot of upheaval. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tjreedy at UDel.Edu Tue Dec 16 20:04:41 2003 From: tjreedy at UDel.Edu (Terry Reedy) Date: Tue Dec 16 20:05:14 2003 Subject: [Python-Dev] Re: Pie-thon benchmarks References: <200312151937.hBFJbYX18012@c-24-5-183-134.client.comcast.net> <200312151957.hBFJvWi18151@c-24-5-183-134.client.comcast.net> <200312161830.hBGIUhE20354@c-24-5-183-134.client.comcast.net> Message-ID: <001a01c3c439$c83a74c0$0400a8c0@HPPAVILION> > > meaning of every bytecode). So I would include at > > least the syntax and standard lib test files (- > > exec and eval tests) as part of the benchmark. > Right. I don't want to include the entire test suite, but I do want Sorry, I meant builtins test, *not* the whole suite of stdlib tests. Terrj From greg at cosc.canterbury.ac.nz Tue Dec 16 20:09:14 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Dec 16 20:09:21 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312161843.hBGIhkE20496@c-24-5-183-134.client.comcast.net> Message-ID: <200312170109.hBH19DG16699@oma.cosc.canterbury.ac.nz> Guido: > > from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, \ > > normalize, row_ops, \ > > summary > > But I'm -0 on this change; I fear that someone accidentally adding a > trailing comma ... will be very puzzled How about a "suite" of things to import? from OPAL.Modules.Financial.Reports.COGS import: generate, webOptions, normalize row_ops, summary Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido at python.org Tue Dec 16 20:16:18 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 20:16:24 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Wed, 17 Dec 2003 14:09:14 +1300." <200312170109.hBH19DG16699@oma.cosc.canterbury.ac.nz> References: <200312170109.hBH19DG16699@oma.cosc.canterbury.ac.nz> Message-ID: <200312170116.hBH1GId21276@c-24-5-183-134.client.comcast.net> > > > from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, \ > > > normalize, row_ops, \ > > > summary > > > > But I'm -0 on this change; I fear that someone accidentally adding a > > trailing comma ... will be very puzzled > > How about a "suite" of things to import? > > from OPAL.Modules.Financial.Reports.COGS import: > generate, webOptions, normalize > row_ops, summary Nah. But I liked the suggestion seen here earlier of allowing parenthesis: from OPAL.Modules.Financial.Reports.COGS import (generate, webOptions, normalize, row_ops, summary) --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Tue Dec 16 20:19:51 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Dec 16 20:19:57 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312170109.hBH19DG16699@oma.cosc.canterbury.ac.nz> References: <200312161843.hBGIhkE20496@c-24-5-183-134.client.comcast.net> Message-ID: <5.1.1.6.0.20031216201656.02363ec0@telecommunity.com> At 02:09 PM 12/17/03 +1300, Greg Ewing wrote: >Guido: > > > > from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, \ > > > normalize, row_ops, \ > > > summary > > > > But I'm -0 on this change; I fear that someone accidentally adding a > > trailing comma ... will be very puzzled > >How about a "suite" of things to import? > > from OPAL.Modules.Financial.Reports.COGS import: > generate, webOptions, normalize > row_ops, summary First impression: cool! Second impression: "suites" contain statements, not a list. Third impression: why no comma after 'normalize'? From greg at cosc.canterbury.ac.nz Tue Dec 16 20:53:35 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Dec 16 20:53:40 2003 Subject: [Python-Dev] Relative import In-Reply-To: <5.1.1.6.0.20031216201656.02363ec0@telecommunity.com> Message-ID: <200312170153.hBH1rZG16888@oma.cosc.canterbury.ac.nz> > Second impression: "suites" contain statements, not a list. That's why I put "suite" in quotes. It wouldn't really be a suite, just an indented block of stuff. But if you want, it could be a real suite that only happens to allow import statements... from OPAL.Modules.Financial.Reports.COGS: import generate, webOptions, normalize import row_ops, summary Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From janssen at parc.com Tue Dec 16 21:05:42 2003 From: janssen at parc.com (Bill Janssen) Date: Tue Dec 16 21:06:02 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Tue, 16 Dec 2003 16:22:49 PST." <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> Message-ID: <03Dec16.180547pst."58611"@synergy1.parc.xerox.com> > Note that ALL current module references would be type C according > to this. Yes, this is a semantic change, but one that I believe > would be for the best in the long run, and don't think would cause > a lot of upheaval. Yes, I like this. Bill From andrew-pythondev at puzzling.org Tue Dec 16 21:45:50 2003 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Tue Dec 16 21:45:56 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312170153.hBH1rZG16888@oma.cosc.canterbury.ac.nz> References: <5.1.1.6.0.20031216201656.02363ec0@telecommunity.com> <200312170153.hBH1rZG16888@oma.cosc.canterbury.ac.nz> Message-ID: <20031217024550.GB473@frobozz> On Wed, Dec 17, 2003 at 02:53:35PM +1300, Greg Ewing wrote: > > Second impression: "suites" contain statements, not a list. > > That's why I put "suite" in quotes. It wouldn't really be > a suite, just an indented block of stuff. > > But if you want, it could be a real suite that only > happens to allow import statements... > > from OPAL.Modules.Financial.Reports.COGS: > import generate, webOptions, normalize > import row_ops, summary But this is also a very odd suite, because you can't use other statements... But it does make me think of: with OPAL.Modules.Financial.Reports.COGS: import .generate, .webOptions, .normalize import .row_ops, .summary Which is like a bastard child of the relative import proposal and the with statement proposal... -Andrew. From eppstein at ics.uci.edu Tue Dec 16 22:07:52 2003 From: eppstein at ics.uci.edu (David Eppstein) Date: Tue Dec 16 22:07:46 2003 Subject: [Python-Dev] Re: Relative import References: <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> Message-ID: In article <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz>, Greg Ewing wrote: > So I think we really want *three* kinds of module reference: > > A: Explicitly absolute > B: Explicitly relative to the current module > C: Searched for upwards in the package hierarchy from the current > module > > (Note that C is a generalisation of the current "ambiguous" > references which only look in two places.) > > Suggested syntaxes for these: > > A: a.b.c. Path ends with a dot > > B: .a.b.c Path begins with a dot > > C: a Path neither begins nor ends with a dot > a.b.c Is funny punctuation really the right way to spell an important distinction like this? -- David Eppstein http://www.ics.uci.edu/~eppstein/ Univ. of California, Irvine, School of Information & Computer Science From tdelaney at avaya.com Tue Dec 16 22:20:32 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Tue Dec 16 22:20:38 2003 Subject: [Python-Dev] Re: Relative import Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0B9B@au3010avexu1.global.avaya.com> > From: David Eppstein > > Is funny punctuation really the right way to spell an important > distinction like this? That's what I've been wondering. I really don't like the proposal to use . in various places to signify relative and absolute imports. It seems non-obvious both in understandability and readability. Every example I've seen has looked wrong to me. I'll try to think of something I prefer, but I am definitely -1 on the proposed syntax changes. Tim Delaney From anthony at interlink.com.au Tue Dec 16 22:27:52 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Dec 16 22:28:23 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0B9B@au3010avexu1.global.avaya.com> Message-ID: <200312170327.hBH3RrpU004309@localhost.localdomain> >>> "Delaney, Timothy C (Timothy)" wrote > > Is funny punctuation really the right way to spell an important > > distinction like this? > > That's what I've been wondering. I really don't like the proposal to use > . in various places to signify relative and absolute imports. It seems > non-obvious both in understandability and readability. I'd have to agree. In the same way that backticks are painful to deal with (they're small and non-obvious), the difference between from foo import bar and from .foo import bar is too small to be pleasant. I also have to wonder about the use case for .. and ... - looking through my code, I can't see anything where using ... rather than a fully qualified name would make the code any more readable. -- Anthony Baxter It's never too late to have a happy childhood. From greg at cosc.canterbury.ac.nz Tue Dec 16 22:31:50 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Dec 16 22:32:35 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Message-ID: <200312170331.hBH3Vog17551@oma.cosc.canterbury.ac.nz> David Eppstein : > Is funny punctuation really the right way to spell an important > distinction like this? Maybe not, but I haven't seen anything obviously better proposed yet. I've no idea what else to to for relative paths. A leading dot doesn't seem all that bad to me for that. I concede that a trailing dot would look rather odd, though. The absolute-path case could be spelled using a special name for the root of the package hierarchy, but then we need to think of a name that's sufficiently unlikely as a real module name. I don't think I like the idea of re-using the word "global", although I'm not sure. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tim.one at comcast.net Tue Dec 16 23:42:45 2003 From: tim.one at comcast.net (Tim Peters) Date: Tue Dec 16 23:42:46 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312170116.hBH1GId21276@c-24-5-183-134.client.comcast.net> Message-ID: [Guido] > Nah. But I liked the suggestion seen here earlier of allowing > parenthesis: > > from OPAL.Modules.Financial.Reports.COGS import (generate, webOptions, > normalize, row_ops, summary) So do I. Something I'm still acutely aware of despite not using Emacs for years is how much trouble it would be to teach python-mode.el about a unique new way to continue a statement. Parens would "just work". Ditto for IDLE, and tokenize.py, etc etc etc etc. I bet tools parsing import statements for *content* (like pychecker, and perhaps pyclbr.py -- the others mentioned are just looking at form) would need modification regardless, though. From duncan at rcp.co.uk Wed Dec 17 04:17:25 2003 From: duncan at rcp.co.uk (Duncan Booth) Date: Wed Dec 17 04:17:39 2003 Subject: [Python-Dev] rexec.py unuseable References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> <20031216212312.GL17021@lkcl.net> Message-ID: Luke Kenneth Casson Leighton wrote in news:20031216212312.GL17021@lkcl.net: > foo3.bar() // executed okay > foo3.bar.__set_ccl__(deny_ccl) > > foo3.bar() > *** EXCEPTION: execute capability barred to "all functions". This behaviour seems to imply a rather fundamental change to the way that Python handles bound methods. At present every time you access foo.bar you get a new object created, but I think from your description that you want to be able to associate a ccl with all the foo.bar objects. Or to put it another way: fn1 = foo3.bar fn1() // executed okay fn2 = foo3.bar fn2.__set_ccl__(deny_ccl) fn2() *** EXCEPTION: execute capability barred to "all functions". fn1() *** ??? What should this do. If I set a ccl on fn2, should it also affect access to fn1? They are separate objects, so shouldn't they have separate ccls? If they share a common ccl, then how is it going to be stored without impacting heavily on every method call? -- Duncan Booth duncan.booth at suttoncourtenay.org.uk int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3" "\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure? From theller at python.net Wed Dec 17 06:32:32 2003 From: theller at python.net (Thomas Heller) Date: Wed Dec 17 06:32:47 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) - custom Py_Initialize() In-Reply-To: <002301c3c29c$f2753480$0201010a@kret> (Wiktor Sadowski's message of "Mon, 15 Dec 2003 00:49:42 +0100") References: <002301c3c29c$f2753480$0201010a@kret> Message-ID: "Wiktor Sadowski" writes: > Adding a few lines (below) to Python would enable custom > Py_Initialize() and would allow to setup on time different than > existing (import.c-zipimport.c) import mechanisms and changing some > other Python-initialization internals without messing with Python > core. > > It should make life easier for embedders and help writing specialized > Python executables (like Zope or Quixote). (We have such specialized > executable to run packed Python programs remotely,kind of Java Web > Start). From what I know Python installers (py2exe,Gordon McMillan's > installer) could benefit from such addition as well. > > All these functions simply expose some modules globals. They don't > change anything in Python core. > > Best regards, > Wiktor Sadowski > > > /*pythonrun.c*/ > void Internal_Py_SetInitialized(int){initialized=i;} > void Internal_PyUnicode_Init(void){_PyUnicode_Init();} > void Internal_PyUnicode_Fini(void){_PyUnicode_Fini();} > void Internal_PyGILState_Init(PyInterpreterState *s, PyThreadState > *ts){_PyGILState_Init(s,ts);} > void Internal_PyGILState_Fini(void){_PyGILState_Fini();} > long Internal_PyGC_Collect(void){return PyGC_Collect();} > > /*pythonrun.h*/ > PyAPI_FUNC(void) Internal_Py_SetInitialized(int i); > PyAPI_FUNC(void) Internal_PyUnicode_Init(void); > PyAPI_FUNC(void) Internal_PyUnicode_Fini(void); > PyAPI_FUNC(void) Internal_PyGILState_Init(PyInterpreterState *, > PyThreadState *); > PyAPI_FUNC(void) Internal_PyGILState_Fini(void); > PyAPI_FUNC(long) Internal_PyGC_Collect(void); > I would rather like a function like Py_InitializeEx(struct _init*), where struct _init would have components like PYTHONPATH, optimize, and so on. Thomas From ncoghlan at iinet.net.au Wed Dec 17 07:20:17 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Wed Dec 17 07:20:24 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <20031216213658.GN17021@lkcl.net> References: <1071587601.3fdf2111afc59@mcherm.com> <20031216161603.GG17021@lkcl.net> <76A3670A-300B-11D8-B3E2-000A27B19B96@cwi.nl> <20031216213658.GN17021@lkcl.net> Message-ID: <3FE04A01.7010500@iinet.net.au> Luke Kenneth Casson Leighton wrote: > i'd like to introduce you to a new concept which is idential > in form to an ACL - access control list - except that instead > of "users" being allowed or denied access to perform certain > operations you have instead _functions_ being allowed or > denied access to perform certain operations. I don't think this discussion is going to get anywhere until you _do_ take the time to understand the difference between the model you are proposing and the capability model that seemed to be preferred the last time executing untrusted Python code safely was discussed. ONE LINE SUMMARIES ACL/CCL/whatever you want to call it: Access the feature, then have permission either granted or denied Capabilities: If you can access it, you have permission to use it. If you aren't allowed to use it, you won't even be able to see it. Below is my attempt at explaining in more detail the difference between the two as I understand it. I must admit I am quite curious about the fact that you seem to have plenty of time to _write_ about your proposal, but no time to listen to some quite reasonable responses. ACCESS CONTROL LISTS An access control list is neutral to what a 'user' is, so I'll switch to the term 'Entity' instead. The important part is that it is a mapping: "Entity X is allowed to perform action Y on other Entity Z". It serves as a post hoc check - _after_ Entity X tries to do Y to Z, the control list for Y & Z is checked, then X is either allowed to proceed, or told to rack off, depending on what the control list reports. That is, just because X _can_ try to perform action Y on Z (the relevant method or whatever is accessible to X), doesn't mean that X _may_ perform action Y (the control list grants permission to X). With control lists, the concepts of access to a feature, and permission to use that feature, are handled separately. E.g. I can try to delete the windows directory on my machine at work (I have access to that directory, and access to the delete command), but I'm not an administrator, so the attempt will fail (I don't have permission to delete that directory). This is exactly what you are describing, even though the users are 'functions' rather than 'logged in users'. CAPABILITIES Capabilities work differently. Instead of applying a post-hoc check that slows down _every_ attempt to perform an action, they unify the concepts of 'can' and 'may'. If X is able to express the command "do Y to Z", then X is also _permitted_ to do Y to Z. No post-hoc checks required. For example, suppose I wasn't allowed to open writable files. I call: >>> target = file("myfile.txt", "w") With ACL's, I would get an error along the lines of: AccessDenied: No permission to open writable files With capabilities, I would get an error along the lines of: InvalidArgument: "w" is not a recognised file mode Does this make any more sense? Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From Jack.Jansen at cwi.nl Wed Dec 17 07:51:12 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Wed Dec 17 07:51:14 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) - custom Py_Initialize() In-Reply-To: References: <002301c3c29c$f2753480$0201010a@kret> Message-ID: On 17-dec-03, at 12:32, Thomas Heller wrote: > I would rather like a function like Py_InitializeEx(struct _init*), > where struct _init would have components like PYTHONPATH, optimize, and > so on. You'd have to be very careful to design the struct in such a way that you'd have good compatibility. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From ncoghlan at iinet.net.au Wed Dec 17 07:58:20 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Wed Dec 17 07:58:29 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> Message-ID: <3FE052EC.90705@iinet.net.au> Greg Ewing wrote: > So I think we really want *three* kinds of module reference: > > A: Explicitly absolute > B: Explicitly relative to the current module > C: Searched for upwards in the package hierarchy from the current > module > > (Note that C is a generalisation of the current "ambiguous" > references which only look in two places.) > Alternate spellings (plus category D, which is "Python 2.3 semantics"): A: from __absolute__.dotted.name import foo B: from __relative__.dotted.name import bar C: from __heirarchy__.dotted.name import foobar D: from dotted.name import barfoo I believe this spelling would only require import to handle the special cases as the first 'package' name (although having the compiler recognise them may be a later optimisation), and should also work with straight "import __absolute__.dotted.name.foo" statements. Then Guido's migration plan would apply to the progression of the D semantics from the Python 2.3 model to being a shorthand for the absolute semantics. And once _that_ has happened, then we could get rid of the '__absolute__' version in the name of TOOWTDI. Regards, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From mcherm at mcherm.com Wed Dec 17 08:56:37 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Wed Dec 17 08:56:58 2003 Subject: [Python-Dev] Re: rexec.py unuseable Message-ID: <1071669397.3fe06095c9f3c@mcherm.com> Luke: Seeing as you've now had time enough to read what I'd written, I was going to respond to some of the points that you made, but then I read Nick Coghlan's posting[1], and I really don't have anything else to add. [1] http://mail.python.org/pipermail/python-dev/2003-December/041105.html Well, that's not quite true... I guess I _do_ have something small to add. In designing a security policy for file systems, there are very few things that you can do with the protected items. Separate "read", "write", and "execute" permissions seem appropriate. Furthermore, there tends to be a clear idea of a "user", and in many situations we want all access to items (files) to depend only on who that "user" is (sandboxing an email program is an exception... are you listening MS?). Finally, we often want to provide centralized administration of access rights to file systems. ACLs are a good match for these design specs. However, controlling what arbitrary programs may do is a very different beast. First of all, the kinds of restrictions that we want are HIGHLY variable. Sometimes we want to prohibit all file access. Sometimes we want to allow read-only access to a certain file only. Sometimes we want access only to a certain directory. Sometimes we don't care at all about file system access, we're concerned instead with providing an object which can be queried but only in certain ways -- and the specifics of what is permitted and what isn't are completely dependent on the particular application. On top of that, there's rarely a clear concept of "users" that controls what actions we want to permit. Instead, we usually want to allow or deny certain capabilities to particular _sections of code_. (Sometimes that will be a module, sometimes a plug-in, sometimes a child program.) All of these things suggest that maintaining a list of who is permitted to access what is simply not the best model for providing security in a programming language. I think that capabilities ARE an excellent model. Sections of code already have the concept of "access" to certain objects and variables (at least since the invention of structured programming in the late 1960s). And _if_ unforgability and private data structures are provided as elemental building tools, then it is relatively straightforward to write code that enforces whatever set of access rules is required, regardless of how complex it may be. -- Michael Chermside From mickey at tm.informatik.uni-frankfurt.de Wed Dec 17 09:01:00 2003 From: mickey at tm.informatik.uni-frankfurt.de (Michael Lauer) Date: Wed Dec 17 09:01:08 2003 Subject: [Python-Dev] bug in python arm-linux?: start_new_thread fails after popen Message-ID: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> Hi, I have a python program which works fine on x86 but doesn't work on any of my arm-linux devices (Zaurus, Ipaq, SIMpad) - all of them are running Python 2.3.2 on top of glibc 2.3.2+linuxthreads on top of kernel 2.4.18-rmk6-pxa3 respectively kernel 2.4.19-rmk7. After a long week of debugging I now can reproduce the behaviour with the following minimal case: ---------------------------------------------------------------------- import thread from time import sleep import sys import os def createPipe(): print os.popen( "ls -l" ).read() def threadMain( name ): while True: sys.stderr.write( name+": i'm still running" ) sys.stderr.flush() sleep( 1 ) if __name__ == '__main__': createPipe() print "BEFORE start_new_thread" thread.start_new_thread( threadMain, ( "1", ) ) print "AFTER start_new_thread" createPipe() print "BEFORE start_new_thread" thread.start_new_thread( threadMain, ( "2", ) ) print "AFTER start_new_thread" sleep( 5 ) ----------------------------------------------------------------------- This program - as is - just hangs in the first start_new_thread() and never comes back. If you outcomment the first call to createPipe() out, then it works fine. In the first - hanging - case, an strace shows: ------------------------------------------------------------------------- write(1, "BEFORE start_new_thread\n", 24BEFORE start_new_thread ) = 24 rt_sigprocmask(SIG_BLOCK, ~[33], [RTMIN], 8) = 0 pipe([3, 4]) = 0 clone(child_stack=0x14e080, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND) = 949 write(4, "\314V\1@\5\0\0\0\0\0\0\0\0\0\0\0\34\213\37@\10c\1@\0\0"..., 148) = 148 rt_sigprocmask(SIG_SETMASK, NULL, ~[KILL STOP 33], 8) = 0 write(4, "\0\21\3@\0\0\0\0\0\0\0\0hO\n\0\2503\23\0\377\376\373\377"..., 148) = 148 rt_sigprocmask(SIG_SETMASK, NULL, ~[KILL STOP 33], 8) = 0 rt_sigsuspend(~[KILL STOP RTMIN 33] -------------------------------------------------------------------------- The program must be kill -9'ed at this point. Can anyone explain this behaviour to me - did I find a bug? Sincerely, -- :M: -------------------------------------------------------------------------- Dipl.-Inf. Michael 'Mickey' Lauer mickey@tm.informatik.uni-frankfurt.de Raum 10b - ++49 69 798 28358 Fachbereich Informatik und Biologie -------------------------------------------------------------------------- From barry at python.org Wed Dec 17 09:45:40 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 09:45:48 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312170116.hBH1GId21276@c-24-5-183-134.client.comcast.net> References: <200312170109.hBH19DG16699@oma.cosc.canterbury.ac.nz> <200312170116.hBH1GId21276@c-24-5-183-134.client.comcast.net> Message-ID: <1071672339.27043.4.camel@anthem> On Tue, 2003-12-16 at 20:16, Guido van Rossum wrote: > Nah. But I liked the suggestion seen here earlier of allowing > parenthesis: > > from OPAL.Modules.Financial.Reports.COGS import (generate, webOptions, > normalize, row_ops, summary) +1 -Barry From jepler at unpythonic.net Wed Dec 17 10:14:50 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Wed Dec 17 10:15:01 2003 Subject: [Python-Dev] bug in python arm-linux?: start_new_thread fails after popen In-Reply-To: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> Message-ID: <20031217151450.GB28436@unpythonic.net> On Wed, Dec 17, 2003 at 03:01:00PM +0100, Michael Lauer wrote: > Hi, I have a python program which works fine on x86 but doesn't work on > any of my arm-linux devices (Zaurus, Ipaq, SIMpad) - all of them are > running Python 2.3.2 on top of glibc 2.3.2+linuxthreads on top of kernel > 2.4.18-rmk6-pxa3 respectively kernel 2.4.19-rmk7. Using threads and fork together seems to be a big smelly armpit in Python. There are also problems on redhat9, where signals in a fork+exec'd subprocess are blocked, for instance. This seemed to be a consequence of blocking all signals in thread_pthread.h:PyThread_start_new_thread(). Perhaps pthread_atfork() could be used to fix this problem, though I know next to nothing about pthreads beyond the documentation I glanced at back when I first became aware of my thread+fork problem and before writing this message. Jeff From ncoghlan at iinet.net.au Wed Dec 17 10:22:05 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Wed Dec 17 10:22:18 2003 Subject: [Python-Dev] Relative import In-Reply-To: References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> <3FE052EC.90705@iinet.net.au> Message-ID: <3FE0749D.2030705@iinet.net.au> John J Lee wrote: > I can see that people not knowing how to spell "hierarchy" might be a > problem here Heh, good point. I always get confused by the place of the 'heir' in the 'hierarchy'. I had a seriously hard time coming up with a one word name that suggested the right semantics for that category. If the idea gets any interest, it'd be nice to find a better word for that case. I was tempted to use __ambiguous__, though :) Hmm, __scan__ could be an option. The other two are fairly specific about where to look, but category 'C' requests a scan of the package heirarchy. A: from __absolute__.dotted.name import foo B: from __relative__.dotted.name import bar C: from __scan__.dotted.name import foobar D: from dotted.name import barfoo I suspect the 'as' clause would be heavily used with this construct, but it wouldn't be compulsory - as far as module referencing in the code goes, I don't think there's anything illegal about the special names. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From Boris.Boutillier at arteris.net Wed Dec 17 11:13:19 2003 From: Boris.Boutillier at arteris.net (Boris Boutillier) Date: Wed Dec 17 11:13:30 2003 Subject: [Python-Dev] Relative import In-Reply-To: <3FE052EC.90705@iinet.net.au> References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> <3FE052EC.90705@iinet.net.au> Message-ID: <3FE0809F.2020702@arteris.net> Just to give you a practical example, we are using python for internal tools in the electronic Company I'm working for. I needed to have a non ambiguous file lookup when using import ( We set external dependencies for the flow when opening a file, so you can't use a 'if this file exist' check, when opening a file, it must always exists) I made a special import function, that was looking for a prefix to the package name: import loc.dotted.name import glb.dotted.name import __.dotted.name import std.dotted.name import xxx.dotted.name -> raise an error "loc", look in the current directory of the module (looking to his __file__ attribute) "__ "look to the parent directory "glb" look at the root of the user working repository ( a builtins set before anything else is done , users don't call python they the program myPython ) "std" look outside the database ( in the python installation dir mainly) and set back the old import behaviour to ensure that import inside standard modules don't fail ) I know the loc,glb __ and std names are no good for a general case as modules can have these names, but this is just to give you an actual use of these relatives imports. Boris Nick Coghlan wrote: > Greg Ewing wrote: > >> So I think we really want *three* kinds of module reference: >> >> A: Explicitly absolute >> B: Explicitly relative to the current module >> C: Searched for upwards in the package hierarchy from the current >> module >> >> (Note that C is a generalisation of the current "ambiguous" >> references which only look in two places.) >> > > Alternate spellings (plus category D, which is "Python 2.3 semantics"): > > A: from __absolute__.dotted.name import foo > B: from __relative__.dotted.name import bar > C: from __heirarchy__.dotted.name import foobar > D: from dotted.name import barfoo > > I believe this spelling would only require import to handle the > special cases as the first 'package' name (although having the > compiler recognise them may be a later optimisation), and should also > work with straight "import __absolute__.dotted.name.foo" statements. > > Then Guido's migration plan would apply to the progression of the D > semantics from the Python 2.3 model to being a shorthand for the > absolute semantics. > > And once _that_ has happened, then we could get rid of the > '__absolute__' version in the name of TOOWTDI. > > Regards, > Nick. > From skip at pobox.com Wed Dec 17 11:14:03 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Dec 17 11:14:03 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0B9B@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0B9B@au3010avexu1.global.avaya.com> Message-ID: <16352.32971.627159.669571@montanaro.dyndns.org> >> Is funny punctuation really the right way to spell an important >> distinction like this? Tim> That's what I've been wondering. I really don't like the proposal Tim> to use . in various places to signify relative and absolute Tim> imports. It seems non-obvious both in understandability and Tim> readability. Agreed. Let's shoot for something more Pythonic. I can imagine having to count periods in import statements in much the way I used to have to count closing parens in Lisp code on a CDC Cyber (way before I had ever heard of language-sensitive editing). Skip From pb at nexus.co.uk Wed Dec 17 11:39:34 2003 From: pb at nexus.co.uk (Phil Blundell) Date: Wed Dec 17 11:39:42 2003 Subject: [Python-Dev] Re: bug in python arm-linux?: start_new_thread fails after popen In-Reply-To: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> Message-ID: <20031217163934.GA13660@nexus.co.uk> on Wed, Dec 17, 2003 at 03:01:00PM +0100, Michael Lauer wrote: > This program - as is - just hangs in the first start_new_thread() and > never comes back. > If you outcomment the first call to createPipe() out, then it works > fine. I can't reproduce this problem on my Debian/ARM machine. I'll try it on the iPAQ under Familiar later. My guess is that your glibc is too old. p. From theller at python.net Wed Dec 17 11:42:14 2003 From: theller at python.net (Thomas Heller) Date: Wed Dec 17 11:42:33 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) - custom Py_Initialize() In-Reply-To: (Jack Jansen's message of "Wed, 17 Dec 2003 13:51:12 +0100") References: <002301c3c29c$f2753480$0201010a@kret> Message-ID: Jack Jansen writes: > On 17-dec-03, at 12:32, Thomas Heller wrote: >> I would rather like a function like Py_InitializeEx(struct _init*), >> where struct _init would have components like PYTHONPATH, optimize, and >> so on. > > You'd have to be very careful to design the struct in such a way that > you'd have good compatibility. Sure, but it can be done. The alternative would be a Py_InitializeN() call with many parameters. Would that be preferrable? Thomas From charleshixsn at earthlink.net Wed Dec 17 11:55:40 2003 From: charleshixsn at earthlink.net (Charles Hixson) Date: Wed Dec 17 11:59:54 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <1071509846.970.105.camel@anthem> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <16349.59071.26004.502201@montanaro.dyndns.org> <1071509846.970.105.camel@anthem> Message-ID: <3FE08A8C.305@earthlink.net> Barry Warsaw wrote: >On Mon, 2003-12-15 at 11:52, Skip Montanaro wrote: > > > ... > >Sorry, let me rephrase. I'd love to ban relative imports. Personally, >I think all imports should be specified using fully-qualified absolute >package paths. > >-Barry > > If I'm understanding this right you want to ban imports that are machine independant. That can't be right, but that's what it sounds like to me. To me an "absolute import" or a "fully-qualified absolute package path" sounds like you know the exact directory that the import is occuring from. E.g.: /home/someone/projects/python/workingprogress/currentproject/thisone.py I can see the use for being able to import that, but usually I would prefer to import either from something relative to the current directory, or relative to the python site-lib, or some such. It's also quite nice to not need to bother to exactly specify just which of those I will find it in. (This is what the pythonpath is for.) Now to me an import relative to the current directory is a relative import, and I assume that this is also what is meant by a local import. By allowing files to cluster relative to the code being written one is able to keep related projects and libraries together rather than just having all the code spread out aimlessly in a huge flat directory. I don't see in what what this is inferior. Since you didn't say why you wanted to ban relative imports, I don't understand your reasons. I can't even guess at them. Even if it were for some efficiency condern about hierarchical directories, that would merely say that the compiled modules should be stuffed into a library folder. (Well, more than one...the user needs to have write permission to the libraries that he creates, but it's just as well that he not have write permissions to the system libraries...without taking special steps.) From guido at python.org Wed Dec 17 12:21:57 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 12:22:02 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 01:22:05 +1000." <3FE0749D.2030705@iinet.net.au> References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> <3FE052EC.90705@iinet.net.au> <3FE0749D.2030705@iinet.net.au> Message-ID: <200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> It's interesting that the "scan upwards on the package name" gets so much support. Back when packages were first introduced, there was a period of time when they were a user-level construct, implemented by a module named "ni" (after the Knights who say "Ni!"). This implemented a full scan upward as the only way. When I reimplemented packages in C, I decided that the scan wasn't used much, and could actually be detrimental (I feared that too many global modules would be accidentally hidden), so I changed it to only looking in only two places: the current package and the toplevel (sys.path). But at the time it was a given that the import syntax would not be changed. Now that we're entertaining new import syntax, we can do better. How about this for Python 3.0: import P.Q (1) from P.Q import X (2) would only look for P in sys.path (I'll call this "absolute import"); and from ...P.Q import X (3) would scan for P upwards from the current package. It would require exactly three dots. P.Q of course stands for any dotted name. Python 2.x would continue to use the current ("ambiguous import") semantics for (1) and (2), but would also support (3) with the Python 3.0 semantics. It might also be possible to set a "no ambiguous imports" flag to switch to 3.0 semantics for (1) and (2). This might be a __future__ statement in the package's __init__.py, e.g. from __future__ import absolute_import I think this addresses all the concerns I've seen brought up in this thread so far against my original proposal and the variations that were subsequently suggested. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Dec 17 12:26:13 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 12:26:23 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: Your message of "Wed, 17 Dec 2003 06:19:51 PST." <1071670791.3fe0660761aa6@mcherm.com> References: <1071670791.3fe0660761aa6@mcherm.com> Message-ID: <200312171726.hBHHQDq22544@c-24-5-183-134.client.comcast.net> [Michael Chermside] > I disagree. Right now, having access to a class object basically > gives one the ability to create new objects of that type. I > think that's just fine... and I don't mind applying it to the > file object. I'd think that the thing to do with untrusted code > is to deny it access to the 'file' type object, thus denying it > the ability to create new 'file's directly. > > After all, it's not like file has lots of useful classmethods... > client code which is not supposed to create new files has no > particular need to access the 'file' object. Oh yes, except > that it may have INSTANCES of 'file' and could access > f.__class__. But if it is RESTRICTED code, then the only > legitimate use of f.__class__ is to do typechecking (it's > arguable whether that is good design, but it does seem to be > a legitimate use), so for restricted code we return something > like this: > > class FakeFile: > def __eq__(self, other): > return other == file > > [...] Are you aware of the original issue, which is that as soon as you have a file *instance* (which might have been given to you by a very restrictive open() variant), you can always get to the file *class* using the __class__ attribute? Access to the __class__ attribute is useful for all sorts of reasons. > Guido writes: > > Re the capabilities discussion, this is Python 3.0 material if I ever > > saw some > > I agree. But I can dream, can't I? Yes, even for 3.0 this is still a dream... --Guido van Rossum (home page: http://www.python.org/~guido/) From lkcl at lkcl.net Wed Dec 17 12:56:32 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Wed Dec 17 12:56:08 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <3FE04CDD.1050109@iinet.net.au> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195856.GK17021@lkcl.net> <3FE04CDD.1050109@iinet.net.au> Message-ID: <20031217175632.GA2117@lkcl.net> > the two as I understand it. I must admit I am quite curious about the > fact that you seem to have plenty of time to _write_ about your > proposal, but no time to listen to some quite reasonable responses. nick, thank you for pointing this out. that i do not explicitly acknowledge other people's point is a common failing of mine that has frequently got me into significant communication difficulties. yes i _do_ appreciate and understand what people say, and take the time to read it: i don't necessarily always actually explicitly say that and i really appreciate you picking up on this. if anyone else who is responding on this thread feels offended because they do not believe that i am listening to what they have to say PLEASE SAY SO. thanks, l. From lkcl at lkcl.net Wed Dec 17 14:06:52 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Wed Dec 17 14:06:49 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <1071669397.3fe06095c9f3c@mcherm.com> References: <1071669397.3fe06095c9f3c@mcherm.com> Message-ID: <20031217190652.GE2117@lkcl.net> On Wed, Dec 17, 2003 at 05:56:37AM -0800, Michael Chermside wrote: > Luke: > > Seeing as you've now had time enough to read what I'd written, yes i have now, except not yet the link you recommended to 'E', which i will get to, soon. > I was going to > respond to some of the points that you made, but then I read Nick Coghlan's > posting[1], and I really don't have anything else to add. > > [1] http://mail.python.org/pipermail/python-dev/2003-December/041105.html am absorbing that: it looks like a good definition of acls and capabilities to me! > Well, that's not quite true... I guess I _do_ have something small to add. > In designing a security policy for file systems, there are very few things > that you can do with the protected items. Separate "read", "write", and > "execute" permissions seem appropriate. yes. it's worth pointing out that NT file systems have a lot more than just rwx (which are covered by "generic read" permission, "generic write" and "generic execute"). if i remember correctly, NT file system access permissions also include separate permissions for directory access and creation that are deliberately distinct from file creation (and maybe even file access, too, but it's been a while so i can't be sure). they also have "change security descriptor" as a permission. ... *click*. darn, it. darn it quite a lot. it's been such a long time that i've been confusing "security descriptor" with "access control list". although... thinking about it some more i don't think it makes much difference... definition of an NT (vax/vms) security descriptor: a security descriptor is a set of "access control lists". there are four ACLs, and they are for different purposes. i can nail down two of them for you: one is a "System" ACL which restricts what the OPERATING SYSTEM can do. another is a "Users" ACL which restricts what _users_ can do (and groups, which also under NT have their own identity just like a user has its own identity.) there is also some additional information on a per-acl basis and it controls the way in which the ACL is used. ... but all that is by-the-by because, i believe, only _one_ ACL is needed for the purposes at hand. anyway, sorry to interrupt the flow, there. :) > Furthermore, there tends to be a > clear idea of a "user", and in many situations we want all access to items > (files) to depend only on who that "user" is (sandboxing an email program > is an exception... are you listening MS?). Finally, we often want to > provide centralized administration of access rights to file systems. > ACLs are a good match for these design specs. fully designed, properly designed ACLs are, imo, way over the top for file systems, which is why POSIX ACLs are just an extension of the standard POSIX filesystem rwx/ugo. for NT (and VAX/VMS), the design of the security system came first, as a "generic thing", which was then applied to file systems as _one_ component, albeit a critical one, of the NT and VMS operating systems. the NT/VMS security model has room for 32-bits of "permissions" on an ACE (access control entry). 16-bit of those are "generic", and 16-bit of them are "application specific". the file-system-specific permissions were created by the writers of the NTFS - nt file system - and they gave them life and meaning, so to speak, but only in the context of an NTFS and access to it. > However, controlling what arbitrary programs may do is a very different > beast. in the context of something like "POSIX ACLs" or in the context of something like "POSIX file permissions", yes, absolutely, i agree with you entirely. in the context of something as flexible as NT/VMS security descriptors, no, i don't agree, because NT SDs are very generic and had to be "specialised" to fit onto the NTFS. which implies that something as flexible as NT/VMS SDs could equally and as easily be adapted to fit controlling what arbitrary programs do. _and_, in the NT operating system and its core [mostly user-space] components, this _has_ already been done [and i presume likewise in VMS]. for example, there is a spoolss (printer service) and there is a SAM (security accounts manager) database service. the authors / designers of these two services created their own set of 16-bit application-specific permissions, gave them life and meaning in the context of their individual services, and made sure that each function _in_ that service double-checked the permissions against the type of operation that that function performed. to give you an example, this is some pseudo-code of the SamrCreateUser function: static SEC_DES[] sec_des_array = { ... ... samr_create_user_security_descriptor, ... }; STATUS32 SamrCreateUser(pHND *context, UNICODE_STRING *username) { USER_TOK *user = GetUserToken(context); if (!check_security_descriptor(user, context, SECDES_SAM_CREATE_USER)) { return NT_STATUS_ACCESS_DENIED; } .... .... } the context handle stores references to quite a bit of state information about the connection (it's a DCE/RPC thing). the SECDES_SAM_CREATE_USER permission is an application-specific bit that is specifically and only used to check access permission to the SamrCreateUser function. > First of all, the kinds of restrictions that we want are HIGHLY > variable. Sometimes we want to prohibit all file access. Sometimes we > want to allow read-only access to a certain file only. Sometimes we > want access only to a certain directory. Sometimes we don't care at > all about file system access, we're concerned instead with providing > an object which can be queried but only in certain ways -- and the > specifics of what is permitted and what isn't are completely dependent > on the particular application. > > On top of that, there's rarely a clear concept of "users" that controls > what actions we want to permit. Instead, we usually want to allow or > deny certain capabilities to particular _sections of code_. (Sometimes > that will be a module, sometimes a plug-in, sometimes a child program.) yes. and i believe that the > All of these things suggest that maintaining a list of who [or rather "what" - i.e. "Entities", yes?] > is permitted > to access what is simply not the best model for providing security in > a programming language. i am curious: how does that logically follow? in my experience (windows nt services and therefore also samba-tng's services), access control lists have been an extremely successful means to provide security in almost every single program in the NT operating system (i don't mean XP i don't mean windows 95) for almost 15 to 20 years. it seems quite logical therefore to propose that such a successful security model be adapted to fit a programming language, or at least to learn from its success (and failings). change of subject ----------------- i believe that it _is_ possible to express what nick coghlan defines capabilities to be _into_ an ACL. how? by creating a security permission called "can you see it and execute it?" let's call it SEE_AND_EXECUTE then, the language (interpreter) does this (following the example of the SamrCreateUser function, above): MakeFunctionCall(Context *stack, CodeObject *code) { char *callee_fn_name = GetCalleeName(stack); SEC_DES *sd = GetSecurityDescriptorForCode(code); /* check the capability to even _see_ this function! */ if (!check_sec_des(sd, callee_fn_name, SEE_AND_EXECUTE)) { return Exception("Function name %s does not exist", code->function_name); } /* okay, they can see it [and execute it] */ .... .... } does that make sense at all? l. From exarkun at intarweb.us Wed Dec 17 14:17:52 2003 From: exarkun at intarweb.us (Jp Calderone) Date: Wed Dec 17 14:20:03 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <20031217190652.GE2117@lkcl.net> References: <1071669397.3fe06095c9f3c@mcherm.com> <20031217190652.GE2117@lkcl.net> Message-ID: <20031217191752.GA5281@intarweb.us> On Wed, Dec 17, 2003 at 07:06:52PM +0000, Luke Kenneth Casson Leighton wrote: > [snip] > > i believe that it _is_ possible to express what nick coghlan > defines capabilities to be _into_ an ACL. > > how? > > by creating a security permission called "can you see it and execute it?" > let's call it SEE_AND_EXECUTE > > then, the language (interpreter) does this (following the example of the > SamrCreateUser function, above): > > MakeFunctionCall(Context *stack, CodeObject *code) > { > char *callee_fn_name = GetCalleeName(stack); > SEC_DES *sd = GetSecurityDescriptorForCode(code); > > /* check the capability to even _see_ this function! */ > if (!check_sec_des(sd, callee_fn_name, SEE_AND_EXECUTE)) > { > return Exception("Function name %s does not exist", > code->function_name); > } > > /* okay, they can see it [and execute it] */ > > .... > .... > > } > > does that make sense at all? > What appears to be missing here is the ability for a piece of code to take the existing permissions which it does posess and create new permissions within them. With the system you propose, if a function `foo' has permission to call function `bar', how does foo delegate that permission to `foobar'? If it cannot, then it becomes very difficult to write well factored code which can also exist within the security framework. Jp -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20031217/ec981992/attachment.bin From lkcl at lkcl.net Wed Dec 17 14:40:32 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Wed Dec 17 14:40:05 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) Message-ID: <20031217194031.GF2117@lkcl.net> > Are you aware of the original issue, which is that as soon as you have > a file *instance* (which might have been given to you by a very > restrictive open() variant), you can always get to the file *class* > using the __class__ attribute? Access to the __class__ attribute is > useful for all sorts of reasons. that is why i am proposing that ACLs be added to every "Entity" in language - functions, modules, attributes etc. [even if the implementation is such that they aren't _actually_ every-object-in-python-has-another-damn-pointer] in this way, you would have one ACL on the open() function which restricts it to file-read-only, another ACL on the open.__class__ attribute which would be, for the sake of argument [("all functions", DENY, ABSOLUTELY_EVERYTHING)] etc. l. From lkcl at lkcl.net Wed Dec 17 14:58:46 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Wed Dec 17 14:58:38 2003 Subject: Access Control (was Re: [Python-Dev] rexec.py unuseable) In-Reply-To: <20031217190652.GE2117@lkcl.net> References: <1071669397.3fe06095c9f3c@mcherm.com> <20031217190652.GE2117@lkcl.net> Message-ID: <20031217195846.GG2117@lkcl.net> okay, to recap. jack jansen explains the difference between capabilities and ACLs: With capabilities you have an (unforgable) right to do something and no-one cares about your identity. with ACLs you have an unforgable identity which is checked against the ACL. nick coghlan gives a definition of a capability as: If you can access it, you have permission to use it. If you aren't allowed to use it, you won't even be able to see it. nick coghlan also gives a definition of an access control list as: "Entity X is allowed to perform action Y on other Entity Z". where an "Entity" may be an object, a function, a user, a module, an identity - anything - but in the context of the python language, it is presumed that "Entity" definitely NOT include "user" or anything remotely connected to the host operating system's concept of "identity". luke leighton defines "action Y" as sufficiently general to include "seeing" as well as "doing". [the reason for such is that the purpose of capabilities may then be perceived to be a subset of the purpose of ACLs.] nick coghlan describes capabilities as a "post-hoc check" which he then concludes that, by nature, the use of capabilities is therefore fast. mike chermside succinctly explains the need for fine-grained access control down to code-section level, but he disagrees with the necessity for having lists of access controls. luke leighton endeavours to justify his opinion that capabilities are appropriate for functional languages c but completely inappropriate for object-orientated languages like python (the function() and the __class__ thing). luke leighton gives a waffly explanation of the NT security descriptor concept and provides some example usage in NT services. he also gives some example ways by which NT SDs could be adapted for use by the python interpreter. guido goes agog and type-casts this discussion as definitely python 3.0-esque. l. From lkcl at lkcl.net Wed Dec 17 15:34:43 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Wed Dec 17 15:34:18 2003 Subject: [Python-Dev] Re: rexec.py unuseable Message-ID: <20031217203443.GA3885@lkcl.net> > > > What appears to be missing here is the ability for a piece of code to take > the existing permissions which it does posess and create new permissions > within them. > > With the system you propose, if a function `foo' has permission to call > function `bar', how does foo delegate that permission to `foobar'? If it > cannot, then it becomes very difficult to write well factored code which can > also exist within the security framework. jp, i briefly touched on this in earlier messages, but have not expanded on it since. your question is therefore very useful. there should exist a permission called "sub-objects-get-a-copy-of-parent-permissions-at-sub-object-create-time" and i believe there also exists (in NT 5.0) _yet another_ permission that says "sub-objects-inherits-parent-permissions" to be honest, i don't know _all_ of the internal details of the NT 5.0 enhancements to security descriptors. if i recall correctly, the "gets-copy-of-permissions-on-create" isn't actually a separate permission but an ACE qualifier bit - a bit like the DENY and GRANT qualifications. in this way, permissions can be considered to be recursive (both at create time and at access-for-purpose-X time). the _old_ style of permissions, NT 4.0 and below, you had to MANUALLY add the permissions, recursively, to all sub-objects. and incidentally, that's one reason why you can only upgrade from an NT 4.0 NTFS to an NT 5.0 NTFS and not the other way round, because the security descriptors are jigged around and irreversibly converted. in that earlier message, i made a recommendation that the concept of "recursive" application of permissions NOT be considered for implementation (until at least the old nt4.0 style was implemented and well understood). but it _is_ important that the "sub-objects-get-a-copy-of-parent-permissions-at-sub-object-create-time" concept be implemented. otherwise, as you say, there's no way to delegate permissions. i'm assuming in the above description, perhaps incorrectly, that function 'foobar' is a member attribute of the function 'foo'? please clarify! l. From Jack.Jansen at cwi.nl Wed Dec 17 15:35:33 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Wed Dec 17 15:35:39 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) - custom Py_Initialize() In-Reply-To: References: <002301c3c29c$f2753480$0201010a@kret> Message-ID: <902A3726-30D0-11D8-B275-000A27B19B96@cwi.nl> On 17-dec-03, at 17:42, Thomas Heller wrote: > Jack Jansen writes: > >> On 17-dec-03, at 12:32, Thomas Heller wrote: >>> I would rather like a function like Py_InitializeEx(struct _init*), >>> where struct _init would have components like PYTHONPATH, optimize, >>> and >>> so on. >> >> You'd have to be very careful to design the struct in such a way that >> you'd have good compatibility. > > Sure, but it can be done. > The alternative would be a Py_InitializeN() call with many parameters. > Would that be preferrable? I'm unsure. I'm almost tempted to say I'd like to pass in a char ** and use environ-style (we're not really worried about performance here). -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From Jack.Jansen at cwi.nl Wed Dec 17 15:40:30 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Wed Dec 17 15:40:38 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> <3FE052EC.90705@iinet.net.au> <3FE0749D.2030705@iinet.net.au> <200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> Message-ID: <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> On 17-dec-03, at 18:21, Guido van Rossum wrote: > It's interesting that the "scan upwards on the package name" gets so > much support. I would actually be surprised if people who say they want "scan upward" actually want something else: they want to scan upward up to a limit. If you have a complex package with, say, three levels you may want to scan up through the levels of your own package. However, if your package is later incorporated in a third party package I can hardly imagine that you want to scan in there too. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From guido at python.org Wed Dec 17 16:00:39 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 16:00:46 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Wed, 17 Dec 2003 21:40:30 +0100." <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> <3FE052EC.90705@iinet.net.au> <3FE0749D.2030705@iinet.net.au> <200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> Message-ID: <200312172100.hBHL0dq22885@c-24-5-183-134.client.comcast.net> [Guido] > > It's interesting that the "scan upwards on the package name" gets > > so much support. [Jack] > I would actually be surprised if people who say they want "scan > upward" actually want something else: they want to scan upward up to > a limit. There's a negation missing here, but I think I understand you. > If you have a complex package with, say, three levels you may want > to scan up through the levels of your own package. However, if your > package is later incorporated in a third party package I can hardly > imagine that you want to scan in there too. I think someone actually posted a use case: they were referring to a global 3rd party module from within a package, and then later they were moving the module into the package. For both to work without changes to the import statement the scan would have to go all the way up. If we can disregard this use case, I still like my original proposal better, because it is more explicit: you have to use the right number of dots, and (like Unix paths starting with ../ or ../../ etc.) it is still completely explicit what you are referring to (assuming the current package/directory is known). But I expect the ... version will work just as well in practice: in debugged code, the ... refers to a particular ancestor, so what's above that is irrelevant; and what's below that is typically only two or at most three levels deep -- I should think a group of cooperating package authors who can't keep a namespace of that size coordinated will have bigger problems... And the advantage of the ... version is that it's more conspicuous than a single dot (for the most common case), and you don't have to count dots. --Guido van Rossum (home page: http://www.python.org/~guido/) From neal at metaslash.com Wed Dec 17 16:09:29 2003 From: neal at metaslash.com (Neal Norwitz) Date: Wed Dec 17 16:09:34 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python bltinmodule.c, 2.304, 2.305 In-Reply-To: References: Message-ID: <20031217210929.GX13702@epoch.metaslash.com> On Wed, Dec 17, 2003 at 12:43:35PM -0800, rhettinger@users.sourceforge.net wrote: > > Guido grants a Christmas wish: > sorted() becomes a regular function instead of a classmethod. Would sort() be a better name? When I told someone who's done quite a bit of java recently about sorted, he asked if it returned a bool whether the list was sorted or not. :-) Neal From python at rcn.com Wed Dec 17 16:19:27 2003 From: python at rcn.com (Raymond Hettinger) Date: Wed Dec 17 16:19:49 2003 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Python bltinmodule.c, 2.304, 2.305 In-Reply-To: <20031217210929.GX13702@epoch.metaslash.com> Message-ID: <011701c3c4e3$741ec620$f1bd958d@oemcomputer> [Raymond] > > Guido grants a Christmas wish: > > sorted() becomes a regular function instead of a classmethod. [Neal] > Would sort() be a better name? When I told someone who's done quite a > bit of java recently about sorted, he asked if it returned a bool > whether the list was sorted or not. :-) I think sorted() is better: * the context makes it clear that it's not a predicate * the name helps distinguish it from list.sort() * it fits with reversed() * looking at actual code using sorted(), it reads well my two cents, Raymond Hettinger From ws-news at gmx.at Wed Dec 17 16:35:39 2003 From: ws-news at gmx.at (Werner Schiendl) Date: Wed Dec 17 16:35:49 2003 Subject: [Python-Dev] Re: Relative import References: <200312170109.hBH19DG16699@oma.cosc.canterbury.ac.nz> <200312170116.hBH1GId21276@c-24-5-183-134.client.comcast.net> Message-ID: Hi, "Guido van Rossum" wrote > Nah. But I liked the suggestion seen here earlier of allowing > parenthesis: > > from OPAL.Modules.Financial.Reports.COGS import (generate, webOptions, > normalize, row_ops, summary) > +1 / Werner From ws-news at gmx.at Wed Dec 17 16:53:19 2003 From: ws-news at gmx.at (Werner Schiendl) Date: Wed Dec 17 16:53:27 2003 Subject: [Python-Dev] Re: Relative import References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz><3FE052EC.90705@iinet.net.au><3FE0749D.2030705@iinet.net.au><200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> <200312172100.hBHL0dq22885@c-24-5-183-134.client.comcast.net> Message-ID: "Guido van Rossum" wrote > > I think someone actually posted a use case: they were referring to a > global 3rd party module from within a package, and then later they > were moving the module into the package. For both to work without > changes to the import statement the scan would have to go all the way > up. > Why should this sort of thing work "automatically"? When I restructure a package's modules, I don't usually assume that all imports will continue to work unchanged. The main use case IMHO is to move an entire package (or self contained module) in a package to achive some sort of required structuring or, more important, resolve naming conflicts. This includes adding or re-naming some organizational package names do to company merging or aquisitions. > If we can disregard this use case, I still like my original proposal > better, because it is more explicit: you have to use the right > number of dots, and (like Unix paths starting with ../ or ../../ etc.) > it is still completely explicit what you are referring to (assuming > the current package/directory is known). > +1 I like your original proposal more than this one. There should be some solution about from-less import however, to support the fully-qualified name syntax. For example, I prefer writing "struct.pack" over writing just "pack" because it is more explicit to the reader what happens. Or e. g. os.path.split is another example. Maybe just omit the dots for the name? > But I expect the ... version will work just as well in practice: in > debugged code, the ... refers to a particular ancestor, so > what's above that is irrelevant; and what's below that is typically > only two or at most three levels deep -- I should think a group of > cooperating package authors who can't keep a namespace of that size > coordinated will have bigger problems... I'd prefer to explicitly specify the number of steps "up" the hierarchy. - It prevents accidentially taking a wrong module (e. g. a "global" one) because a module is missing. - It prevents changing semantics because someone (else) adds a module with that name on some other level of the hierarchy. - Explicit is better than implicit ;-) > > And the advantage of the ... version is that it's more conspicuous > than a single dot (for the most common case), and you don't have to > count dots. > If there are more than 2 (or maybe 3) dots in the original proposal's use, I think something is wrong with package structure - not with the syntax. And you are normally importing modules from your own package with relative imports, so there should hardly be a need to "count the dots" I think. / Werner From greg at cosc.canterbury.ac.nz Wed Dec 17 17:36:36 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 17 17:36:42 2003 Subject: [Python-Dev] Relative import In-Reply-To: <3FE052EC.90705@iinet.net.au> Message-ID: <200312172236.hBHMaaw25148@oma.cosc.canterbury.ac.nz> Nick Coghlan : > A: from __absolute__.dotted.name import foo > B: from __relative__.dotted.name import bar > C: from __heirarchy__.dotted.name import foobar > D: from dotted.name import barfoo > > Then Guido's migration plan would apply to the progression of the D > semantics from the Python 2.3 model to being a shorthand for the > absolute semantics. > > And once _that_ has happened, then we could get rid of the > '__absolute__' version in the name of TOOWTDI. That's where I disagree - I think that type C semantics should be the default/least ugly/easiest to get, not type A. But I wouldn't like any of the others to be as ugly as those, either. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From edcjones at erols.com Wed Dec 17 17:41:20 2003 From: edcjones at erols.com (Edward C. Jones) Date: Wed Dec 17 17:45:37 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning Message-ID: <3FE0DB90.9060604@erols.com> I use Gentoo 1.4 Linux with gcc 3.2.2. The C program given below gives the following compiler message: silly.c: In function `silly': silly.c:5: warning: suggest explicit braces to avoid ambiguous `else' I doubt if there is any bug in Py_DECREF. But would it be possible to jiggle the Py_DECREF macros to suppress this message? In my real code, I, of course, just added braces. ----------------------- #include static PyObject* silly(PyObject* obj, PyObject *args) { if (1) Py_DECREF(Py_None); return NULL; } static PyMethodDef silly_Methods[] = { {"silly", silly, METH_VARARGS, ""}, {NULL, NULL, 0, NULL} /* sentinel */ }; void initsilly(void) { (void) Py_InitModule("silly", silly_Methods); } ----------------------- From greg at cosc.canterbury.ac.nz Wed Dec 17 17:50:33 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 17 17:50:37 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: <1071670791.3fe0660761aa6@mcherm.com> Message-ID: <200312172250.hBHMoXt25172@oma.cosc.canterbury.ac.nz> Michael Chermside : > Greg: > > In light of this, it may have been a serious mistake to > > unify the 'file' type and the 'open' function. Is it too > > late to back out of that decision? > > Guido: > > I think so. > > I disagree. Right now, having access to a class object basically > gives one the ability to create new objects of that type. I > think that's just fine... and I don't mind applying it to the > file object. I'd think that the thing to do with untrusted code > is to deny it access to the 'file' type object, thus denying it > the ability to create new 'file's directly. It would be a lot better if we could get away from the idea of a "restricted mode" in the sense of a flag somewhere that a bunch of things have to take notice of in order to behave securely, because that model of security is prone to springing leaks -- as happened in a big way when new-style classes were introduced. The spirit behind my suggestion was to start thinking about ways in which functionality could be separated out so that this kind of special-casing for security purposes isn't needed. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From jjl at pobox.com Wed Dec 17 18:08:17 2003 From: jjl at pobox.com (John J Lee) Date: Wed Dec 17 18:08:58 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz><3FE052EC.90705@iinet.net.au><3FE0749D.2030705@iinet.net.au><200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> <200312172100.hBHL0dq22885@c-24-5-183-134.client.comcast.net> Message-ID: On Wed, 17 Dec 2003, Werner Schiendl wrote: [...] > There should be some solution about from-less import however, to support the > fully-qualified name syntax. For example, I prefer writing "struct.pack" > over writing just "pack" because it is more explicit to the reader what > happens. Or e. g. os.path.split is another example. Yes. > Maybe just omit the dots for the name? As Guido pointed out, it would be a shame to break the rule that 'import x' currently lets you refer to plain old 'x'. And it would be ugly to have '.x' appear in programs, I think (and Guido mentioned with statements, though maybe he's kidding about that). So, why not this? from ... import x John From greg at cosc.canterbury.ac.nz Wed Dec 17 18:24:12 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 17 18:24:24 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> Message-ID: <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> > import P.Q (1) > from P.Q import X (2) > > would only look for P in sys.path (I'll call this "absolute import"); > and > > from ...P.Q import X (3) > > would scan for P upwards from the current package. > > I think this addresses all the concerns I've seen brought up in this > thread so far No, it doesn't address my concern that scan-upwards semantics should be the default, in the sense of being what should be used in the absence of a reason to do otherwise, and should therefore have the most straightforward syntax. Also, you don't seem to have anything for the explicitly-relative case, which might not be strictly necessary, but I think I'd feel more comfortable if it were available somehow. Thinking about the double_underscore names, it might not be so bad if they were a bit shorter, e.g. A: __root__.P.Q B: __here__.P.Q __here__.__parent__.P.Q C: P.Q Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From jepler at unpythonic.net Wed Dec 17 18:31:26 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Wed Dec 17 18:31:19 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <20031217151450.GB28436@unpythonic.net> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> Message-ID: <20031217233126.GA4692@unpythonic.net> On Wed, Dec 17, 2003 at 09:14:50AM -0600, Jeff Epler wrote: > There are also problems on redhat9, where signals in a fork+exec'd > subprocess are blocked, for instance. This seemed to be a consequence > of blocking all signals in thread_pthread.h:PyThread_start_new_thread(). To follow up on my own message, here is a program that demonstrates the problem I ran into with threads and signals. doit() uses system() to create a process that should kill itself with a signal nearly instantly, otherwise it sleeps for one second. If doit() is called from the main thread or before any threads are created, it prints a time well under 1 second. If it is run from a thread, it takes just over a second, because the delivery of the signal in the subprocess is blocked. Typical output: $ python thread-signal-problem.py not threaded Elapsed time 0.00420594215393 subthread Elapsed time 1.00974297523 main thread Elapsed time 0.00419104099274 import thread, time, os def doit(): t = time.time() os.system("kill $$; sleep 1") t1 = time.time() print "Elapsed time", t1-t print "not threaded" doit() print "subthread" thread.start_new(doit, ()) time.sleep(2) print "main thread" doit() From tdelaney at avaya.com Wed Dec 17 18:39:48 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Wed Dec 17 18:39:54 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0D23@au3010avexu1.global.avaya.com> > From: Edward C. Jones > > I use Gentoo 1.4 Linux with gcc 3.2.2. The C program given > below gives > the following compiler message: > > silly.c: In function `silly': > silly.c:5: warning: suggest explicit braces to avoid ambiguous `else' > > ----------------------- > #include > > static PyObject* silly(PyObject* obj, PyObject *args) > { > if (1) > Py_DECREF(Py_None); If I were you, I would pay attention to the warning. > I, of course, just added braces. If you mean you changed it to: static PyObject* silly(PyObject* obj, PyObject *args) { if (1) { Py_DECREF(Py_None); } then I would say that's the right thing to do - all the time, in all code. Tim Delaney From skip at pobox.com Wed Dec 17 18:46:36 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Dec 17 18:47:14 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0D23@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0D23@au3010avexu1.global.avaya.com> Message-ID: <16352.60124.324511.476586@montanaro.dyndns.org> Tim> If you mean you changed it to: Tim> static PyObject* silly(PyObject* obj, PyObject *args) Tim> { Tim> if (1) Tim> { Tim> Py_DECREF(Py_None); Tim> } Tim> then I would say that's the right thing to do - all the time, in Tim> all code. Then you also agree we should bracify the Py_*REF macros, right? Skip From klm at zope.com Wed Dec 17 18:57:23 2003 From: klm at zope.com (Ken Manheimer) Date: Wed Dec 17 19:04:22 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> Message-ID: On Thu, 18 Dec 2003, Greg Ewing wrote: > Thinking about the double_underscore names, it might not > be so bad if they were a bit shorter, e.g. > > A: __root__.P.Q > B: __here__.P.Q > __here__.__parent__.P.Q > C: P.Q Why separate __here__ and __parent__? (I may just be confused, here.) Aren't both about getting the relative package container? Discussing schemes for explicit relative and absolute imports with some colleagues a few days ago, we liked '__pkg__' for the package containing the current module - and __pkg__.__pkg__ to mean its' container (and so on, for those non-programmers out there:-). Someone also wanted explicit expression of absolute, which we thought might be '__python__'. A: __python__.P.Q B: __pkg__.P.Q __pkg__.__pkg__.P.Q C: P.Q That said, *i* much prefer the leading '.' scheme - i think the underscores are distinctly *not* easier to read, and i'm not bothered, anyway, by the "punctuality" of the leading-'.' - i think it stands out enough to be noticable, and the fractured-path implication of it conveys the relativeness... Ken klm@zope.com From guido at python.org Wed Dec 17 19:12:09 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 19:12:16 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: Your message of "Thu, 18 Dec 2003 11:50:33 +1300." <200312172250.hBHMoXt25172@oma.cosc.canterbury.ac.nz> References: <200312172250.hBHMoXt25172@oma.cosc.canterbury.ac.nz> Message-ID: <200312180012.hBI0C9E23257@c-24-5-183-134.client.comcast.net> > It would be a lot better if we could get away from the idea > of a "restricted mode" in the sense of a flag somewhere that > a bunch of things have to take notice of in order to behave > securely, because that model of security is prone to springing > leaks -- as happened in a big way when new-style classes were > introduced. Right. Restricted mode currently uses both paradigms: you only have access to the builtins that are given to you in the __builtins__ dict -- this is pure capability stuff, and IMO it works well -- and some builtin operations behave differently when you're in restricted mode -- this is the ACL stuff, and Samuele revealed serious holes in it. > The spirit behind my suggestion was to start thinking about > ways in which functionality could be separated out so that > this kind of special-casing for security purposes isn't > needed. Right. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Dec 17 19:20:07 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 19:20:14 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 12:24:12 +1300." <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> References: <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> Message-ID: <200312180020.hBI0K7T23311@c-24-5-183-134.client.comcast.net> > > I think this addresses all the concerns I've seen brought up in this > > thread so far > > No, it doesn't address my concern that scan-upwards semantics > should be the default, in the sense of being what should be > used in the absence of a reason to do otherwise, and should > therefore have the most straightforward syntax. It doesn't address that becaseu I don't want that to be the default. I'm considering adding syntax for relative imports so that the default import syntax can be absolute only, rather than ambiguous. IMO scan-upwards as the default is worse than ambiguous. > Also, you don't seem to have anything for the explicitly-relative > case, which might not be strictly necessary, but I think > I'd feel more comfortable if it were available somehow. That's why I originally proposed single, double, triple (etc.) leading dots. > Thinking about the double_underscore names, it might not > be so bad if they were a bit shorter, e.g. I *really* don't like using double-underscore names here unless it was for extremely uncommon cases. There are two proposals that I can live with: my original proposal with the leading dots counting the number of levels up, or the triple-dot proposal with scan-up semantics. In both cases, the default semantics would switch to absolute for Python 3.0. I cannot accept scan-up as default, and I don't think we can come up with two separate clearly distinguishable non-default syntaxes to separate scan-up and explicit relative semantics, so you have to pick one there. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Wed Dec 17 19:24:02 2003 From: tim.one at comcast.net (Tim Peters) Date: Wed Dec 17 19:24:03 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <3FE0DB90.9060604@erols.com> Message-ID: [Edward C. Jones] > I use Gentoo 1.4 Linux with gcc 3.2.2. The C program given below gives > the following compiler message: > > silly.c: In function `silly': > silly.c:5: warning: suggest explicit braces to avoid ambiguous `else' > > I doubt if there is any bug in Py_DECREF. But would it be possible to > jiggle the Py_DECREF macros to suppress this message? In my real code, > I, of course, just added braces. Sounds good to me (i.e., add the braces and be done with it). It's a silly warning because Py_DECREF expands to a two-branch if/else, and there is no ambiguity in reality. It would be a refreshing change if someone suggested that gcc change instead . [Timothy Delany] > If you mean you changed it to: > > static PyObject* silly(PyObject* obj, PyObject *args) > { > if (1) > { > Py_DECREF(Py_None); > } > > then I would say that's the right thing to do - all the > time, in all code. [Skip Montanaro] > Then you also agree we should bracify the Py_*REF macros, right? > That can't work, alas -- the macro has no idea whether curly braces are syntactically legal in the context it's invoked in. Timmy *can* know that, and insert them or not accordingly. This is what leads to the bletcherous do { the real macro guts go here } while(0) form of macro definition. Py_DECREF could be rewritten in that form -- but I've puked enough for one day . From guido at python.org Wed Dec 17 19:24:13 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 19:24:20 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Wed, 17 Dec 2003 23:08:17 GMT." References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz><3FE052EC.90705@iinet.net.au><3FE0749D.2030705@iinet.net.au><200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> <200312172100.hBHL0dq22885@c-24-5-183-134.client.comcast.net> Message-ID: <200312180024.hBI0ODP23337@c-24-5-183-134.client.comcast.net> > So, why not this? > > from ... import x Yes, I forgot about that. My original proposal allowed using just one or more dots; the three-dots proposal with scan-up semantics should also allow just that. In terms of the Grammar file: dotted_name: NAME ('.' NAME)* | '...' [NAME ('.' NAME)*] BTW, it seems that for breaking up long imports, the parentheses proposal has received universal acclaim. So if we add either of the dotted-from proposals to 2.4, we should also add the parenthesized syntax: from ...... import (......), and for symmetry also without from. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Wed Dec 17 19:34:55 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Dec 17 19:34:53 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: References: <3FE0DB90.9060604@erols.com> Message-ID: <16352.63023.993326.798113@montanaro.dyndns.org> Tim> [Skip Montanaro] >> Then you also agree we should bracify the Py_*REF macros, right? >> Tim> That can't work, alas -- the macro has no idea whether curly braces Tim> are syntactically legal in the context it's invoked in. I meant the Py_*REF macros which expand to "if" statements and you'd bracify the clauses, e.g., Py_DECREF would become: #define Py_DECREF(op) \ if (_Py_DEC_REFTOTAL _Py_REF_DEBUG_COMMA \ --(op)->ob_refcnt != 0) { \ _Py_CHECK_REFCNT(op) \ } else { \ _Py_Dealloc((PyObject *)(op)) \ } Py_INCREF would be left alone, and the X variants would become #define Py_XINCREF(op) if ((op) == NULL) {;} else { Py_INCREF(op) } #define Py_XDECREF(op) if ((op) == NULL) {;} else {Py_DECREF(op) } Skip From zack at codesourcery.com Wed Dec 17 19:36:21 2003 From: zack at codesourcery.com (Zack Weinberg) Date: Wed Dec 17 19:36:41 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: (Tim Peters's message of "Wed, 17 Dec 2003 19:24:02 -0500") References: Message-ID: <87y8tbezcq.fsf@codesourcery.com> "Tim Peters" writes: > This is what leads to the bletcherous > > do { > the real macro guts go here > } while(0) > > form of macro definition. Py_DECREF could be rewritten in that form -- but > I've puked enough for one day . Should, not just could. Yeah, it's bletcherous, but it's the only way to be sure. (Short of inline functions.) zw From guido at python.org Wed Dec 17 19:41:00 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 19:41:08 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Wed, 17 Dec 2003 22:53:19 +0100." References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz><3FE052EC.90705@iinet.net.au><3FE0749D.2030705@iinet.net.au><200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> <200312172100.hBHL0dq22885@c-24-5-183-134.client.comcast.net> Message-ID: <200312180041.hBI0f1G23377@c-24-5-183-134.client.comcast.net> > > I think someone actually posted a use case: they were referring to a > > global 3rd party module from within a package, and then later they > > were moving the module into the package. For both to work without > > changes to the import statement the scan would have to go all the way > > up. > > Why should this sort of thing work "automatically"? > When I restructure a package's modules, I don't usually assume that all > imports will continue to work unchanged. > > The main use case IMHO is to move an entire package (or self contained > module) in a package to achive some sort of required structuring or, more > important, resolve naming conflicts. This includes adding or re-naming some > organizational package names do to company merging or aquisitions. I agree that the use case is not very strong. There really seem to be two camps here; some people like scan-up semantics so much they want to make it the default, others hate it and want to banish even the bastardized version of it that is the current default ("ambiguous import" or perhaps we can call it "search current then global"). > > If we can disregard this use case, I still like my original proposal > > better, because it is more explicit: you have to use the right > > number of dots, and (like Unix paths starting with ../ or ../../ etc.) > > it is still completely explicit what you are referring to (assuming > > the current package/directory is known). > > > > +1 > > I like your original proposal more than this one. I like its preciseness, but I have to admit that the three dots are enticing. > There should be some solution about from-less import however, to support the > fully-qualified name syntax. For example, I prefer writing "struct.pack" > over writing just "pack" because it is more explicit to the reader what > happens. Or e. g. os.path.split is another example. > > Maybe just omit the dots for the name? from ... import os.path could be made to work (even though today, in a from-import statement, the name after the import keyword cannot have dots in it). > > But I expect the ... version will work just as well in practice: in > > debugged code, the ... refers to a particular ancestor, so > > what's above that is irrelevant; and what's below that is typically > > only two or at most three levels deep -- I should think a group of > > cooperating package authors who can't keep a namespace of that size > > coordinated will have bigger problems... > > I'd prefer to explicitly specify the number of steps "up" the hierarchy. > > - It prevents accidentially taking a wrong module (e. g. a "global" one) > because a module is missing. But that would case pretty obvious error message, wouldn't it? > - It prevents changing semantics because someone (else) adds a module with > that name on some other level of the hierarchy. > > - Explicit is better than implicit ;-) You know, I'm torn on this one. Maybe some more people can "vote" on their preferences. > > And the advantage of the ... version is that it's more conspicuous > > than a single dot (for the most common case), and you don't have to > > count dots. > > If there are more than 2 (or maybe 3) dots in the original proposal's use, I > think something is wrong > with package structure - not with the syntax. Right. :-) > And you are normally importing modules from your own package with relative > imports, so there should hardly be a need to "count the dots" I think. Of course. --Guido van Rossum (home page: http://www.python.org/~guido/) From tdelaney at avaya.com Wed Dec 17 20:09:43 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Wed Dec 17 20:09:49 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0D79@au3010avexu1.global.avaya.com> > From: Skip Montanaro [mailto:skip@pobox.com] > > Tim> If you mean you changed it to: > > Tim> static PyObject* silly(PyObject* obj, PyObject *args) > Tim> { > Tim> if (1) > Tim> { > Tim> Py_DECREF(Py_None); > Tim> } > > Tim> then I would say that's the right thing to do - all > the time, in > Tim> all code. > > Then you also agree we should bracify the Py_*REF macros, > right? Maybe, maybe not. In this case, not bracifying the macros revealed what is IMO a poor coding practice. So you could argue that not having the braces there is a Good Thing(TM). However, I do prefer that macros be as self-contained as functions, since when reading them, they look exactly the same. The problem in the original case is that the `if` clause does not contain the braces that it should have - and as a result, can lead to ambiguous code (in terms of readability). My opinion is that no matter what code you are writing in C, blocks should be delimited appropriately. That way there's no possibility of confusion. Fortunately, I get to write the coding standards for my team, so I don't have to put up with anyone else's wrong ideas :) Tim Delaney From tdelaney at avaya.com Wed Dec 17 20:14:27 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Wed Dec 17 20:14:33 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0D7E@au3010avexu1.global.avaya.com> > From: Skip Montanaro > > #define Py_DECREF(op) \ > if (_Py_DEC_REFTOTAL _Py_REF_DEBUG_COMMA \ > --(op)->ob_refcnt != 0) { \ > _Py_CHECK_REFCNT(op) \ > } else { \ > _Py_Dealloc((PyObject *)(op)) \ > } > > Py_INCREF would be left alone, and the X variants would become > > #define Py_XINCREF(op) if ((op) == NULL) {;} else { Py_INCREF(op) } > #define Py_XDECREF(op) if ((op) == NULL) {;} else {Py_DECREF(op) } Then you will probably end up with different warnings - semicolons following closing braces, etc. As Tim Peters said, the way to deal with that is to enclose the entire bit in a do ( ... } while (0) no-op (after the compiler optimises it away) but that - as he rightly pointed out - is pretty blecherous. #define Py_XINCREF(op) do { if ((op) != NULL) { Py_INCREF(op) } } while (0) #define Py_XDECREF(op) do { if ((op) != NULL) { Py_DECREF(op) } } while (0) or #define Py_XINCREF(op) do { if ((op) != NULL) Py_INCREF(op) } while (0) #define Py_XDECREF(op) do { if ((op) != NULL) Py_DECREF(op) } while (0) Tim Delaney From greg at cosc.canterbury.ac.nz Wed Dec 17 20:57:24 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 17 20:57:30 2003 Subject: [Python-Dev] Relative import In-Reply-To: <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> Message-ID: <200312180157.hBI1vOC25998@oma.cosc.canterbury.ac.nz> Jack Jansen : > If you have a complex package with, say, three levels you may want to > scan up through the levels of your own package. However, if your package > is later incorporated in a third party package I can hardly imagine that > you want to scan in there too. The main use I have in mind for scan-upwards is for when you're importing something that's *not* part of your package. It's probably top-level, but someone might want to put you and it into a package one day, so you shouldn't hard-wire a top-level reference to it. That's why I think scan-upwards should be the default -- because you never know when someone might want to repackage you into a different environment in the future. Referring to related modules in your package is a different matter -- for that, you should probably be using explicitly relative references. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido at python.org Wed Dec 17 21:02:33 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 21:02:39 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 14:57:24 +1300." <200312180157.hBI1vOC25998@oma.cosc.canterbury.ac.nz> References: <200312180157.hBI1vOC25998@oma.cosc.canterbury.ac.nz> Message-ID: <200312180202.hBI22Xu23508@c-24-5-183-134.client.comcast.net> > The main use I have in mind for scan-upwards is for when you're > importing something that's *not* part of your package. It's probably > top-level, but someone might want to put you and it into a package one > day, so you shouldn't hard-wire a top-level reference to it. You could hardwire a relative reference to it though. > That's why I think scan-upwards should be the default -- because you > never know when someone might want to repackage you into a different > environment in the future. I don't know if I want to encourage this kind of repackaging. There are various downsides (such as possible deployment of two differently repackaged versions of the same code) and I think some of the burden should be placed on the repackager. After all, after repackaging, the repackager is really responsible for the repackaged code. The nice thing about absolute imports is that it's very simple to write a tool that changes all absolute imports of a given package into some other import, if you really want that. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Wed Dec 17 21:28:30 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 21:28:41 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <20031216141746.GA3145@burma.localdomain> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> Message-ID: <1071714510.27808.83.camel@anthem> On Tue, 2003-12-16 at 09:17, Gustavo Niemeyer wrote: > I have done this many times, so let me try to describe at least one > legitimate usage case. > > A couple of weeks ago I wrote a software which needs a third > party package to work. OTOH, since it's a small package, I don't > want to force the user to go after the package, even because I'd > have to ensure that my code always work with the available version > of that package. > > Thanks to the relative module importing mechanism, solving that is no > harder than copying the third party package into my own package > namespace. > > This idea could probably be expressed in some other way, hacking > sys.path or whatever, but I belive this is a fairly common pattern, > and I vote for introducing a scheme to differ between local/global > importing which would not break the current flexibility. I guess the difference is when you're writing an application vs. writing a library. When writing an application (like Zope or Mailman), we already manipulate sys.path to include some non-standard locations. Incorporating a third-party package into the application then is a simple matter of arranging to have its top level directory somewhere on sys.path. In Mailman, I do this with the email package, and various codecs. When writing a library, that's harder because you're imposing requirements on the downstream consumer of your library. As an application developer, I don't have much sympathy for relative imports. As a library developer, I can see the point. -Barry From barry at python.org Wed Dec 17 21:33:43 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 21:33:53 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> Message-ID: <1071714822.27808.88.camel@anthem> On Tue, 2003-12-16 at 13:26, Guido van Rossum wrote: > A lot people have presented a good case for relative imports. Nobody > has argued to keep the status quo (where imports are ambiguous about > whether they are meant to be absolute or relative). So I suggest that > in 2.4, we introduce the leading dot notation for relative import, > while still allowing relative import without a leading dot. In > 2.5 we can start warning about relative import without a leading dot > (although that will undoubtedly get complaints from folks who have > code that needs to work with 2.3 still). In 3.0 we can retire > ambiguous import. I'll just note that where the current status quo trips /me/ up most is when I accidentally have a local module with the same name as a global module, and then I write an import statement expecting to get the standard library module, but end up getting the local module. That's why when I tend to think about this, I start wanting a way to spell "definitely give me the global one, no matter what". IOW, I feel like I want a way to bypass relative module lookups. I'm still catching up on this thread, but I wanted to throw this out there... -Barry From barry at python.org Wed Dec 17 21:44:38 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 21:44:49 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <200312180041.hBI0f1G23377@c-24-5-183-134.client.comcast.net> References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> <3FE052EC.90705@iinet.net.au> <3FE0749D.2030705@iinet.net.au> <200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> <200312172100.hBHL0dq22885@c-24-5-183-134.client.comcast.net> <200312180041.hBI0f1G23377@c-24-5-183-134.client.comcast.net> Message-ID: <1071715478.27808.90.camel@anthem> On Wed, 2003-12-17 at 19:41, Guido van Rossum wrote: > You know, I'm torn on this one. Maybe some more people can "vote" on > their preferences. Don't we really need a PEP so we know what we're voting on? :) -Barry From barry at python.org Wed Dec 17 21:49:36 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 21:49:46 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180202.hBI22Xu23508@c-24-5-183-134.client.comcast.net> References: <200312180157.hBI1vOC25998@oma.cosc.canterbury.ac.nz> <200312180202.hBI22Xu23508@c-24-5-183-134.client.comcast.net> Message-ID: <1071715775.27808.95.camel@anthem> On Wed, 2003-12-17 at 21:02, Guido van Rossum wrote: > I don't know if I want to encourage this kind of repackaging. I concur. I'm sure it's just me, but supporting this repacking use case makes me very uncomfortable. It feels like I'm loosing a solid grounding about where and how Python finds modules named in import statements. -Barry From barry at python.org Wed Dec 17 21:52:03 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 21:52:10 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> References: <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> Message-ID: <1071715922.27808.98.camel@anthem> On Thu, 2003-12-18 at 07:24, Greg Ewing wrote: > Thinking about the double_underscore names, it might not > be so bad if they were a bit shorter, e.g. > > A: __root__.P.Q > B: __here__.P.Q > __here__.__parent__.P.Q > C: P.Q I like this, although I'm not so sure about __here__.__parent__. What I like about this is the explicitness of where your search starts. -Barry From barry at python.org Wed Dec 17 21:53:17 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 21:53:26 2003 Subject: [Python-Dev] Relative import In-Reply-To: References: Message-ID: <1071715997.27808.100.camel@anthem> On Wed, 2003-12-17 at 18:57, Ken Manheimer wrote: > That said, *i* much prefer the leading '.' scheme - i think the > underscores are distinctly *not* easier to read, and i'm not bothered, > anyway, by the "punctuality" of the leading-'.' - i think it stands > out enough to be noticable, and the fractured-path implication of it > conveys the relativeness... Just to continue my contrarian ways, the leading-dot thingie was one of the things that always bugged me about zcml. :) -Barry From barry at python.org Wed Dec 17 21:59:47 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 21:59:57 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <3FDEDF30.1050904@iinet.net.au> References: <200312151842.hBFIg0K17763@c-24-5-183-134.client.comcast.net> <3FDEDF30.1050904@iinet.net.au> Message-ID: <1071716386.27808.104.camel@anthem> On Tue, 2003-12-16 at 05:32, Nick Coghlan wrote: > If absolute imports were to be the only type allowed, then it would seem > that the only possible location for naming conflicts is in the _first_ > element. True. > So if I wanted to use two different third party modules, both of which > have unfortunately chosen the same name for the top-level package, the > only way to let them co-exist is to redo _all_ of the imports in one or > the other of them. > > Whereas, if relative pathing is possible, I believe that all I have to > do is push them one level down in the package heirarchy (using distinct > names that I invent), and neither of them ever even knows about the > other's existence. I can get at both of them unambiguously, by using my > new top=level names, and neither package even knows that it is no longer > starting at the top of the import heirarchy. > > Or is there some other solution being proposed to this problem, and I > just haven't understood it? Has this ever happened to you in practice? It seems like the way out would be to start adopting a Java-like convention for package names. The problem with that in current Python is that you can't (easily) weave a package's contents from different locations in the file system. -Barry From barry at python.org Wed Dec 17 22:02:04 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 22:02:15 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <3FE08A8C.305@earthlink.net> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <16349.59071.26004.502201@montanaro.dyndns.org> <1071509846.970.105.camel@anthem> <3FE08A8C.305@earthlink.net> Message-ID: <1071716523.27808.107.camel@anthem> On Wed, 2003-12-17 at 11:55, Charles Hixson wrote: > Barry Warsaw wrote: > >Sorry, let me rephrase. I'd love to ban relative imports. Personally, > >I think all imports should be specified using fully-qualified absolute > >package paths. > > > If I'm understanding this right you want to ban imports that are machine > independant. That can't be right, but that's what it sounds like to me. Nope, because "absolute package paths" are always calculated relative to directories in sys.path when mapping a package path to a file system path. -Barry From andrew-pythondev at puzzling.org Wed Dec 17 22:04:09 2003 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Wed Dec 17 22:04:15 2003 Subject: [Python-Dev] Relative import In-Reply-To: <1071714822.27808.88.camel@anthem> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> Message-ID: <20031218030409.GG473@frobozz> On Wed, Dec 17, 2003 at 09:33:43PM -0500, Barry Warsaw wrote: > > I'll just note that where the current status quo trips /me/ up most is > when I accidentally have a local module with the same name as a global > module, and then I write an import statement expecting to get the > standard library module, but end up getting the local module. > > That's why when I tend to think about this, I start wanting a way to > spell "definitely give me the global one, no matter what". IOW, I feel > like I want a way to bypass relative module lookups. Alternatively, maybe what you want is a way to say "definitely give me the standard library one, no matter what", e.g. from stdlib import codecs This would allow other packages to have their own codecs module/package without worrying about confusing import errors. To make this backwards compatible, though, we'd probably need the equivalent of C++'s "using namespace" statement, with no declaration being equivalent to: using namespace stdlib Or perhaps more python-like: __import_base__ = 'stdlib' Python 3.0 might default to __import_base__ = '', to force people to be explicit. Actually, the proposed with statement could be used here, e.g. import stdlib with stdlib: from . import codecs, user, gc looks reasonably clear to me, and for long package paths, it could be very convenient: import zope.app.interfaces with zope.app.interfaces: from .location import ILocation from .foo import IFoo, IBar # etc -Andrew. From barry at python.org Wed Dec 17 22:05:03 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 22:05:12 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0B9B@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0B9B@au3010avexu1.global.avaya.com> Message-ID: <1071716702.27808.111.camel@anthem> On Tue, 2003-12-16 at 22:20, Delaney, Timothy C (Timothy) wrote: > That's what I've been wondering. I really don't like the proposal to > use . in various places to signify relative and absolute imports. It > seems non-obvious both in understandability and readability. I agree. That's why I like something like __root__.foo.bar and __here__.foo.bar. Totally explicit and will avoid deep groove digging in the skull trying to remember if a leading dot means it should be absolute or relative. :) -Barry From tim.one at comcast.net Wed Dec 17 22:06:13 2003 From: tim.one at comcast.net (Tim Peters) Date: Wed Dec 17 22:06:14 2003 Subject: [Python-Dev] Relative import In-Reply-To: <1071714822.27808.88.camel@anthem> Message-ID: [Barry] > ... > I'll just note that where the current status quo trips /me/ up most is > when I accidentally have a local module with the same name as a global > module, and then I write an import statement expecting to get the > standard library module, but end up getting the local module. Indeed, this is common on c.l.py and the Tutor lists, with a twist: someone writes a module in their *current* directory, with a name like random.py, and then "Python breaks mysteriously all over the place", because every import of random ends up with something utterly unexpected. For some reason, "random.py" seems most often to be the actual name used in these cases. Possibly related: I remember pissing away a day a year or so ago trying to track down a segfault in a Zope test that happened only when I ran the test on my McLean box. In the end, it turned out that the test in question was (after untangling the usual layers of Zopish indirection ) deliberately trying to import a module that didn't exist, with an oddball name. It just so happened that somebody had whittled down a segfaulting Python program and uploaded it to SourceForge, with the very same oddball name. I had downloaded that program into my Python build directory, and Zope kept provoking its segfault as a side effect of (unintentionally!) importing it. In large part, I've come to loathe imports -- I never have the foggiest idea how they're going to get satisfied in code I didn't write, and especially in code that plays every trick in (and out) of the book to hide what it's really doing with sys.path mutations, import hooks, poking stuff by magic into sys.modules via well-hidden side effects in scattered __init__.py files, ... it makes Java's rigid, wordy, explicit import scheme look very attractive some days. That you can't trick that into doing arbitrarily unexpected things by magic at runtime isn't entirely a bad thing . From barry at python.org Wed Dec 17 22:08:21 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 22:08:28 2003 Subject: [Python-Dev] Relative import In-Reply-To: <20031218030409.GG473@frobozz> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> Message-ID: <1071716900.27808.114.camel@anthem> On Wed, 2003-12-17 at 22:04, Andrew Bennetts wrote: > On Wed, Dec 17, 2003 at 09:33:43PM -0500, Barry Warsaw wrote: > > > > That's why when I tend to think about this, I start wanting a way to > > spell "definitely give me the global one, no matter what". IOW, I feel > > like I want a way to bypass relative module lookups. > > Alternatively, maybe what you want is a way to say "definitely give me the > standard library one, no matter what", e.g. > > from stdlib import codecs Interesting. I see the subtle distinction. One searches only the standard library, the other searches every directory on sys.path. I'm not sure I'd have much need to restrict the search to just the standard library. Plus, would that include site-packages? -Barry From andrew-pythondev at puzzling.org Wed Dec 17 22:16:45 2003 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Wed Dec 17 22:16:51 2003 Subject: [Python-Dev] Relative import In-Reply-To: <1071716900.27808.114.camel@anthem> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> Message-ID: <20031218031645.GH473@frobozz> On Wed, Dec 17, 2003 at 10:08:21PM -0500, Barry Warsaw wrote: > On Wed, 2003-12-17 at 22:04, Andrew Bennetts wrote: > > On Wed, Dec 17, 2003 at 09:33:43PM -0500, Barry Warsaw wrote: > > > > > > That's why when I tend to think about this, I start wanting a way to > > > spell "definitely give me the global one, no matter what". IOW, I feel > > > like I want a way to bypass relative module lookups. > > > > Alternatively, maybe what you want is a way to say "definitely give me the > > standard library one, no matter what", e.g. > > > > from stdlib import codecs > > Interesting. I see the subtle distinction. One searches only the > standard library, the other searches every directory on sys.path. I'm > not sure I'd have much need to restrict the search to just the standard > library. Plus, would that include site-packages? I should have been clearer. What I'm basically proposing is to make site-package called "stdlib", leaving a default python install with a very uncluttered top-level namespace, thus reducing unexpected name conflicts. Otherwise, this idea doesn't require any modification to import semantics, except for some magic to make stdlib the default root for imports for backwards compatibility. Thinking about it a bit harder, though, I'm not sure it's as easy as I initially thought, because this would then make importing 3rd party packages not work, i.e. currently you can do: import urllib import pygtk without problems, because they are both top-level. If urllib moved into a "stdlib" package, it's hard to imagine a clean way where old code that does those two lines would still work. I still like the idea of having the standard library in its own package, but I guess that will have to wait until Python 3.0. -Andrew. From greg at cosc.canterbury.ac.nz Wed Dec 17 23:21:24 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 17 23:21:30 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180020.hBI0K7T23311@c-24-5-183-134.client.comcast.net> Message-ID: <200312180421.hBI4LNb27278@oma.cosc.canterbury.ac.nz> Guido: > I cannot accept scan-up as default In that case, it's probably better not to have it at all, and I'd go for your first proposal. One leading dot seems to stand out pretty well to my eyes -- more is not necessary. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tim at multitalents.net Wed Dec 17 23:43:45 2003 From: tim at multitalents.net (Tim Rice) Date: Wed Dec 17 23:43:52 2003 Subject: [Python-Dev] 2.3.3 portability audit Message-ID: A few things I found in the release23-maint branch. python-2.3/src/Doc/tools/checkargs.pm:#!/uns/bin/perl probably a typo. ^^^ While most python scripts use #!/usr/bin/env python this one has #!/bin/env python-2.3/src/Lib/bsddb/dbshelve.py:#!/bin/env python Most scripts that don't use #!/usr/bin/env python have #! /usr/local/bin/python These 5 have #!/usr/bin/python python-2.3/src/Lib/idlelib/idle:#!/usr/bin/python python-2.3/src/Lib/idlelib/idle.py:#!/usr/bin/python python-2.3/src/Lib/test/test_bz2.py:#!/usr/bin/python python-2.3/src/Lib/test/test_optparse.py:#!/usr/bin/python python-2.3/src/Mac/Contrib/Tabcleaner/Tabcleaner.py:#!/usr/bin/python Intentional? python-2.3/src/Lib/test/test_largefile.py:#!python This may be intentional, I don't know. python-2.3/src/Lib/test/test_pep263.py:#! -*- coding: koi8-r -*- Is this ! supposed to be there? ^ I think not. (could be wrong) python-2.3/src/Doc/tools/makesec.sh:#!/bin/bash Bash doesn't come standard on all supported platforms. The only bashism I saw with a quick look was the last line. rm -f lib.{aux,log} -- Tim Rice Multitalents (707) 887-1469 tim@multitalents.net From guido at python.org Wed Dec 17 23:48:08 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 23:48:20 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 17:21:24 +1300." <200312180421.hBI4LNb27278@oma.cosc.canterbury.ac.nz> References: <200312180421.hBI4LNb27278@oma.cosc.canterbury.ac.nz> Message-ID: <200312180448.hBI4m8J23710@c-24-5-183-134.client.comcast.net> > > I cannot accept scan-up as default > > In that case, it's probably better not to have it at all, > and I'd go for your first proposal. One leading dot seems > to stand out pretty well to my eyes -- more is not necessary. But you're okay with two or more dots for extra levels up, right? (Note that this leading-dot algorithm is nearly the same as the old MacOS pathname syntax, except they did it with colons: :foo was foo in the current directory, ::foo was foo in the parent directory, etc., and foo:bar was an absolute pathname. The only difference is that for backwards compatibility reasons they interpreted no colons as relative to the current directory rather than absolute.) --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Wed Dec 17 23:53:18 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 17 23:53:23 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180448.hBI4m8J23710@c-24-5-183-134.client.comcast.net> Message-ID: <200312180453.hBI4rIg27605@oma.cosc.canterbury.ac.nz> Guido: > But you're okay with two or more dots for extra levels up, right? Yes, that's fine. > (Note that this leading-dot algorithm is nearly the same as the old > MacOS pathname syntax I noticed that, too. Also I think VMS did something similar with dots in its pathnames. So there are precedents... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido at python.org Wed Dec 17 23:55:03 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 23:55:13 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Wed, 17 Dec 2003 21:33:43 EST." <1071714822.27808.88.camel@anthem> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> Message-ID: <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> > I'll just note that where the current status quo trips /me/ up most is > when I accidentally have a local module with the same name as a global > module, and then I write an import statement expecting to get the > standard library module, but end up getting the local module. This can be interpreted in two ways though. You could be unfortunate enough to create a module in your package that has the same name as a standard module you also need to import from inside the same package -- that's currently impossible. But the most common variant I've seen is what Tim Peters describes: having a module in the current directory that accidentally hides a standard module. The latter case does not involve packages and will not be fixed by any import syntax -- it can only be fixed by changing the default sys.path *not* to include the current directory, just like over the last decade or so Unix shells have stopped putting '.' in $PATH. But that has other downsides: it would be a lot harder to begin writing a program broken up into modules. Hmm, maybe the current directory should be last? That way you can import your own modules except if they clash with the standard library. Hm, but there are so many standard library modules that it might still cause frequent complaints from mystified beginners. I'm beginning to agree (uselessly) with Tim: import itself is a problem. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Thu Dec 18 00:04:37 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 18 00:04:43 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> Message-ID: <200312180504.hBI54ba27649@oma.cosc.canterbury.ac.nz> Guido: > Hmm, maybe the current directory should be last? That way you can > import your own modules except if they clash with the standard > library. Hm, but there are so many standard library modules that it > might still cause frequent complaints from mystified beginners. Maybe in Python 3.0 the current directory should appear as a package with some standard name. Then applications could use relative imports to import their own modules, and absolute ones to import standard modules, without fear of conflict. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From andrew-pythondev at puzzling.org Thu Dec 18 00:10:21 2003 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Thu Dec 18 00:10:31 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> Message-ID: <20031218051021.GI473@frobozz> On Wed, Dec 17, 2003 at 08:55:03PM -0800, Guido van Rossum wrote: [...] > -- that's currently impossible. But the most common variant I've seen > is what Tim Peters describes: having a module in the current directory > that accidentally hides a standard module. > > The latter case does not involve packages and will not be fixed by any > import syntax -- it can only be fixed by changing the default sys.path > *not* to include the current directory, just like over the last decade > or so Unix shells have stopped putting '.' in $PATH. Or it can be fixed by not having the entire standard library at the toplevel, but instead inside a single "stdlib" package. That way 3rd party packages could never conflict with the standard library (unless they were crazy enough to call one of their own modules "stdlib"...). This way, no matter how large the standard library grows, the potential for module name conflicts doesn't grow. It'd probably simplify things slightly, too -- the site-packages directory could go away entirely, and all packages, standard or not, would be on an equal footing. The standard library would be no more special than, say, Zope. My /usr/lib/python would just contain stdlib/ twisted/ zope/ Numeric/ etc. The backwards compatibility problems probably means this isn't feasible until Python 3.0, though :( -Andrew. From fdrake at acm.org Thu Dec 18 00:31:20 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Dec 18 00:31:32 2003 Subject: [Python-Dev] 2.3.3 portability audit In-Reply-To: References: Message-ID: <16353.15272.514696.150572@sftp.fdrake.net> Tim Rice writes: > python-2.3/src/Doc/tools/checkargs.pm:#!/uns/bin/perl > probably a typo. ^^^ ... > python-2.3/src/Doc/tools/makesec.sh:#!/bin/bash > Bash doesn't come standard on all supported platforms. > The only bashism I saw with a quick look was the last line. > rm -f lib.{aux,log} I've fixed these two on the 2.3 maintenance branch and the trunk. Thanks! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tim.one at comcast.net Thu Dec 18 00:38:57 2003 From: tim.one at comcast.net (Tim Peters) Date: Thu Dec 18 00:38:59 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180448.hBI4m8J23710@c-24-5-183-134.client.comcast.net> Message-ID: [Guido] > ... > (Note that this leading-dot algorithm is nearly the same as the old > MacOS pathname syntax, except they did it with colons: :foo was foo in > the current directory, ::foo was foo in the parent directory, etc., > and foo:bar was an absolute pathname. The only difference is that for > backwards compatibility reasons they interpreted no colons as relative > to the current directory rather than absolute.) More relevant to most of the world is that it's even more similar to the pathname syntax in the Win9x/ME command.com shells: C:\Code\python\PCbuild>cd ...\Zope C:\Code\Zope> I expect that basing Python syntax on obsolete, hated OSes is a smart contrarian move > From barry at python.org Thu Dec 18 01:49:54 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 18 01:50:04 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> Message-ID: <1071730193.17717.4.camel@anthem> On Wed, 2003-12-17 at 23:55, Guido van Rossum wrote: > This can be interpreted in two ways though. You could be unfortunate > enough to create a module in your package that has the same name as a > standard module you also need to import from inside the same package > -- that's currently impossible. Right, and I'd like to make that possible. The least disruptive way I see of doing that is to introduce a magic root package from which things can be imported from. E.g. import __root__.re > But the most common variant I've seen > is what Tim Peters describes: having a module in the current directory > that accidentally hides a standard module. Yep, I've been bit by that too. > The latter case does not involve packages and will not be fixed by any > import syntax -- it can only be fixed by changing the default sys.path > *not* to include the current directory, just like over the last decade > or so Unix shells have stopped putting '.' in $PATH. Yep. I haven't had '.' in my $PATH for a bazillion years. :) > But that has other downsides: it would be a lot harder to begin > writing a program broken up into modules. Hmm, maybe the current > directory should be last? That way you can import your own modules > except if they clash with the standard library. Hm, but there are so > many standard library modules that it might still cause frequent > complaints from mystified beginners. > > I'm beginning to agree (uselessly) with Tim: import itself is a > problem. :-) There ya go! That's thinking outside the box: let's deprecate import altogether :) -Barry From barry at python.org Thu Dec 18 01:52:32 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 18 01:52:43 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180504.hBI54ba27649@oma.cosc.canterbury.ac.nz> References: <200312180504.hBI54ba27649@oma.cosc.canterbury.ac.nz> Message-ID: <1071730352.17717.7.camel@anthem> On Thu, 2003-12-18 at 13:04, Greg Ewing wrote: > Maybe in Python 3.0 the current directory should appear as a > package with some standard name. Then applications could use > relative imports to import their own modules, and absolute > ones to import standard modules, without fear of conflict. In a previous thread, we already came up with __me__ as a magic name referring to the current module. Let's use __my__ as the prefix for importing a relative module. too-tired-to-wink-ly y'rs, -Barry From art at wiktorsadowski.com Thu Dec 18 02:38:07 2003 From: art at wiktorsadowski.com (Wiktor Sadowski) Date: Thu Dec 18 02:37:46 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) - custom Py_Initialize() Message-ID: <028401c3c539$e1e3db40$0201010a@kret> Thomas,consider this: specialized_python.exe --path/foo.px (command line) where foo.px is zipped ctypes-venster program -modules,some code,binaries(images etc.) specialized_python.exe could be statically linked against C ctypes,and could contain ctypes and venster python code (freezed) (specialized_python.exe could also work as "regular" python.exe and successfully run ".py" , ".pyc" files) Now associate ".px" with specialized_python.exe and you have clickable packed python programs. To build such specialized_python.exe you would need your own PyRun_*NotSoSimple*FileExFlags , new Py_Main, modified Py_Initialize and probably new or modified import engine. And with proposed addition specialized_python.exe could use existing Python shared library! I'm afraid it wouldn't be possible with Py_InitializeEx, you are proposing. For one of our clients we created pymozilla.exe to run packed websites(zipped html,xml,css,images and python "cgi" files) It's pretty cool. :-) Current Py_Initialize() is ok for distributed Python framework , but it's a pain if you'd like to use the language and not necessary each aspect of the framework. Why not make Python shared library more usable? Wiktor From fincher.8 at osu.edu Thu Dec 18 03:51:18 2003 From: fincher.8 at osu.edu (Jeremy Fincher) Date: Thu Dec 18 02:53:50 2003 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Python bltinmodule.c, 2.304, 2.305 In-Reply-To: <011701c3c4e3$741ec620$f1bd958d@oemcomputer> References: <011701c3c4e3$741ec620$f1bd958d@oemcomputer> Message-ID: <200312180351.18465.fincher.8@osu.edu> On Wednesday 17 December 2003 04:19 pm, Raymond Hettinger wrote: > I think sorted() is better: > > * the context makes it clear that it's not a predicate > * the name helps distinguish it from list.sort() > * it fits with reversed() > * looking at actual code using sorted(), it reads well sorted() returns a list, and reversed() returns an iterator. That seems like a recipe for confusion, *especially* because they "fit" together and do completely different things. Jeremy From martin at v.loewis.de Wed Dec 17 16:30:23 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 18 02:57:30 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031216212312.GL17021@lkcl.net> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> <20031216212312.GL17021@lkcl.net> Message-ID: Luke Kenneth Casson Leighton writes: > all i can do is recommend a framework and some guidelines on what > conventions could be fitted over that framework. In that case, your proposal will likely die. I don't think the framework can provide the feature that rexec.py originally provided. To prove that, I would need a complete proposal how precisely what ACLs are set on what objects, and how exactly I invoke code for restricted execution (i.e. what API do I call in what order). As you won't provide such a proposal, and as nobody else might, either, it is likely that this gets us nowhere. Regards, Martin From tim.one at comcast.net Thu Dec 18 03:21:28 2003 From: tim.one at comcast.net (Tim Peters) Date: Thu Dec 18 03:21:29 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> Message-ID: [Guido] > ... > I'm beginning to agree (uselessly) with Tim: import itself is a > problem. :-) Not useless at all -- that's bonding! Technically useless but humanly essential . It's no mystery *why* import is "a problem", but I'm not sure insight implies a cure: it's one of the very few places in Python where you can't explicitly say what you intend. Even in an inheritance hierarchy from hell, if I really don't want to chance that please.hit_me() will get satisfied from a class other than BlackJack, I can force it easily -- BlackJack.hit_me(please) There aren't any names to spell "the roots" of all the places Python may search to satisfy an import, so no way either to be explicit about intent. In contrast, nobody can mistake what import java.util.Random; intends. It's not a deep thing, it's just being able to name the intended context. I confess I'm keener on that than on growing more ways to spell more kinds of searches. From lkcl at lkcl.net Thu Dec 18 04:29:22 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Thu Dec 18 04:29:36 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> <20031216212312.GL17021@lkcl.net> Message-ID: <20031218092922.GF6324@lkcl.net> On Wed, Dec 17, 2003 at 10:30:23PM +0100, Martin v. L?wis wrote: > Luke Kenneth Casson Leighton writes: > > > all i can do is recommend a framework and some guidelines on what > > conventions could be fitted over that framework. > martin, to clarify: "all i can do" is the wrong preamble phrase. "i would like to" is better. i was trying to be ... self-denigrating, over-careful, something like that. > originally provided. To prove that, I would need a complete proposal > how precisely what ACLs are set on what objects, and how exactly I > invoke code for restricted execution (i.e. what API do I call in what > order). what api, in what order, what code is invoked in (inside the python executable), _how_ ACLs are set on what objects, yes. _what_ acls are set on what objects is conditional on whether i receive funding to do so, or whether some other people can be of significant assistance. why? because there's a lot of them. what i was trying to say above, "all i can do is recommend a framework" is build up to an idea of adding in a framework, similar to Exceptions, by which ANY generically-defined restriction system can be plugged in to the python language. the idea being that if no such a system is not plugged in, the performance penalty on python is insignificant, and no user-code is restricted. however, this may all turn out to be unnecessary [quote from greg ewing]: > The spirit behind my suggestion was to start thinking about > ways in which functionality could be separated out so that > this kind of special-casing for security purposes isn't > needed. so, with the correct codebase reordering, a simple capabilities based system can be added, the problem goes away. yes? l. From martin at v.loewis.de Wed Dec 17 17:59:09 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 18 04:44:03 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <3FE0DB90.9060604@erols.com> References: <3FE0DB90.9060604@erols.com> Message-ID: "Edward C. Jones" writes: > silly.c: In function `silly': > silly.c:5: warning: suggest explicit braces to avoid ambiguous `else' > > I doubt if there is any bug in Py_DECREF. But would it be possible to > jiggle the Py_DECREF macros to suppress this message? It would be possible. Can you please submit a bug report? Or, better yet, a patch? Regards, Martin From anthony at interlink.com.au Thu Dec 18 04:50:50 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Dec 18 04:51:09 2003 Subject: [Python-Dev] 2.3.3 and beyond Message-ID: <200312180950.hBI9ooi1005762@localhost.localdomain> This should hopefully be no suprise to anyone, but I'm going to be cutting 2.3.3 final in about 12-15 hours time. Once that's done, I'll be handing the branch over to Jack so he can do his Macintosh magic. I _had_ hoped to spend a lot of time getting various platform issues resolved for 2.3.3 - this didn't end up happening (for reasons that ryhme with "junkies are scum"). Once Jack's done his Mac magic, I'll try to throw some effort at this for 2.3.4. I'm planning a 2.3.4 for around the May timeframe, assuming Tim doesn't create more stinking critical bugs in the meantime. (Oh, sure, he'll _claim_ the bugs were always there, and he was just fixing them... ) Anthony -- Anthony Baxter It's never too late to have a happy childhood. From ncoghlan at iinet.net.au Thu Dec 18 05:42:02 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu Dec 18 05:42:09 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <1071716386.27808.104.camel@anthem> References: <200312151842.hBFIg0K17763@c-24-5-183-134.client.comcast.net> <3FDEDF30.1050904@iinet.net.au> <1071716386.27808.104.camel@anthem> Message-ID: <3FE1847A.7030104@iinet.net.au> Barry Warsaw wrote: > On Tue, 2003-12-16 at 05:32, Nick Coghlan wrote: >> [snip description of potential naming conflicts] > Has this ever happened to you in practice? Well, no. But others seemed concerned about it, so I thought I'd try to clarify the problem - the scripts I write don't need anything more than the standard library and win32all. > It seems like the way out would be to start adopting a Java-like > convention for package names. The problem with that in current Python > is that you can't (easily) weave a package's contents from different > locations in the file system. I still like the various suggestions for _giving_ those parts of the system special names. That's what I was trying for, even if the names I chame up with weren't any good. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at iinet.net.au Thu Dec 18 05:58:42 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu Dec 18 05:58:48 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: <200312180012.hBI0C9E23257@c-24-5-183-134.client.comcast.net> References: <200312172250.hBHMoXt25172@oma.cosc.canterbury.ac.nz> <200312180012.hBI0C9E23257@c-24-5-183-134.client.comcast.net> Message-ID: <3FE18862.50208@iinet.net.au> Guido van Rossum wrote: >>It would be a lot better if we could get away from the idea >>of a "restricted mode" in the sense of a flag somewhere that >>a bunch of things have to take notice of in order to behave >>securely, because that model of security is prone to springing >>leaks -- as happened in a big way when new-style classes were >>introduced. > > > Right. Restricted mode currently uses both paradigms: you only have > access to the builtins that are given to you in the __builtins__ dict > -- this is pure capability stuff, and IMO it works well -- and some > builtin operations behave differently when you're in restricted mode > -- this is the ACL stuff, and Samuele revealed serious holes in it. What if instead of 'builtin behaves differently in restricted mode' we had 'restricted __builtins__ contains a DIFFERENT builtin, that happens to have the same name'? That is, in addition to the ability to simply deny access to a specific builtin function or class, there was the ability to _replace_ one before giving it to the restricted code. Regards, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From paoloinvernizzi at dmsware.com Thu Dec 18 06:03:38 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Thu Dec 18 06:03:45 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: References: <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> Message-ID: Tim Peters wrote: > In contrast, nobody can mistake what > > import java.util.Random; > > intends. It's not a deep thing, it's just being able to name the intended > context. I confess I'm keener on that than on growing more ways to spell > more kinds of searches. I *strongly* agree!!! --- Paolo Invernizzi From mwh at python.net Thu Dec 18 08:00:25 2003 From: mwh at python.net (Michael Hudson) Date: Thu Dec 18 08:00:34 2003 Subject: [Python-Dev] bug in python arm-linux?: start_new_thread fails after popen In-Reply-To: <20031217151450.GB28436@unpythonic.net> (Jeff Epler's message of "Wed, 17 Dec 2003 09:14:50 -0600") References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> Message-ID: <2my8taffh2.fsf@starship.python.net> Jeff Epler writes: > On Wed, Dec 17, 2003 at 03:01:00PM +0100, Michael Lauer wrote: >> Hi, I have a python program which works fine on x86 but doesn't work on >> any of my arm-linux devices (Zaurus, Ipaq, SIMpad) - all of them are >> running Python 2.3.2 on top of glibc 2.3.2+linuxthreads on top of kernel >> 2.4.18-rmk6-pxa3 respectively kernel 2.4.19-rmk7. > > Using threads and fork together seems to be a big smelly armpit in Python. > There are also problems on redhat9, where signals in a fork+exec'd > subprocess are blocked, for instance. This is in no way restricted to RH9... > This seemed to be a consequence of blocking all signals in > thread_pthread.h:PyThread_start_new_thread(). Perhaps > pthread_atfork() Perhaps. Cheers, mwh -- I have no disaster recovery plan for black holes, I'm afraid. Also please be aware that if it one looks imminent I will be out rioting and setting fire to McDonalds (always wanted to do that) and probably not reading email anyway. -- Dan Barlow From barry at python.org Thu Dec 18 08:00:29 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 18 08:00:44 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <3FE1847A.7030104@iinet.net.au> References: <200312151842.hBFIg0K17763@c-24-5-183-134.client.comcast.net> <3FDEDF30.1050904@iinet.net.au> <1071716386.27808.104.camel@anthem> <3FE1847A.7030104@iinet.net.au> Message-ID: <1071752429.17717.24.camel@anthem> On Thu, 2003-12-18 at 05:42, Nick Coghlan wrote: > Barry Warsaw wrote: > > On Tue, 2003-12-16 at 05:32, Nick Coghlan wrote: > >> [snip description of potential naming conflicts] > > Has this ever happened to you in practice? > > Well, no. But others seemed concerned about it, so I thought I'd try to > clarify the problem - the scripts I write don't need anything more than > the standard library and win32all. I understand. The reason I ask is because it's been A Concern for as long as I've used Python, but in practice it's never really been an issue. -Barry From skip at pobox.com Thu Dec 18 08:10:58 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 18 08:11:00 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180453.hBI4rIg27605@oma.cosc.canterbury.ac.nz> References: <200312180448.hBI4m8J23710@c-24-5-183-134.client.comcast.net> <200312180453.hBI4rIg27605@oma.cosc.canterbury.ac.nz> Message-ID: <16353.42850.434223.45725@montanaro.dyndns.org> Greg> Also I think VMS did something similar with dots in its Greg> pathnames. So there are precedents... Yes, but the dots were kind of hard to miss what with the directory portion of a path enclosed in [ ... ] Skip From mcherm at mcherm.com Thu Dec 18 08:15:24 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Thu Dec 18 08:15:26 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) Message-ID: <1071753324.3fe1a86c3827e@mcherm.com> [Michael Chermside] > ... INSTANCES of 'file' and could access > f.__class__. [...] so for restricted code we return something > like this: > > class FakeFile: > def __eq__(self, other): > return other == file > [Guido] > Are you aware of the original issue, which is that as soon as you have > a file *instance* (which might have been given to you by a very > restrictive open() variant), you can always get to the file *class* > using the __class__ attribute? Access to the __class__ attribute is > useful for all sorts of reasons. Yes, I'm aware of that issue... in fact that's what I'm trying to address here, but I'm obviously not doing a good job of expressing myself. I was thinking of the restricted code having access to an instance of 'file' named 'myFileInstance'. This instance has peculiar behavior when you access 'myFileInstance.__class__'. Instead of returning the class object 'file', it returns a class object like 'FakeFile' (which I *should* have made a subclass of 'type'). This whole plan is based on two misconceptions. The first is that access to __class__ is the only way to use introspection to access things of greater power (like the class) from an instance. It's NOT the only way, I'm just hoping that the others can be fully enumerated (somewhat shaky, but perhaps possible) and can be fixed in similar ways (maybe true). The other misconception (apparently) is that the only real USE for accessing the __class__ attribute is for testing class membership. You said above that it's useful "for all sorts of reasons"... I'm wondering what they are. And yes... I *also* realize that there's not a whole lot of difference between an instance which behaves oddly under introspection and one which is wrapped in a proxy. But I realize that providing a secure proxy is somewhat difficult and I'm wondering if the other approach is more useful. [Guido] > Yes, even for 3.0 this is still a dream... True enough. So feel free to just drop the issue if you've got better (ie, more immediate) things to spend your time on. I wouldn't want to see you get a pie by default because there wasn't time to complete a test suite! -- Michael Chermside From mwh at python.net Thu Dec 18 08:27:58 2003 From: mwh at python.net (Michael Hudson) Date: Thu Dec 18 08:28:01 2003 Subject: [Python-Dev] Relative import In-Reply-To: <20031218031645.GH473@frobozz> (Andrew Bennetts's message of "Thu, 18 Dec 2003 14:16:45 +1100") References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> Message-ID: <2mu13yfe75.fsf@starship.python.net> Andrew Bennetts writes: > I still like the idea of having the standard library in its own > package, but I guess that will have to wait until Python 3.0. Guido tends to shoot people who suggest this, watch out :-) Cheers, mwh -- Lisp nearing the age of 50 is the most modern language out there. GC, dynamic, reflective, the best OO model extant including GFs, procedural macros, and the only thing old-fashioned about it is that it is compiled and fast. -- Kenny Tilton, comp.lang.python From mwh at python.net Thu Dec 18 08:42:25 2003 From: mwh at python.net (Michael Hudson) Date: Thu Dec 18 08:42:28 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python bltinmodule.c, 2.304, 2.305 In-Reply-To: <20031217210929.GX13702@epoch.metaslash.com> (Neal Norwitz's message of "Wed, 17 Dec 2003 16:09:29 -0500") References: <20031217210929.GX13702@epoch.metaslash.com> Message-ID: <2mptemfdj2.fsf@starship.python.net> Neal Norwitz writes: > On Wed, Dec 17, 2003 at 12:43:35PM -0800, rhettinger@users.sourceforge.net wrote: >> >> Guido grants a Christmas wish: >> sorted() becomes a regular function instead of a classmethod. > > Would sort() be a better name? No (). > When I told someone who's done quite a bit of java recently about > sorted, he asked if it returned a bool whether the list was sorted > or not. :-) In Python, that would be issorted (please ignore callable ). I'd expect sort() to mutate its argument. Cheers, mwh -- "The future" has arrived but they forgot to update the docs. -- R. David Murray, 9 May 2000 From skip at pobox.com Thu Dec 18 08:50:12 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 18 08:50:14 2003 Subject: [Python-Dev] Relative import In-Reply-To: <20031218051021.GI473@frobozz> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> <20031218051021.GI473@frobozz> Message-ID: <16353.45204.728479.296988@montanaro.dyndns.org> Andrew> Or it can be fixed by not having the entire standard library at Andrew> the toplevel, but instead inside a single "stdlib" package. ... Andrew> The backwards compatibility problems probably means this isn't Andrew> feasible until Python 3.0, though :( Not necessarily. It has been proposed (more than once I think) that such a stdlib package could be created in parallel using symlinks. At its simplest, I think all you'd need would be a stdlib directory, a nearly empty __init__.py file and symlinks to all the current standard modules and packages. (The __init__.py file would take care of importing builtin modules.) I would choose a very short name for such a package ("std" seems best) even though it might create clashes because it will be used a lot. Skip From mwh at python.net Thu Dec 18 09:01:52 2003 From: mwh at python.net (Michael Hudson) Date: Thu Dec 18 09:01:56 2003 Subject: [Python-Dev] Relative import In-Reply-To: <16353.45204.728479.296988@montanaro.dyndns.org> (Skip Montanaro's message of "Thu, 18 Dec 2003 07:50:12 -0600") References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> <20031218051021.GI473@frobozz> <16353.45204.728479.296988@montanaro.dyndns.org> Message-ID: <2mfzfifcmn.fsf@starship.python.net> Skip Montanaro writes: > Andrew> Or it can be fixed by not having the entire standard library at > Andrew> the toplevel, but instead inside a single "stdlib" package. > ... > Andrew> The backwards compatibility problems probably means this isn't > Andrew> feasible until Python 3.0, though :( > > Not necessarily. It has been proposed (more than once I think) that such a > stdlib package could be created in parallel using symlinks. At its > simplest, I think all you'd need would be a stdlib directory, a nearly empty > __init__.py file and symlinks to all the current standard modules and > packages. (The __init__.py file would take care of importing builtin > modules.) Are you sure that you won't get multiple copies of the same module floating around? E.g. will you have from httplib import HTTPConnection as HC1 from std.httplib import HTTPConnection as HC2 HC1 is HC2 I think in the scheme sketched above this will be false, which kills the idea stone dead. I might be wrong, though. Cheers, mwh -- Counting lines is probably a good idea if you want to print it out and are short on paper, but I fail to see the purpose otherwise. -- Erik Naggum, comp.lang.lisp From pf_moore at yahoo.co.uk Thu Dec 18 09:03:26 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Thu Dec 18 09:03:13 2003 Subject: [Python-Dev] Re: Relative import References: <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> <200312180020.hBI0K7T23311@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: > There are two proposals that I can live with: my original proposal > with the leading dots counting the number of levels up, or the > triple-dot proposal with scan-up semantics. In both cases, the > default semantics would switch to absolute for Python 3.0. If that's what you;d like a vote on, I prefer the triple-dot proposal. As far as semantics go, I have no opinion (I don't have any use for either option) but I *really* dislike the look of the single dot. I'm not a great fan of the triple dot, but it's the lesser of the two evils. I have to admit to some confusion here, though. At the moment, the following works: pkg\__init__.py: print "importing pkg" import a pkg\a.py: print "importing a" import b pkg\b.py: print "importing b" >>> import pkg importing pkg importing a importing b Is the proposal that any of this *stop* working? I assume not. Otherwise, I'm -1 on the whole thing. I see no reason to break this usage, and good reason (I can rename pkg - before final release, obviously - without needing to do a search-and-replace edit on everything in the package). I think this whole thing needs a PEP, though. If only to be completely clear on what is and is not changing, and to record discarded options. Paul. -- This signature intentionally left blank From paoloinvernizzi at dmsware.com Thu Dec 18 09:19:56 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Thu Dec 18 09:20:06 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <16353.45204.728479.296988@montanaro.dyndns.org> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> <20031218051021.GI473@frobozz> <16353.45204.728479.296988@montanaro.dyndns.org> Message-ID: Skip Montanaro wrote: > It has been proposed (more than once I think) that such a > stdlib package could be created in parallel using symlinks. Symlink is one of the things that I would like to have on my XP box... :-\ --- Paolo Invernizzi From pf_moore at yahoo.co.uk Thu Dec 18 09:40:30 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Thu Dec 18 09:40:48 2003 Subject: [Python-Dev] Re: Relative import References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> Message-ID: Barry Warsaw writes: > On Tue, 2003-12-16 at 13:26, Guido van Rossum wrote: >> A lot people have presented a good case for relative imports. Nobody >> has argued to keep the status quo (where imports are ambiguous about >> whether they are meant to be absolute or relative). So I suggest that >> in 2.4, we introduce the leading dot notation for relative import, >> while still allowing relative import without a leading dot. In >> 2.5 we can start warning about relative import without a leading dot >> (although that will undoubtedly get complaints from folks who have >> code that needs to work with 2.3 still). In 3.0 we can retire >> ambiguous import. > > I'll just note that where the current status quo trips /me/ up most is > when I accidentally have a local module with the same name as a global > module, and then I write an import statement expecting to get the > standard library module, but end up getting the local module. > > That's why when I tend to think about this, I start wanting a way to > spell "definitely give me the global one, no matter what". IOW, I feel > like I want a way to bypass relative module lookups. That issue I can understand. And I agree there should be a way to state it explicitly. One of the issues here is that this area is a bit under-documented (I know, I should read the source, but I don't have the time right now). At the moment, there are two cases: 1. Import from sys.path. This is what is being called an "absolute" import, and is nice and easy to understand. The key issue is that there is no way to *force* this interpretation in the face of option (2) below. 2. Import from "the package". This is the under-documented bit, but if I understand it correctly, it's basically that from within a module contained in a package, sys.path is conceptually *extended* to include the package's __path__ (which by default contains the directory of the package, but which can be user-modified). Now the big problem here is that behaviour (2) is useful. Simple "relative" imports of one module within a package from another module in the same package are common. Guido's (IMHO ugly) "dot" syntax handles that, by making users explicitly request option (2), and making the current import syntax *only* mean (1). But none of the proposed solutions handle the __path__ variable. I don't have any objection in principle to desupporting __path__ (heck, it would have made thinking about PEP 302 easier, if nothing else) but (a) it would need a deprecation warning, and (b) Guido himself offered a use case in . This definitely needs a PEP. If we're removing support for __path__, the implications need to be thought through (PEP 302, the pkgutil module, etc etc). If we're not, none of the proposals so far have covered how __path__ gets supported in future. A much simpler proposal, just providing an explicit way of saying "Import from sys.path *only*" may be OK without a PEP. But even then, I'd suspect we should have a PEP explaining *why* it has to be this simple. Go on, Guido. We'll be gentle if you write a PEP, and we won't set c.l.p on you :-) Paul. -- This signature intentionally left blank From skip at pobox.com Thu Dec 18 09:49:39 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 18 09:49:46 2003 Subject: [Python-Dev] Relative import In-Reply-To: <2mfzfifcmn.fsf@starship.python.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> <20031218051021.GI473@frobozz> <16353.45204.728479.296988@montanaro.dyndns.org> <2mfzfifcmn.fsf@starship.python.net> Message-ID: <16353.48771.531590.7792@montanaro.dyndns.org> >> It has been proposed (more than once I think) that such a stdlib >> package could be created in parallel using symlinks.... Michael> Are you sure that you won't get multiple copies of the same Michael> module floating around? E.g. will you have Michael> from httplib import HTTPConnection as HC1 Michael> from std.httplib import HTTPConnection as HC2 Michael> HC1 is HC2 Michael> I think in the scheme sketched above this will be false, which Michael> kills the idea stone dead. Maybe the symlink idea won't work. It seems that it would work to import the standard modules from std/__init__.py though. I performed a simple experiment. I created the std directory, added a symlink in it for the random module and added an __init__.py file with this content: import sys import urlparse Here's the result: >>> import random >>> from std import random as random2 >>> random is random2 False >>> import urlparse >>> from std import urlparse as urlparse2 >>> urlparse is urlparse2 True >>> import sys >>> from std import sys as sys2 >>> sys is sys2 True If we added a significant number of modules to std/__init__.py, startup would slow to a crawl. I imagine some sort of delayed import mechanism could be crafted to avoid this problem though. Skip From lists at hlabs.spb.ru Thu Dec 18 12:52:31 2003 From: lists at hlabs.spb.ru (Dmitry Vasiliev) Date: Thu Dec 18 09:52:47 2003 Subject: [Python-Dev] Importing packages from command line Message-ID: <3FE1E95F.7080209@hlabs.spb.ru> Just an idea... 1. python -p package_name Equivalent to: import package_name 2. python -p package.zip Equivalent to: import sys sys.insert(0, "package.zip") import package -- Dmitry Vasiliev (dima at hlabs.spb.ru) From skip at pobox.com Thu Dec 18 09:55:12 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 18 09:55:18 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> <20031218051021.GI473@frobozz> <16353.45204.728479.296988@montanaro.dyndns.org> Message-ID: <16353.49104.97549.745804@montanaro.dyndns.org> Paolo> Symlink is one of the things that I would like to have on my XP Paolo> box... As Michael Hudson suggests, symlinks probably won't work anyway. I have something which looks like a symlink on my Win2k box. Windows calls it a Shortcut in listings. Is that not roughly the same thing as a symlink? Skip From fdrake at acm.org Thu Dec 18 10:16:15 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Dec 18 10:16:37 2003 Subject: [Python-Dev] Doc/ tree closed on 2.3 branch Message-ID: <16353.50367.49731.885916@sftp.fdrake.net> Python's Doc/ tree is closed on the release23-maint branch. Commits may still be made on the trunk. The final 2.3.3 documentation will appear in the development area on python.org shortly: http://www.python.org/dev/doc/ The branch will be open once the 2.3.3 release has been completed. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From paoloinvernizzi at dmsware.com Thu Dec 18 10:18:22 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Thu Dec 18 10:22:14 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <16353.49104.97549.745804@montanaro.dyndns.org> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> <20031218051021.GI473@frobozz> <16353.45204.728479.296988@montanaro.dyndns.org> <16353.49104.97549.745804@montanaro.dyndns.org> Message-ID: <3FE1C53E.1080507@dmsware.com> Skip Montanaro wrote: >I have >something which looks like a symlink on my Win2k box. Windows calls it a >Shortcut in listings. Is that not roughly the same thing as a symlink? > > No, the NT file system as nothing as flexible as a symlink. If we are speaking about symlinks to files, windows shortcuts are barely usable only for pointing to executable for execution in a console (but they have a lot of redirection problems). If you *open* a shortcut you are actually opening the shortcut itself, not the thing it's pointing to ;-( Shortcuts to a directory is only usable by Explorer-like-programs, I don't know any usable operation with them from the console... The only way to use shortcut in python is to turn python shortcut-aware in the import machinery... --- Paolo Invernizzi Ps... cygwin is another story.... From guido at python.org Thu Dec 18 10:38:03 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 18 10:38:09 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: Your message of "Thu, 18 Dec 2003 20:58:42 +1000." <3FE18862.50208@iinet.net.au> References: <200312172250.hBHMoXt25172@oma.cosc.canterbury.ac.nz> <200312180012.hBI0C9E23257@c-24-5-183-134.client.comcast.net> <3FE18862.50208@iinet.net.au> Message-ID: <200312181538.hBIFc3x25070@c-24-5-183-134.client.comcast.net> > >>It would be a lot better if we could get away from the idea > >>of a "restricted mode" in the sense of a flag somewhere that > >>a bunch of things have to take notice of in order to behave > >>securely, because that model of security is prone to springing > >>leaks -- as happened in a big way when new-style classes were > >>introduced. > > > > Right. Restricted mode currently uses both paradigms: you only have > > access to the builtins that are given to you in the __builtins__ dict > > -- this is pure capability stuff, and IMO it works well -- and some > > builtin operations behave differently when you're in restricted mode > > -- this is the ACL stuff, and Samuele revealed serious holes in it. > > What if instead of 'builtin behaves differently in restricted mode' we > had 'restricted __builtins__ contains a DIFFERENT builtin, that happens > to have the same name'? > > That is, in addition to the ability to simply deny access to a specific > builtin function or class, there was the ability to _replace_ one before > giving it to the restricted code. Oh, there is, of course. Sorry, I wasn't clear. There are a few operations that are *not* spelled as built-in function calls that have to be restricted. For example, every object supports getting obj.__dict__ -- but in restricted mode you don't want to allow this. The Python source currently checks for "was I called from restricted mode" to deny such operations. The holes that Samuele revealed had to do with tricking unrestricted code into invoking unsafe things with arbitrary arguments controlled by the restricted code. --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh at python.net Thu Dec 18 10:45:52 2003 From: mwh at python.net (Michael Hudson) Date: Thu Dec 18 10:47:02 2003 Subject: [Python-Dev] speeding function calls (a little) In-Reply-To: <1067487850.24165.53.camel@localhost.localdomain> Message-ID: <425EA205-3171-11D8-8B62-0003931DF95C@python.net> On Thursday, Oct 30, 2003, at 05:24 Europe/Stockholm, Jeremy Hylton wrote: [function calls] > There is an optimization that depends on having no default arguments > (or > keyword arguments or free variables). Why does it depend on not having default arguments? If you supply the right number of arguments (something that's obviously already checked) why does the function having defaults make a jot of difference? > It copies the arguments directly > from the caller's frame into the callee's frame without creating an > argument tuple. > > It's interesting to avoid the copy from caller to callee, but I don't > think it's a big cost relative to everything else we're doing to set up > a frame for calling. (I expect the number of arguments is usually > small.) You would need some way to encode what variables are loaded > from the caller stack and what variables are loaded from the current > frame. Either a different opcode or some kind of flag in the current > LOAD/STORE argument. As I think Phillip managed to convince himself recently, some kind of JIT functionality seems to be needed to do function calls really efficiently. I wonder if libffi does enough... it would be nice if the body of CALL_FUNCTION could look a bit like this: x = POP() PUSH(((some_cast_or_other)x)(f, stack_pointer, oparg)) Gah, this doesn't really seem to work out, on thinking about it. Wins seem more likely to come from knowing with some certainly at the call site that you've not messed the arguments up (and so we're back to wanting a JIT, it seems to me). > One other possibility for optimization is this XXX comment in > fast_function(): > /* XXX Perhaps we should create a specialized > PyFrame_New() that doesn't take locals, but does > take builtins without sanity checking them. > */ > f = PyFrame_New(tstate, co, globals, NULL); > > PyFrame_New() does a fair amount of work that is unnecessary in the > common case. Fair amount? I have a patch that gets ~1.5% on pystone along these lines, but it's a bit scary (makes a "lightweight-frame" subclass that assumes more about things on it's freelist, and so on). I'm not sure the modest gains are worth the complexity, but I'll post it to SF... Cheers, mwh From guido at python.org Thu Dec 18 10:50:44 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 18 10:50:48 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 14:03:26 GMT." References: <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> <200312180020.hBI0K7T23311@c-24-5-183-134.client.comcast.net> Message-ID: <200312181550.hBIFoiN25162@c-24-5-183-134.client.comcast.net> > If that's what you;d like a vote on, I prefer the triple-dot proposal. > As far as semantics go, I have no opinion (I don't have any use for > either option) but I *really* dislike the look of the single dot. I'm > not a great fan of the triple dot, but it's the lesser of the two > evils. Your vote doesn't help me; the triple dot proposal is semantically different from the single dot. > I have to admit to some confusion here, though. At the moment, the > following works: > > pkg\__init__.py: > print "importing pkg" > import a > pkg\a.py: > print "importing a" > import b > pkg\b.py: > print "importing b" > > >>> import pkg > importing pkg > importing a > importing b > > Is the proposal that any of this *stop* working? I assume not. > Otherwise, I'm -1 on the whole thing. I see no reason to break this > usage, and good reason (I can rename pkg - before final release, > obviously - without needing to do a search-and-replace edit on > everything in the package). The idea is indeed to break this in Python 3.0 (but not earlier for backwards compatibility reasons). In 3.0, you would have to (and in 2.4 perhaps you could) write this as from . import a or from ... import a depending on which proposal is accepted. > I think this whole thing needs a PEP, though. If only to be completely > clear on what is and is not changing, and to record discarded options. Correct. I have no time to write it though. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Dec 18 10:59:00 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 18 10:59:06 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 14:40:30 GMT." References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> Message-ID: <200312181559.hBIFx0j25181@c-24-5-183-134.client.comcast.net> [Barry] > > That's why when I tend to think about this, I start wanting a way to > > spell "definitely give me the global one, no matter what". IOW, I feel > > like I want a way to bypass relative module lookups. [Paul Moore] > That issue I can understand. And I agree there should be a way to > state it explicitly. > > One of the issues here is that this area is a bit under-documented (I > know, I should read the source, but I don't have the time right now). > > At the moment, there are two cases: > > 1. Import from sys.path. This is what is being called an "absolute" > import, and is nice and easy to understand. The key issue is that > there is no way to *force* this interpretation in the face of > option (2) below. Right. > 2. Import from "the package". This is the under-documented bit, but if > I understand it correctly, it's basically that from within a module > contained in a package, sys.path is conceptually *extended* to > include the package's __path__ (which by default contains the > directory of the package, but which can be user-modified). That's not a very useful way to think about it; when the module is found on __path__ something very *different* happens than when it is found on sys.path. The difference is noticeable if you ask the imported module for its __name__. If you the import was satisfied from __path__, then __name__ will include the package name. If it was satisfied from sys.path, it won't. And __name__ is related to module identity: all modules are stored in sys.modules using their __name__ as the key, so multiple imports of a module with the same __name__ end up referring to the same module (and only the first import causes the module's code to be executed). > Now the big problem here is that behaviour (2) is useful. Simple > "relative" imports of one module within a package from another module > in the same package are common. Guido's (IMHO ugly) "dot" syntax > handles that, by making users explicitly request option (2), and > making the current import syntax *only* mean (1). Actually, what you consider useful is considered harmless by others. Many large packages, including Zope, have adopted a rule of always using absolute imports, to clarify what kind of import is being used. > But none of the proposed solutions handle the __path__ variable. I > don't have any objection in principle to desupporting __path__ (heck, > it would have made thinking about PEP 302 easier, if nothing else) but > (a) it would need a deprecation warning, and (b) Guido himself offered > a use case in . __path__ has no bearing on the proposals, it is used for relative imports. I think your confusing about how the current import works caused you to think it is relevant. > This definitely needs a PEP. If we're removing support for __path__, > the implications need to be thought through (PEP 302, the pkgutil > module, etc etc). If we're not, none of the proposals so far have > covered how __path__ gets supported in future. We're not, but they have (by not mentioning it); but I agree that a PEP is needed. > A much simpler proposal, just providing an explicit way of saying > "Import from sys.path *only*" may be OK without a PEP. But even then, > I'd suspect we should have a PEP explaining *why* it has to be this > simple. This was where this thread started: someone proposed a way of spelling "import from sys.path only". My big problem with that is that *most* imports are intended to be satisfied by sys.path only, so this spelling should be the default, meaning the current spelling. > Go on, Guido. We'll be gentle if you write a PEP, and we won't set > c.l.p on you :-) Alas, I have no time. --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake at acm.org Thu Dec 18 11:53:54 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Dec 18 11:54:05 2003 Subject: [Python-Dev] Python 2.3.3 documentation Message-ID: <16353.56226.174490.137050@sftp.fdrake.net> Formatted copies of the Python 2.3.3 documentation is available at: ftp://ftp.python.org/pub/python/doc/2.3.3/ The website has not yet been updated. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From klm at zope.com Thu Dec 18 12:20:30 2003 From: klm at zope.com (Ken Manheimer) Date: Thu Dec 18 12:27:10 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <200312181550.hBIFoiN25162@c-24-5-183-134.client.comcast.net> Message-ID: +1 for the '.' proposal, where (to verify my understanding): - '.' indicates "import from the current package", '..' means "import from the package of my package", and so forth, - imports where the package part lacks a leading '.' are absolute, ie from the top level of the path. -1 for the '...' proposal, where '...' means a search from the current package up to the most superior containing package (which is actually a virtual package that entails the entire path). Ie, i am for absolute being the default and explicit-relative being available, and against ambiguous relative aka scanning the hierarchy. (I might be convinced otherwise about the scanning deal, but i think the more deterministic we can make imports, while still keeping them flexible enough to do what people need, the better.) I also like your proposal for a "no ambiguous imports" flag from __future__ import absolute_import Ken klm@zope.com From guido at python.org Thu Dec 18 12:50:41 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 18 12:50:48 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 12:20:30 EST." References: Message-ID: <200312181750.hBIHof425371@c-24-5-183-134.client.comcast.net> [Ken Manheimer] > +1 for the '.' proposal, where (to verify my understanding): > > - '.' indicates "import from the current package", '..' means "import > from the package of my package", and so forth, > > - imports where the package part lacks a leading '.' are absolute, ie > from the top level of the path. > > -1 for the '...' proposal, where '...' means a search from the current > package up to the most superior containing package (which is > actually a virtual package that entails the entire path). > > Ie, i am for absolute being the default and explicit-relative being > available, and against ambiguous relative aka scanning the hierarchy. > (I might be convinced otherwise about the scanning deal, but i think > the more deterministic we can make imports, while still keeping them > flexible enough to do what people need, the better.) Thanks! Your vote is especially important for me because as far as I can remember you proposed the scan-up rule for the very first version of "ni" that we developed together back in 1995! > I also like your proposal for a "no ambiguous imports" flag > > from __future__ import absolute_import OK. Wanna write a PEP? Just kidding, I know you have less time than I do for it. So... *anybody* wanna write a PEP about this? (The parentheses proposal should also be folded in.) --Guido van Rossum (home page: http://www.python.org/~guido/) From klm at zope.com Thu Dec 18 13:07:11 2003 From: klm at zope.com (Ken Manheimer) Date: Thu Dec 18 13:14:09 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <200312181750.hBIHof425371@c-24-5-183-134.client.comcast.net> Message-ID: On Thu, 18 Dec 2003, Guido van Rossum wrote: > Thanks! Your vote is especially important for me because as far as I > can remember you proposed the scan-up rule for the very first version > of "ni" that we developed together back in 1995! Older and*/*or wiser?-) FWIW, i also proposed using a leading '.' (and a leading '_', when '.' was refused because the parser couldn't handle it) for what we're now calling explict relative imports. Ken klm@zope.com From skip at pobox.com Thu Dec 18 16:20:22 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 18 16:21:00 2003 Subject: [Python-Dev] syncing between projects? Message-ID: <16354.6678.463640.780682@montanaro.dyndns.org> The python-mode.el file in the python-mode project is now at 4.41, while the version in Python's Misc directory is still at 4.38. Is there a way to sync them in one cvs operation so they have the same version? Thx, Skip From martin at v.loewis.de Thu Dec 18 16:22:34 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 18 16:23:44 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <20031217233126.GA4692@unpythonic.net> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> Message-ID: Jeff Epler writes: > To follow up on my own message, here is a program that demonstrates the > problem I ran into with threads and signals. Can you find out what $$ is, and what the PIDs and thread IDs of all participating threads are? Regards, Martin From martin at v.loewis.de Thu Dec 18 16:28:50 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 18 16:29:13 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <87y8tbezcq.fsf@codesourcery.com> References: <87y8tbezcq.fsf@codesourcery.com> Message-ID: "Zack Weinberg" writes: > > This is what leads to the bletcherous > > > > do { > > the real macro guts go here > > } while(0) > > > > form of macro definition. Py_DECREF could be rewritten in that form -- but > > I've puked enough for one day . > > Should, not just could. Yeah, it's bletcherous, but it's the only > way to be sure. (Short of inline functions.) Not true. As Tim explains, there is no possible application of the macro which gets misinterpreted. We don't have a single if-statement, we have an if-else-statement in the macro. That never leads to the dangling else problem. Regards, Martin From greg at cosc.canterbury.ac.nz Thu Dec 18 16:33:22 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 18 16:33:32 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: <3FE18862.50208@iinet.net.au> Message-ID: <200312182133.hBILXMV05456@oma.cosc.canterbury.ac.nz> Nick Coghlan : > What if instead of 'builtin behaves differently in restricted mode' we > had 'restricted __builtins__ contains a DIFFERENT builtin, that happens > to have the same name'? That wouldn't solve the file problem, because if the restricted code ever got hold of a file object created by unrestricted code, it would have access to the "real" file class which can open any file. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From martin at v.loewis.de Thu Dec 18 16:27:30 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 18 16:33:51 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <16352.63023.993326.798113@montanaro.dyndns.org> References: <3FE0DB90.9060604@erols.com> <16352.63023.993326.798113@montanaro.dyndns.org> Message-ID: Skip Montanaro writes: > I meant the Py_*REF macros which expand to "if" statements and you'd bracify > the clauses, e.g., Py_DECREF would become: That would still trigger the warning about if(1) Py_DECREF(foo); as this expands to if(1) if(cond(foo)) {action1(foo);} else {action2(foo);} This, in turn, is potentially unintended, as the author of that code may have meant the else to go with the if(1), whereas, in C, it goes with the if(cond(foo)). As Tim explains, the intended semantics is the desired one, too, so there is really no problem in the code. As Tim also explains, do{}while(1) would silence the warning. Regards, Martin From barry at python.org Thu Dec 18 16:36:40 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 18 16:36:45 2003 Subject: [Python-Dev] syncing between projects? In-Reply-To: <16354.6678.463640.780682@montanaro.dyndns.org> References: <16354.6678.463640.780682@montanaro.dyndns.org> Message-ID: <1071783400.31232.10.camel@anthem> On Thu, 2003-12-18 at 16:20, Skip Montanaro wrote: > The python-mode.el file in the python-mode project is now at 4.41, while the > version in Python's Misc directory is still at 4.38. Is there a way to sync > them in one cvs operation so they have the same version? cvs com -r4.41 python-mode.el -Barry From gmcm at hypernet.com Thu Dec 18 16:45:54 2003 From: gmcm at hypernet.com (Gordon McMillan) Date: Thu Dec 18 16:46:51 2003 Subject: [Python-Dev] Relative import In-Reply-To: Message-ID: <3FE1D9C2.32355.4A1EE51A@localhost> I am +10e6 on disambiguating imports. I am +0 on "scan". I am +0.01 on "explicit relative". I can live with any syntax Guido can, so I'm just voting on functionality. The use cases for "ambiguous" imports amount to: 1) I use this cool package, but I'm too lazy to keep up with new releases or document the release required & ensure that it's available & further I'm so arrogant that I don't care that my users might run into problems having 2 copies of the code or that the package developer might be offended by my end run. 2) I structured my package wrong, and now I want to fix it with the mouse. I'm not sure I go as far as Tim, but I think perhaps the PSU should have a regulation about writing import hooks without a license. -- Gordon http://www.mcmillan-inc.com/ From greg at cosc.canterbury.ac.nz Thu Dec 18 16:50:27 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 18 16:50:36 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: <1071753324.3fe1a86c3827e@mcherm.com> Message-ID: <200312182150.hBILoRe05474@oma.cosc.canterbury.ac.nz> Michael Chermside : > The other misconception (apparently) is that the only real USE for > accessing the __class__ attribute is for testing class membership. You > said above that it's useful "for all sorts of reasons"... I'm > wondering what they are. Two that come to mind: * Looking at the class's docstring * Finding out what methods an object has Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From martin at v.loewis.de Thu Dec 18 16:33:52 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 18 16:56:04 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031218092922.GF6324@lkcl.net> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> <20031216212312.GL17021@lkcl.net> <20031218092922.GF6324@lkcl.net> Message-ID: Luke Kenneth Casson Leighton writes: > so, with the correct codebase reordering, a simple capabilities > based system can be added, the problem goes away. > > yes? I still don't see how it could, and I believe I understand pretty well what kind of feature you propose. Restricted execution is just something completely different. Regards, Martin From skip at pobox.com Thu Dec 18 16:56:05 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 18 16:56:33 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: <200312182150.hBILoRe05474@oma.cosc.canterbury.ac.nz> References: <1071753324.3fe1a86c3827e@mcherm.com> <200312182150.hBILoRe05474@oma.cosc.canterbury.ac.nz> Message-ID: <16354.8821.435209.523359@montanaro.dyndns.org> >> You said above that it's useful "for all sorts of reasons"... I'm >> wondering what they are. Greg> Two that come to mind: ... And another is breaking out of rexec hell to do devious things on the system where the code is running. Skip From greg at cosc.canterbury.ac.nz Thu Dec 18 16:57:39 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 18 16:57:59 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Message-ID: <200312182157.hBILvd005491@oma.cosc.canterbury.ac.nz> Paul Moore : > If that's what you'd like a vote on, I prefer the triple-dot > proposal. How would you spell relative references to parent packages using the triple-dot version? Remember we've dropped the idea of search-upwards, so that would have to be made explicit somehow. > pkg\__init__.py: > print "importing pkg" > import a > pkg\a.py: > print "importing a" > import b > pkg\b.py: > print "importing b" > > >>> import pkg > importing pkg > importing a > importing b > > Is the proposal that any of this *stop* working? I think the proposal is for it to keep working for backwards compatibility now, but to stop working in Python 3.0. Is that correct? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Thu Dec 18 16:59:26 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 18 16:59:49 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Message-ID: <200312182159.hBILxQu05504@oma.cosc.canterbury.ac.nz> Paolo Invernizzi : > Skip Montanaro wrote: > > > It has been proposed (more than once I think) that such a > > stdlib package could be created in parallel using symlinks. > > Symlink is one of the things that I would like to have on my XP box... Also, wouldn't that give two ways of getting each of the modules, with all the attendand Bad Things? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Thu Dec 18 17:07:33 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 18 17:07:41 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Message-ID: <200312182207.hBIM7Xn05611@oma.cosc.canterbury.ac.nz> Paul Moore : > But none of the proposed solutions handle the __path__ variable. I > don't have any objection in principle to desupporting __path__ I'll be worried if __path__ is likely to disappear as a result of all this. I'm using it in what will be the next version of my Python GUI library, to switch in different implementations of submodules depending on the platform. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Thu Dec 18 17:13:03 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 18 17:13:13 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <16353.49104.97549.745804@montanaro.dyndns.org> Message-ID: <200312182213.hBIMD3x05633@oma.cosc.canterbury.ac.nz> Skip Montanaro : > I have something which looks like a symlink on my Win2k box. Windows > calls it a Shortcut in listings. Is that not roughly the same thing > as a symlink? Only if it's transparent to the system calls which operate on files (e.g. open()ing it opens the file that it refers to, etc.) I don't believe that's the case with Windows shortcuts. Macintosh "aliases" have the same problem. (And now with MacOS X, we have both aliases *and* symlinks in the same system, just to add to the confusion...) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From barry at python.org Thu Dec 18 17:15:05 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 18 17:15:11 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <200312182157.hBILvd005491@oma.cosc.canterbury.ac.nz> References: <200312182157.hBILvd005491@oma.cosc.canterbury.ac.nz> Message-ID: <1071785704.31232.22.camel@anthem> On Fri, 2003-12-19 at 05:57, Greg Ewing wrote: > How would you spell relative references to parent packages > using the triple-dot version? Does anybody have an example, in released code, of where this kind of repackaging relative imports is being used? I'd like to look at some real-world examples. Thanks, -Barry From lkcl at lkcl.net Thu Dec 18 17:57:48 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Thu Dec 18 17:57:48 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: References: <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> <20031216212312.GL17021@lkcl.net> <20031218092922.GF6324@lkcl.net> Message-ID: <20031218225748.GE2234@lkcl.net> On Thu, Dec 18, 2003 at 10:33:52PM +0100, Martin v. L?wis wrote: > Luke Kenneth Casson Leighton writes: > > > so, with the correct codebase reordering, a simple capabilities > > based system can be added, the problem goes away. > > > > yes? > > I still don't see how it could, neither do i: i believe that the best person you should address that concern to is to greg ewing, who raised this point: "The spirit behind my suggestion was to start thinking about ways in which functionality could be separated out so that this kind of special-casing for security purposes isn't needed." in other words, i think he means that by restructuring the python codebase [and libraries? and its design?] it _may_ be possible to avoid a need for adding ACLs at all. and i believe he may be thinking along the lines of being able to permanently _remove_ at run-time, under python-programmer-control, access to __class__ from objects. somehow. > and I believe I understand pretty well > what kind of feature you propose. Restricted execution is just > something completely different. if you believe that, then i have evidently not explained myself clearly enough. i'll give this one more shot. i'll be honest with you. if it's not obvious enough at the end of this message, i have to give up and stop, and say i tried but did not succeed in explaining this clearly enough to you, which is no failing on your part, but on mine. anyway. first i should outline the pseudo-code modifications needed to Python/ceval.c which are necessary, then after that i will outline what ACLs are needed. what i _can_ say that i haven't gone into detail on [in this message] is _how_ the acls are obtained from the function object. e.g. in GetACLforFunction. but it is worth mentioning that _how_ the acls are _associated_ with the function object (or any other object) is an implementation-specific issue NOT a design / specification issue. static PyObject * call_function(PyObject ***pp_stack, int oparg) { PyObject **pfunc = (*pp_stack) - n - 1; PyObject *func = *pfunc; /* caller function name on stack below?? probably not, but the principle is at least demonstrable */ PyObject *calling_func = GetCallerFunction(*pp_stack) - n - 2; PyACL *acl_for_func = GetACLforFunction(func); ... if (acl_for_func != NULL && !check_permissions(calling_func, acl_for_func, PERMS_EXECUTE) { return Exception("access denied"); } ... } also, to satisfy the requirements of "capabilities", a specific check to make it look like the object doesn't exist is added. it's probably possible to have a look-up table on a per-opcode basis to look up the permissions to be checked against, which alleviates the need to put a call to check_permissions in every darn switch statement. note the check for ACL being NULL: if it's NULL, that's assumed to mean "anything goes". it also means that there's not much performance hit involved unless you actually add permissions. case STORE_ATTR: if (acl_for_func != NULL && !check_permissions(calling_func, acl_for_func, PERMS_CAN_SEE) { return Exception("attrib doesn't exist"); } if (acl_for_func != NULL && !check_permissions(calling_func, acl_for_func, PERMS_WRITE) { return Exception("access denied"); } case DELETE_ATTR: if (acl_for_func != NULL && !check_permissions(calling_func, acl_for_func, PERMS_CAN_SEE) { return Exception("attrib doesn't exist"); } if (acl_for_func != NULL && !check_permissions(calling_func, acl_for_func, PERMS_WRITE+PERMS_DELETE) { return Exception("access denied"); } case LOAD_ATTR: if (acl_for_func != NULL && !check_permissions(calling_func, acl_for_func, PERMS_CAN_SEE) { return Exception("attrib doesn't exist"); } if (acl_for_func != NULL && !check_permissions(calling_func, acl_for_func, PERMS_READ) { return Exception("access denied"); } basically, as long as code is only executed through eval_frame(), then it's likely to be the only place where access control checks are needed to be added. basically, therefore, once calls to restrict operations are added, it's a matter of dropping the ACLs in at the right places. _that's_ how you achieve the same job as the rexec.py code. you place an acl on __builtins__ to restrict "all functions", DENY, EVERYTHING. you place an acl on open() to restrict "all functions", DENY, FILE_WRITE. you place an acl on open.__class__ "all functions", DENY, CAN_SEE+FILE_WRITE in fact you might even be able to get away with putting an acl on EVERY function right at the top-level. but that, as described previously, means that it'd be necessary to get the "ACL-inheritance" to work properly, in order to avoid having to add a DENY FILE_WRITE acl to every single file operation capable of writing, for example. so like i said, i believe it to be a relatively simple job to spec out how to add the means _by which_ ACLs can be usefully evaluated. what i CAN'T tell you is exactly WHERE every single ACL [needed to achieve the same results as rexec] have to be added [such that they will be usefully evaluated]: i simply don't have enough knowledge of the python codebase to do that on my own. [unless someone was happy to pay me for long enough to find out, of course]. is this making sense at all? in some ways, the longer this is left, the harder it is going to be to retrospectively bolt on. there's an adage that says security cannot be easily added in, it has to be designed in from the start. fortunately, i think there are a lot of smart people about :) sincerely, l. From janssen at parc.com Thu Dec 18 18:54:49 2003 From: janssen at parc.com (Bill Janssen) Date: Thu Dec 18 18:55:12 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 14:15:05 PST." <1071785704.31232.22.camel@anthem> Message-ID: <03Dec18.155458pst."58611"@synergy1.parc.xerox.com> > Does anybody have an example, in released code, of where this kind of > repackaging relative imports is being used? I'd like to look at some > real-world examples. I'm citing an example from the Jython compile of the Plucker distiller, for one. That's available as part of the Plucker package from www.plkr.org -- look at the Makefile in parser/python. But I'd guess most real examples are proprietary code (I can think of a couple right now). Bill From jepler at unpythonic.net Thu Dec 18 20:23:55 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Thu Dec 18 20:23:44 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> Message-ID: <20031219012347.GB4692@unpythonic.net> On Thu, Dec 18, 2003 at 10:22:34PM +0100, Martin v. L?wis wrote: > Can you find out what $$ is, and what the PIDs and thread IDs of all > participating threads are? I'm not sure what all information I should try to gather for you. Let me know if you think this is enough to file a bug report with... I changed the example to make it clearer that it's the subprocess ignoring the signal that is the problem, not anything in Python that is taking time to notice the death of a child process. Typical output (of course, pids and ppids change from run to run. I didn't find gettid(), I think thread.get_ident() is not what you were asking for): not threaded os.getpid() -> 6444 os.getppid() -> 6332 thread.get_ident() -> 1074152064 shell process 6445 thread ppid 6444 Elapsed time 0.00640296936035 subthread os.getpid() -> 6444 os.getppid() -> 6332 thread.get_ident() -> 1082547504 shell process 6447 thread ppid 6444 kill failed Elapsed time 1.01621508598 main thread os.getpid() -> 6444 os.getppid() -> 6332 thread.get_ident() -> 1074152064 shell process 6449 thread ppid 6444 Elapsed time 0.00641894340515 :r threadbug.py import thread, time, os def doit(): print "os.getpid() ->", os.getpid() print "os.getppid() ->", os.getppid() print "thread.get_ident() ->", thread.get_ident() t = time.time() os.system("echo shell process $$; echo thread ppid $PPID; kill $$; echo kill failed; sleep 1") t1 = time.time() print "Elapsed time", t1-t print print print "not threaded" doit() print "subthread" thread.start_new(doit, ()) time.sleep(2) print "main thread" doit() From mal at egenix.com Fri Dec 19 03:30:27 2003 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Dec 19 03:30:27 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.204, 2.205 In-Reply-To: References: Message-ID: <3FE2B723.9060108@egenix.com> perky@users.sourceforge.net wrote: > Update of /cvsroot/python/python/dist/src/Objects > In directory sc8-pr-cvs1:/tmp/cvs-serv1651/Objects > > Modified Files: > unicodeobject.c > Log Message: > SF #859573: Reduce compiler warnings on gcc 3.2 and above. > > Index: unicodeobject.c > *************** > *** 2204,2208 **** > > /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */ > ! if (size == 1 && *(unsigned char*)s < 256) { > Py_UNICODE r = *(unsigned char*)s; > return PyUnicode_FromUnicode(&r, 1); > --- 2212,2216 ---- > > /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */ > ! if (size == 1) { > Py_UNICODE r = *(unsigned char*)s; > return PyUnicode_FromUnicode(&r, 1); This "fix" doesn't look right. Please check. > *************** > *** 2406,2409 **** > --- 2414,2421 ---- > else if (*p<1000) > repsize += 2+3+1; > + #ifndef Py_UNICODE_WIDE > + else > + repsize += 2+4+1; > + #else > else if (*p<10000) > repsize += 2+4+1; > *************** > *** 2414,2417 **** > --- 2426,2430 ---- > else > repsize += 2+7+1; > + #endif > } > requiredsize = respos+repsize+(endp-collend); -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 19 2003) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From perky at i18n.org Fri Dec 19 04:23:09 2003 From: perky at i18n.org (Hye-Shik Chang) Date: Fri Dec 19 04:23:20 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.204, 2.205 In-Reply-To: <3FE2B723.9060108@egenix.com> References: <3FE2B723.9060108@egenix.com> Message-ID: <20031219092309.GA81703@i18n.org> On Fri, Dec 19, 2003 at 09:30:27AM +0100, M.-A. Lemburg wrote: > perky@users.sourceforge.net wrote: > >Update of /cvsroot/python/python/dist/src/Objects > >In directory sc8-pr-cvs1:/tmp/cvs-serv1651/Objects > > > >Modified Files: > > unicodeobject.c > >Log Message: > >SF #859573: Reduce compiler warnings on gcc 3.2 and above. > > > >Index: unicodeobject.c > >*************** > >*** 2204,2208 **** > > > > /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */ > >! if (size == 1 && *(unsigned char*)s < 256) { > > Py_UNICODE r = *(unsigned char*)s; > > return PyUnicode_FromUnicode(&r, 1); > >--- 2212,2216 ---- > > > > /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */ > >! if (size == 1) { > > Py_UNICODE r = *(unsigned char*)s; > > return PyUnicode_FromUnicode(&r, 1); > > This "fix" doesn't look right. Please check. > gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o Objects/unicodeobject.o Objects/unicodeobject.c Objects/unicodeobject.c: In function `PyUnicodeUCS2_DecodeLatin1': Objects/unicodeobject.c:2214: warning: comparison is always true due to limited range of data type AFAIK, *(unsigned char*)s is always smaller than 256. Also decoding latin-1 can be done by just casting it into Py_UNICODE. I'm sorry but can you explain more? Hye-Shik From tismer at tismer.com Fri Dec 19 04:32:37 2003 From: tismer at tismer.com (Christian Tismer) Date: Fri Dec 19 04:31:08 2003 Subject: [Python-Dev] Last chance! (was: Does anybody really use frame->f_tstate ?) In-Reply-To: <3FDE6D35.3090100@tismer.com> References: <3FDE6D35.3090100@tismer.com> Message-ID: <3FE2C5B5.8080208@tismer.com> Dear Python community, since I didn't get *any* reply to this request, either the request was bad or there is really nobody using f_tstate in a way that makes it urgent to keep. I will wait a few hours and then make the change to Stackless, and I'd like to propose to do the same to the Python core. Christian Tismer wrote: > Hi colleagues, > > this is my second attempt to get rid of the f_tstate field > in frames. I need to find every user of this field. > > What am I talking about? > Well, Python always has a thread state variable which > is a unique handle to the current thread. This variable > is accessed in many places, and there exists a fast macro > to get at it. > Every executing Python frame also gets a copy on creation. > In some cases, this frame->f_tstate field is used, > in other cases the current tstate variable is used. > > If this sounds foreign to you, please stop reading here. > > ------------------------------------------------------------- > > I would like to get rid of the frame->f_tstate, and I'm trying > to find out if there is a need for it. I don't need it, > for Stackless, it is the opposite, it disturbs. > > There was a small thread about this in June this year, where > Guido convinced me that it is possible to create a traceback > on a frame that comes from a different thread. > > http://mail.python.org/pipermail/python-dev/2003-June/036254.html > > Ok, this is in fact possible, although I don't think > anybody has a need for this. > > My question to all extension writers is this: > If you use frame->f_tstate at all, do you use it just > because it is handy, or do you want to use it for > some other purpose? > > One purpose could be that you really want to create a traceback > on a different than the current thread. I have never seen this, > but who knows, so that's why I'm asking the Python world. > > In most cases, a traceback will be created on a frame > that is currently processd or just has been processed. > Accessing a frame of a different thread that is being processed > might make sense for special debugger cases. > > My proposal is > -------------- > > a) change semantics of PytraceBack_Here to use the current tstate. > > b) if such a special purpose exists, create a new function for it. > > c) if urgent, different needs exist to keep f_tstate, > then let's forget about this proposal. > > Especially for Stackless, I'd be keen of getting rid of this. > > thanks for input -- chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From fredrik at pythonware.com Fri Dec 19 05:03:41 2003 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri Dec 19 05:03:53 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 References: <3FE2B723.9060108@egenix.com> <20031219092309.GA81703@i18n.org> Message-ID: Hye-Shik Chang wrote: > AFAIK, *(unsigned char*)s is always smaller than 256. except when it isn't. see the ANSI C spec for details. From mal at egenix.com Fri Dec 19 05:18:28 2003 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Dec 19 05:18:42 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.204, 2.205 In-Reply-To: <20031219092309.GA81703@i18n.org> References: <3FE2B723.9060108@egenix.com> <20031219092309.GA81703@i18n.org> Message-ID: <3FE2D074.90000@egenix.com> Hye-Shik Chang wrote: > On Fri, Dec 19, 2003 at 09:30:27AM +0100, M.-A. Lemburg wrote: > >>perky@users.sourceforge.net wrote: >> >>>Update of /cvsroot/python/python/dist/src/Objects >>>In directory sc8-pr-cvs1:/tmp/cvs-serv1651/Objects >>> >>>Modified Files: >>> unicodeobject.c >>>Log Message: >>>SF #859573: Reduce compiler warnings on gcc 3.2 and above. >>> >>>Index: unicodeobject.c >>>*************** >>>*** 2204,2208 **** >>> >>> /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */ >>>! if (size == 1 && *(unsigned char*)s < 256) { >>> Py_UNICODE r = *(unsigned char*)s; >>> return PyUnicode_FromUnicode(&r, 1); >>>--- 2212,2216 ---- >>> >>> /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */ >>>! if (size == 1) { >>> Py_UNICODE r = *(unsigned char*)s; >>> return PyUnicode_FromUnicode(&r, 1); >> >>This "fix" doesn't look right. Please check. >> > > gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o Objects/unicodeobject.o Objects/unicodeobject.c > Objects/unicodeobject.c: In function `PyUnicodeUCS2_DecodeLatin1': > Objects/unicodeobject.c:2214: warning: comparison is always true due to limited range of data type > > AFAIK, *(unsigned char*)s is always smaller than 256. > Also decoding latin-1 can be done by just casting it into Py_UNICODE. You are right. I was thinking that there was some reason we needed this to get Unicode working on Crays, but looking at the CVS log, this was probably just the result of adjusting Martin's single character sharing code to work for Latin-1 rather than just ASCII characters. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 19 2003) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From 3gc6p6iqz at aol.com Fri Dec 19 11:23:03 2003 From: 3gc6p6iqz at aol.com (Nichole Villalobos) Date: Fri Dec 19 06:22:00 2003 Subject: [Python-Dev] Python-announce-list-request ocb orzmsmz fur Message-ID: Hi Python-announce-list-request , I have a picture online now. I just want someone to know me before they see me. I just feel better that way. (but believe me you wont be disappointed) Well i am 22 years old. I have a very out going personality. I love to meet new people, i am on the varsity cheerleading squad. I like the little romantic stuff and love to be swept off my feet and suprised. I am currently looking for a relationship. All around if i were to use 3 words to describe me id say Outgoing, sexy, and spontaneous :) And if u wanna chat or get to know me. If u really like what u see. youll do more than just send me a note. Talk to you soon I hope... :) Carolynn ps. my friend Maria is on with me as well. http://seeingnoone.com/confirm/?oc=52211091 I do not wish to go on any blind dates at all, now or in the future: http://seeingnoone.com/remove/?oc=1 m393nnm3m3rem wv From perky at i18n.org Fri Dec 19 06:59:52 2003 From: perky at i18n.org (Hye-Shik Chang) Date: Fri Dec 19 06:59:59 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: References: <20031219092309.GA81703@i18n.org> Message-ID: <20031219115952.GA84061@i18n.org> On Fri, Dec 19, 2003 at 11:03:41AM +0100, Fredrik Lundh wrote: > Hye-Shik Chang wrote: > > > AFAIK, *(unsigned char*)s is always smaller than 256. > > except when it isn't. see the ANSI C spec for details. > Ah. I found. I'm very surprised for that. Thank you! :-) BTW, do we really support architectures with 9bits-sized char? Hye-Shik From mwh at python.net Fri Dec 19 07:10:11 2003 From: mwh at python.net (Michael Hudson) Date: Fri Dec 19 07:10:15 2003 Subject: [Python-checkins] Re: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <20031219115952.GA84061@i18n.org> (Hye-Shik Chang's message of "Fri, 19 Dec 2003 20:59:52 +0900") References: <20031219092309.GA81703@i18n.org> <20031219115952.GA84061@i18n.org> Message-ID: <2mbrq5f1p8.fsf@starship.python.net> Hye-Shik Chang writes: > On Fri, Dec 19, 2003 at 11:03:41AM +0100, Fredrik Lundh wrote: >> Hye-Shik Chang wrote: >> >> > AFAIK, *(unsigned char*)s is always smaller than 256. >> >> except when it isn't. see the ANSI C spec for details. >> > > Ah. I found. I'm very surprised for that. Thank you! :-) > BTW, do we really support architectures with 9bits-sized char? On some kinds of Cray that Python has been built on in the past, I think the smallest addressable unit of memory is 64 bits. So, not quite 96, but getting on that way. I don't think we want to make the lives of people porting to such architectures any harder than it already is... Cheers, mwh -- Need to Know is usually an interesting UK digest of things that happened last week or might happen next week. [...] This week, nothing happened, and we don't care. -- NTK Now, 2000-12-29, http://www.ntk.net/ From ncoghlan at iinet.net.au Fri Dec 19 07:54:16 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri Dec 19 07:54:22 2003 Subject: [Python-checkins] Re: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <2mbrq5f1p8.fsf@starship.python.net> References: <20031219092309.GA81703@i18n.org> <20031219115952.GA84061@i18n.org> <2mbrq5f1p8.fsf@starship.python.net> Message-ID: <3FE2F4F8.80507@iinet.net.au> Michael Hudson wrote: > Hye-Shik Chang writes: [...] >>BTW, do we really support architectures with 9bits-sized char? [...] > I don't think we want to make the > lives of people porting to such architectures any harder than it > already is... TI make chips where the smallest addressable unit is 16-bits and sizeof(char) == sizeof(int) == 16 bits == 1 byte due to the way the C standard is written. I don't think Python is ported to any such chip at present (the one I use is a DSP, and I would seriously question the sanity of anyone who tried to run Python on one of these critters), but it's a possibility that shouldn't be ignored. Porting to such a machine would be rather entertaining (sizeof() gets a _lot_ of work in the code for that DSP). Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From skip at pobox.com Fri Dec 19 08:03:00 2003 From: skip at pobox.com (Skip Montanaro) Date: Fri Dec 19 08:03:05 2003 Subject: [Python-checkins] Re: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <3FE2F4F8.80507@iinet.net.au> References: <20031219092309.GA81703@i18n.org> <20031219115952.GA84061@i18n.org> <2mbrq5f1p8.fsf@starship.python.net> <3FE2F4F8.80507@iinet.net.au> Message-ID: <16354.63236.775441.319110@montanaro.dyndns.org> Nick> Michael Hudson wrote: >> Hye-Shik Chang writes: Nick> [...] >>> BTW, do we really support architectures with 9bits-sized char? Nick> [...] >> I don't think we want to make the lives of people porting to such >> architectures any harder than it already is... Nick> TI make chips where the smallest addressable unit is 16-bits and Nick> sizeof(char) == sizeof(int) == 16 bits == 1 byte due to the way Nick> the C standard is written. It seems to me the right thing to do is to cook up a test in the configure script which checks the number of bits in an unsigned char and sets a cpp macro which the code in question then uses to compile the fast case for 8-bit chars and the slow case otherwise. Skip From ncoghlan at iinet.net.au Fri Dec 19 08:07:38 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri Dec 19 08:08:50 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031218225748.GE2234@lkcl.net> References: <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> <20031216212312.GL17021@lkcl.net> <20031218092922.GF6324@lkcl.net> <20031218225748.GE2234@lkcl.net> Message-ID: <3FE2F81A.7090802@iinet.net.au> Luke Kenneth Casson Leighton wrote: > in some ways, the longer this is left, the harder it is going to > be to retrospectively bolt on. > > there's an adage that says security cannot be easily added in, it > has to be designed in from the start. This is very true, but it hurts an ACL-based approach even worse than it hurts a capabilities based one. To get capabilities to work, the question is: how do we construct an environment where 'builtins' and all other objects passed to code in that environment have been suitably restricted to prevent malicious code from causing damage. The original objects, which are never made available to the untrusted code, don't need to care about trust issues - they just keep working as they always have. To get ACL's to work, _everything_ in Python has to care about trust issues, as they have to know that they should be checking for the existence of an ACL. I can't even begin to imagine how those ACL's might be managed effectively, but I can imagine constructing a special execution environment which only allowed 'safe' objects to be passed in. A 'safe' object would be one of the restricted builtins, or objects able to be constructed using only that restricted set of builtins. The major issue comes in dealing with Python's introspection capabilities without making them completely useless (then again, perhaps 'restricted, with almost no introspection' would be an improvement over 'no restricted mode'. Anyway, despite either approach being Python 3.0 material, the capability method at least seems conceptually possible - deleting entries out of Python namespace dictionaries is a fairly straightforward activity, as is substituting a new implementation for the old 'unsafe' implementation when we want to switch to 'restricted' mode. Whereas handling ACL's would be a completely new approach that spreads its tentacles through much of the CPython source code. For code, capabilities just make more sense - if they can't use it, don't even let them know it's there. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From mcherm at mcherm.com Fri Dec 19 08:11:00 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Fri Dec 19 08:11:05 2003 Subject: [Python-Dev] Importing packages from command line Message-ID: <1071839460.3fe2f8e4dbbad@mcherm.com> Dmitry Vasiliev writes: > Just an idea... > > 1. python -p package_name > > Equivalent to: > > import package_name (1) Sourceforge is a great place to request feature enhancements. Suggestions made on this mailing list is likely to be forgotten sooner or later. (2) Can you explain WHY you would want this feature? Is there some use case? I would prefer NOT to have this, because right now if I'm reading code and it uses "package_name.someFunction()" I can scan upward for "package_name" to find out what it's using. With command line imports, I couldn't do that. So unless you've got a good reason for it, I personally wouldn't want this feature. -- Michael Chermside From skip at pobox.com Fri Dec 19 08:20:52 2003 From: skip at pobox.com (Skip Montanaro) Date: Fri Dec 19 08:20:49 2003 Subject: [Python-checkins] Re: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <16354.63236.775441.319110@montanaro.dyndns.org> References: <20031219092309.GA81703@i18n.org> <20031219115952.GA84061@i18n.org> <2mbrq5f1p8.fsf@starship.python.net> <3FE2F4F8.80507@iinet.net.au> <16354.63236.775441.319110@montanaro.dyndns.org> Message-ID: <16354.64308.552430.331674@montanaro.dyndns.org> Skip> It seems to me the right thing to do is to cook up a test in the Skip> configure script which checks the number of bits in an unsigned Skip> char ... Better yet, let's use CHAR_BIT: #if defined(CHAR_BIT) && CHAR_BIT == 8 ... fast case ... #else ... slow case ... #endif From ark at acm.org Fri Dec 19 09:00:07 2003 From: ark at acm.org (Andrew Koenig) Date: Fri Dec 19 09:00:02 2003 Subject: [Python-Dev] 2.3.3 released? Message-ID: <007d01c3c638$69074c20$6402a8c0@arkdesktop> It appears that the main python.org page is announcing the release of 2.3.3 final, but the release page http://www.python.org/2.3.3/ does not show a place to download the Windows executable. Have I caught it in mid-transition? From fdrake at acm.org Fri Dec 19 09:09:57 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Dec 19 09:10:04 2003 Subject: [Python-Dev] 2.3.3 released? In-Reply-To: <007d01c3c638$69074c20$6402a8c0@arkdesktop> References: <007d01c3c638$69074c20$6402a8c0@arkdesktop> Message-ID: <16355.1717.391064.71007@sftp.fdrake.net> Andrew Koenig writes: > It appears that the main python.org page is announcing the release of 2.3.3 > final, but the release page http://www.python.org/2.3.3/ does not show a > place to download the Windows executable. The Windows installer will be added when it's ready, then the general announcement will be sent. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From skip at pobox.com Fri Dec 19 09:14:59 2003 From: skip at pobox.com (Skip Montanaro) Date: Fri Dec 19 09:15:00 2003 Subject: [Python-Dev] 2.3.3 released? In-Reply-To: <007d01c3c638$69074c20$6402a8c0@arkdesktop> References: <007d01c3c638$69074c20$6402a8c0@arkdesktop> Message-ID: <16355.2019.340005.818253@montanaro.dyndns.org> Andrew> [2.3.3 release teasers] Andrew> Have I caught it in mid-transition? Yes, I believe so. I saw a bunch of checkins related to that, but it appears all the bits have not yet been pushed into place. Skip From fdrake at acm.org Fri Dec 19 09:21:46 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Dec 19 09:21:53 2003 Subject: [Python-Dev] 2.3.3 released? In-Reply-To: <16355.2019.340005.818253@montanaro.dyndns.org> References: <007d01c3c638$69074c20$6402a8c0@arkdesktop> <16355.2019.340005.818253@montanaro.dyndns.org> Message-ID: <16355.2426.281597.99584@sftp.fdrake.net> Skip Montanaro writes: > Yes, I believe so. I saw a bunch of checkins related to that, but it > appears all the bits have not yet been pushed into place. I made the changes to the doc/ area on python.org last night, and everything is up on the site. I think the 2.3.3/ area is updated as well, the Windows installer just isn't ready yet. It'll be there when it is. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From theller at python.net Fri Dec 19 09:40:30 2003 From: theller at python.net (Thomas Heller) Date: Fri Dec 19 09:40:36 2003 Subject: [Python-Dev] 2.3.3 released? In-Reply-To: <16355.2426.281597.99584@sftp.fdrake.net> (Fred L. Drake, Jr.'s message of "Fri, 19 Dec 2003 09:21:46 -0500") References: <007d01c3c638$69074c20$6402a8c0@arkdesktop> <16355.2019.340005.818253@montanaro.dyndns.org> <16355.2426.281597.99584@sftp.fdrake.net> Message-ID: "Fred L. Drake, Jr." writes: > Skip Montanaro writes: > > Yes, I believe so. I saw a bunch of checkins related to that, but it > > appears all the bits have not yet been pushed into place. > > I made the changes to the doc/ area on python.org last night, and > everything is up on the site. I think the 2.3.3/ area is updated as > well, the Windows installer just isn't ready yet. It'll be there when > it is. Hm, I was waiting for Anthony in IRC #python-dev to upload the windows installer. Sorry if I misunderstood something... Thomas From anthony at interlink.com.au Fri Dec 19 10:29:23 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Dec 19 10:29:51 2003 Subject: [Python-Dev] 2.3.3 released? In-Reply-To: <007d01c3c638$69074c20$6402a8c0@arkdesktop> Message-ID: <200312191529.hBJFTN8Y009795@localhost.localdomain> >>> "Andrew Koenig" wrote > It appears that the main python.org page is announcing the release of 2.3.3 > final, but the release page http://www.python.org/2.3.3/ does not show a > place to download the Windows executable. > > Have I caught it in mid-transition? Yep, timezones make it tricky to manage this stuff sometimes. I'll send the release announcement in the morning when I've checked it's all good. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From guido at python.org Fri Dec 19 10:31:42 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 19 10:31:51 2003 Subject: [Python-Dev] Last chance! (was: Does anybody really use frame->f_tstate ?) In-Reply-To: Your message of "Fri, 19 Dec 2003 10:32:37 +0100." <3FE2C5B5.8080208@tismer.com> References: <3FDE6D35.3090100@tismer.com> <3FE2C5B5.8080208@tismer.com> Message-ID: <200312191531.hBJFVgw27430@c-24-5-183-134.client.comcast.net> > since I didn't get *any* reply to this request, > either the request was bad or there is really > nobody using f_tstate in a way that makes it > urgent to keep. > I will wait a few hours and then make the change > to Stackless, and I'd like to propose to do the > same to the Python core. I saved the message, but haven't had the time yet to think things through. I *did* notice at least one case where using f_tstate might actually be a mistake: theoretically it's possible that two or more threads alternate calling next() on a generator (if they wrap it in a critical section); AFAICT the f_tstate is never updated. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Dec 19 10:39:45 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 19 10:39:49 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: Your message of "Fri, 19 Dec 2003 20:59:52 +0900." <20031219115952.GA84061@i18n.org> References: <20031219092309.GA81703@i18n.org> <20031219115952.GA84061@i18n.org> Message-ID: <200312191539.hBJFdjC27495@c-24-5-183-134.client.comcast.net> > > > AFAIK, *(unsigned char*)s is always smaller than 256. > > > > except when it isn't. see the ANSI C spec for details. > > Ah. I found. I'm very surprised for that. Thank you! :-) > BTW, do we really support architectures with 9bits-sized char? I would expect that a lot of our code assumes 8-bit characters, and I personally wouldn't mind if Python was limited to such platforms. They aren't very important for attracting new users, and certainly they don't seem to be a growing kind of platform... (Probably because so much other software makes the same assumption. :-) So IMO your fix is fine. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Fri Dec 19 10:40:13 2003 From: tim.one at comcast.net (Tim Peters) Date: Fri Dec 19 10:40:19 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <20031219115952.GA84061@i18n.org> Message-ID: [Hye-Shik Chang] > BTW, do we really support architectures with 9bits-sized char? I don't think so. There are assumptions that a char is 8 bits scattered throughout Python's code, not so much in the context of using characters *as* characters, but more indirectly by assuming that the number of *bits* in an object of a non-char type T can be computed as sizeof(T)*8. Skip's idea of making config smarter about this is a good one, but instead of trying to "fix stuff" for a case that's probably never going to arise, and that can't really be tested anyway until it does, I'd add a block like this everywhere we know we're relying on 8-bit char: #ifdef HAS_FUNNY_SIZE_CHAR #error "The following code needs rework when a char isn't 8 bits" #endif /* A comment explaining why the following code needs rework * when a char isn't 8 bits. */ Crays are a red herring here. It's true that some Cray *hardware* can't address anything smaller than 64 bits, and that's also true of some other architectures. char is nevertheless 8 bits on all such 64-bit boxes I know of (and since I worked in a 64-bit world for 15 years, I know about most of them ). On Crays, this is achieved (albeit at major expense) in software: by *software* convention, a pointer-to-char stores the byte offset in the *most*-significant 3 bits of a pointer, and long-winded generated coded picks that part at runtime, loading or storing 8 bytes at a time (the HW can't do less than that), shifting and masking and or'ing to give the illusion of byte addressing for char. Some Alphas do something similar, but that HW's loads and stores simply ignore the last 3 bits of a memory address, and the CPU has special-purpose instructions to help generated code do the subsequent extraction and insertion of 8-bit chunks efficiently and succinctly. From tismer at tismer.com Fri Dec 19 12:49:15 2003 From: tismer at tismer.com (Christian Tismer) Date: Fri Dec 19 12:47:42 2003 Subject: [Python-Dev] Last chance! In-Reply-To: <200312191531.hBJFVgw27430@c-24-5-183-134.client.comcast.net> References: <3FDE6D35.3090100@tismer.com> <3FE2C5B5.8080208@tismer.com> <200312191531.hBJFVgw27430@c-24-5-183-134.client.comcast.net> Message-ID: <3FE33A1B.5030903@tismer.com> Guido van Rossum wrote: ... > I saved the message, but haven't had the time yet to think things > through. > > I *did* notice at least one case where using f_tstate might actually > be a mistake: theoretically it's possible that two or more threads > alternate calling next() on a generator (if they wrap it in a critical > section); AFAICT the f_tstate is never updated. Right, f_tstate is never updated. I think there is another inconsistent situation, which can be created easily. If a generator is run by a different thread than it's creator, then the frame is run in that other thread. eval_fame correctly uses tstate, but if tracing is activated, call_trace uses frame->f_tstate for no obvious reason, which will probably mess up the tracing flags of the wrong thread. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From blink at msn.com Sat Dec 20 05:49:02 2003 From: blink at msn.com (Reuben Crenshaw) Date: Fri Dec 19 15:55:28 2003 Subject: [Python-Dev] You tried diets and workouts, add this and be amazed!! bpvwvczpyztptp Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031220/9fee7bad/attachment.html From Jack.Jansen at cwi.nl Fri Dec 19 17:57:02 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Fri Dec 19 17:57:07 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <200312191539.hBJFdjC27495@c-24-5-183-134.client.comcast.net> References: <20031219092309.GA81703@i18n.org> <20031219115952.GA84061@i18n.org> <200312191539.hBJFdjC27495@c-24-5-183-134.client.comcast.net> Message-ID: On 19-dec-03, at 16:39, Guido van Rossum wrote: >>>> AFAIK, *(unsigned char*)s is always smaller than 256. >>> >>> except when it isn't. see the ANSI C spec for details. >> >> Ah. I found. I'm very surprised for that. Thank you! :-) >> BTW, do we really support architectures with 9bits-sized char? > > I would expect that a lot of our code assumes 8-bit characters, and I > personally wouldn't mind if Python was limited to such platforms. Then there is a lot of code that could be tuned for this. Since I'm using gcc 3.3 (which came with OSX 10.3) I get lots of warnings about comparisons that are always true due to the size of the operands. I looked at a couple of these and I think they were all char-related. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From jack at performancedrivers.com Fri Dec 19 16:30:31 2003 From: jack at performancedrivers.com (Jack Diederich) Date: Fri Dec 19 22:03:52 2003 Subject: [Python-Dev] Relative import In-Reply-To: <20031218031645.GH473@frobozz> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> Message-ID: <20031219213031.GD1246@wopr> On Thu, Dec 18, 2003 at 02:16:45PM +1100, Andrew Bennetts wrote: > On Wed, Dec 17, 2003 at 10:08:21PM -0500, Barry Warsaw wrote: > > On Wed, 2003-12-17 at 22:04, Andrew Bennetts wrote: > > > On Wed, Dec 17, 2003 at 09:33:43PM -0500, Barry Warsaw wrote: > > > > > > > > That's why when I tend to think about this, I start wanting a way to > > > > spell "definitely give me the global one, no matter what". IOW, I feel > > > > like I want a way to bypass relative module lookups. > > > > > > Alternatively, maybe what you want is a way to say "definitely give me the > > > standard library one, no matter what", e.g. > > > > > > from stdlib import codecs > > > > Interesting. I see the subtle distinction. One searches only the > > standard library, the other searches every directory on sys.path. I'm > > not sure I'd have much need to restrict the search to just the standard > > library. Plus, would that include site-packages? > > I should have been clearer. What I'm basically proposing is to make > site-package called "stdlib", leaving a default python install with a very > uncluttered top-level namespace, thus reducing unexpected name conflicts. I'm at least a little confused with what people want, as far as I can tell import semantics have three things * what module to import * what name to give that import * where to look for it The 'from' keyword seems to be getting overloaded to mean both what module and where to find it, which the 'stdlib' tries to resolve. Most of the discussion has been about how to improve 'where to look' I like syntax that reads most important left-to-right, so what about from MODULE import NAMES as RENAME searching HOW or import NAMES as RENAME from MODULE searching HOW searching could be 'asbolute' 'relative' or any of the other suggested words. If you wanted to get fancy it could be a list, if it isn't a list people who truly care could cascade try/except on ImportError. -jackdied ps, I like the second version better but it is less like the current version, which is a drawback. From barry at python.org Fri Dec 19 23:13:15 2003 From: barry at python.org (Barry Warsaw) Date: Fri Dec 19 23:13:26 2003 Subject: [Python-Dev] Relative import In-Reply-To: <20031219213031.GD1246@wopr> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> Message-ID: <1071893594.1718.166.camel@anthem> On Fri, 2003-12-19 at 16:30, Jack Diederich wrote: > I'm at least a little confused with what people want, as far as I can tell > import semantics have three things > * what module to import > * what name to give that import > * where to look for it > > The 'from' keyword seems to be getting overloaded to mean both what module and > where to find it, which the 'stdlib' tries to resolve. Most of the > discussion has been about how to improve 'where to look' Very interesting observation. > I like syntax that reads most important left-to-right, so what about > from MODULE import NAMES as RENAME searching HOW > or > import NAMES as RENAME from MODULE searching HOW > > searching could be 'asbolute' 'relative' or any of the other suggested > words. If you wanted to get fancy it could be a list, if it isn't a list > people who truly care could cascade try/except on ImportError. I want to think about that more, but at first glance, it has some appeal. Neat! -Barry From tim.one at comcast.net Fri Dec 19 23:32:45 2003 From: tim.one at comcast.net (Tim Peters) Date: Fri Dec 19 23:32:51 2003 Subject: [Python-Dev] Relative import In-Reply-To: <1071893594.1718.166.camel@anthem> Message-ID: [Jack Diederich] >> ... >> I like syntax that reads most important left-to-right, so what about >> from MODULE import NAMES as RENAME searching HOW >> or >> import NAMES as RENAME from MODULE searching HOW >> >> searching could be 'asbolute' 'relative' or any of the other >> suggested words. If you wanted to get fancy it could be a list, if >> it isn't a list people who truly care could cascade try/except on >> ImportError. [Barry Warsaw] > I want to think about that more, but at first glance, it has some > appeal. Neat! Ditto. I think Jack's on to something there. I particularly like the second form, as it's an ongoing irritation to me that some imports begin with "from" and others with "import". In the current "from" form, the most important thing to me is almost never the module I'm importing the thing from, but (of course) the thing I'm importing. Going from import math to import sin from math (or vice versa) is also an easier edit than from import math to from math import sin (because the former doesn't require transposition). Adding a distinct clause to influence search details seems downright Pythonic. Maybe *too* Pythonic . From tim.one at comcast.net Sat Dec 20 00:22:22 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Dec 20 00:22:27 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <200312191539.hBJFdjC27495@c-24-5-183-134.client.comcast.net> Message-ID: [Guido] > I would expect that a lot of our code assumes 8-bit characters, and I > personally wouldn't mind if Python was limited to such platforms. > They aren't very important for attracting new users, and certainly > they don't seem to be a growing kind of platform... (Probably because > so much other software makes the same assumption. :-) Fine by me too. The first mainframe I used was a Univac 1108. There were a *lot* of competing HW architectures at that time, and manufacturers didn't agree about character size any more than they agreed about floating-point format or semantics, or the natural size of "a word". Univac was forward-looking, though: they didn't want their hardware to become obsolete if a different character size than the one they preferred clicked, so a control bit in the CPU could be set to treat their 36-bit words as either 6 6-bit characters, or as 4 9-bit characters. It worked! We're *still* equally comfortable with 6-bit bytes as with 9-bit bytes . I was betting on 6-bit bytes at the time, because that also worked well with CDC's 60-bit words. FORTRAN didn't even admit to the existence of lower case at the time, so 64 characters was way more than enough for anything anyone really needed to say to a computer. half-the-bits-in-these-new-fangled-bytes-are-just-wasted-ly y'rs - tim From whisper at oz.net Sat Dec 20 00:32:26 2003 From: whisper at oz.net (David LeBlanc) Date: Sat Dec 20 00:51:36 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: Message-ID: > [Guido] > > I would expect that a lot of our code assumes 8-bit characters, and I > > personally wouldn't mind if Python was limited to such platforms. > > They aren't very important for attracting new users, and certainly > > they don't seem to be a growing kind of platform... (Probably because > > so much other software makes the same assumption. :-) > > Fine by me too. > > The first mainframe I used was a Univac 1108. There were a *lot* of > competing HW architectures at that time, and manufacturers didn't agree > about character size any more than they agreed about floating-point format > or semantics, or the natural size of "a word". Univac was > forward-looking, > though: they didn't want their hardware to become obsolete if a different > character size than the one they preferred clicked, so a control > bit in the > CPU could be set to treat their 36-bit words as either 6 6-bit characters, > or as 4 9-bit characters. It worked! We're *still* equally comfortable > with 6-bit bytes as with 9-bit bytes . > > I was betting on 6-bit bytes at the time, because that also > worked well with > CDC's 60-bit words. FORTRAN didn't even admit to the existence of lower > case at the time, so 64 characters was way more than enough for anything > anyone really needed to say to a computer. > > half-the-bits-in-these-new-fangled-bytes-are-just-wasted-ly y'rs - tim > I would think the lesson to be learned from this is that one should not lock the software into any particular number of bits per character. The coming flood of 64 bit machines could make 16 bit unicode attractive. It's an ever more global world and "we" should keep in mind that in the next decade most of the world's programming is going to be done in India and China if American corporations have their way. soon-to-be-looking-for-the-carton-the-mainframe-came-in-to-live-ingly-y'rs Dave LeBlanc Seattle, WA USA From zack at codesourcery.com Fri Dec 19 13:08:52 2003 From: zack at codesourcery.com (Zack Weinberg) Date: Sat Dec 20 02:35:53 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "18 Dec 2003 22:28:50 +0100") References: <87y8tbezcq.fsf@codesourcery.com> Message-ID: <87wu8sy91n.fsf@egil.codesourcery.com> martin@v.loewis.de (Martin v. L?wis) writes: > "Zack Weinberg" writes: > >> > This is what leads to the bletcherous >> > >> > do { >> > the real macro guts go here >> > } while(0) >> > >> > form of macro definition. Py_DECREF could be rewritten in that form -- but >> > I've puked enough for one day . >> >> Should, not just could. Yeah, it's bletcherous, but it's the only >> way to be sure. (Short of inline functions.) > > Not true. As Tim explains, there is no possible application of the > macro which gets misinterpreted. We don't have a single if-statement, > we have an if-else-statement in the macro. That never leads to the > dangling else problem. It does, however, lead to a different problem: if (condition) Py_DECREF (foo); else // oops, syntax error here ... In general I consider it appropriate to wrap _all_ macros that don't return a value in do { ... } while (0), because that way you don't have to worry about which of them truly need it. zw From arigo at tunes.org Sat Dec 20 03:43:57 2003 From: arigo at tunes.org (Armin Rigo) Date: Sat Dec 20 03:48:49 2003 Subject: [Python-Dev] Re: [pypy-dev] Last chance! (was: Does anybody really use frame->f_tstate ?) In-Reply-To: <3FE2C5B5.8080208@tismer.com> References: <3FDE6D35.3090100@tismer.com> <3FE2C5B5.8080208@tismer.com> Message-ID: <20031220084357.GA25998@vicky.ecs.soton.ac.uk> Hello Christian, On Fri, Dec 19, 2003 at 10:32:37AM +0100, Christian Tismer wrote: > since I didn't get *any* reply to this request, > either the request was bad or there is really > nobody using f_tstate in a way that makes it > urgent to keep. Sorry for the delay, we were kind of busy this week here... There are references to f_tstate in Psyco but nothing critical. Using PyThreadState_GET() instead should be fine. A bientot, Armin. From ncoghlan at iinet.net.au Sat Dec 20 04:09:48 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Dec 20 04:11:46 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: References: Message-ID: <3FE411DC.3010401@iinet.net.au> Tim Peters wrote: > [Hye-Shik Chang] > >>BTW, do we really support architectures with 9bits-sized char? [...] > Skip's idea of making config smarter about this is a good one, but instead > of trying to "fix stuff" for a case that's probably never going to arise, > and that can't really be tested anyway until it does, I'd add a block like > this everywhere we know we're relying on 8-bit char: > > #ifdef HAS_FUNNY_SIZE_CHAR > #error "The following code needs rework when a char isn't 8 bits" > #endif > /* A comment explaining why the following code needs rework > * when a char isn't 8 bits. > */ This would probably be appropriate for those TI DSP's I mentioned. While they genuinely have a 16-bit char type, they're also intended for use as co-processors, rather than as the main controller for an application. That is, a more standard CPU should be used to handle the general application programming, while the DSP is used for the serious number crunching (that's what it is made for, after all). Anyone who _thinks_ they want to run Python on the DSP core almost certainly needs to have a long hard think about their system design. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From martin at v.loewis.de Sat Dec 20 05:15:21 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sat Dec 20 05:17:51 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <20031219012347.GB4692@unpythonic.net> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> Message-ID: Jeff Epler writes: > > Can you find out what $$ is, and what the PIDs and thread IDs of all > > participating threads are? > > I'm not sure what all information I should try to gather for you. Let me > know if you think this is enough to file a bug report with... I changed > the example to make it clearer that it's the subprocess ignoring the > signal that is the problem, not anything in Python that is taking time > to notice the death of a child process. That is an important observation; signals that are blocked in the parent process will be blocked in the child process as well. I'm not sure what to do about this: We apparently *want* the signals blocked in the thread, but we don't want them to be blocked in the process invoked through system(). Proposals are welcome. Regards, Martin From martin at v.loewis.de Sat Dec 20 05:23:43 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sat Dec 20 05:24:56 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: References: Message-ID: "David LeBlanc" writes: > I would think the lesson to be learned from this is that one should > not lock the software into any particular number of bits per > character. The coming flood of 64 bit machines could make 16 bit > unicode attractive. You are talking about an entirely different issue here. This thread is about the number of bits in a "char", which, in C, is the same thing as a "byte". The number of bits for a "character" is independent. 16-bit Unicode is attractive already, although it is dying, to make place for 32-bit Unicode. However, new 64-bit architectures will make sure they support an 8-bit data type, and compiler vendors will make sure that "char" maps to that 8-bit data type (most likely, they also will make char signed by default). There is just too much software that breaks if you could not address bytes anymore. Primarily, the entire networking interfaces would break down, which is a risk that new architectures are unlikely to take. > It's an ever more global world and "we" should keep in mind that in > the next decade most of the world's programming is going to be done > in India and China if American corporations have their way. Certainly, but unrelated to the issue at hand. Regards, Martin From martin at v.loewis.de Sat Dec 20 05:31:18 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sat Dec 20 05:40:20 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <87wu8sy91n.fsf@egil.codesourcery.com> References: <87y8tbezcq.fsf@codesourcery.com> <87wu8sy91n.fsf@egil.codesourcery.com> Message-ID: "Zack Weinberg" writes: > It does, however, lead to a different problem: > > if (condition) > Py_DECREF (foo); > else // oops, syntax error here How so? This expands to if(condition) if(cond2)action1; else action2; else ... This is perfectly well-formed C, and it groups as my indentation suggests. There is no real problem with this macro. It is just gcc complaining falsely. > In general I consider it appropriate to wrap _all_ macros that don't > return a value in do { ... } while (0), because that way you don't > have to worry about which of them truly need it. Hmm. Yes, this could be done - but it is unfortunate that this needs to be done only to silence gcc. Regards, Martin From lkcl at lkcl.net Sat Dec 20 06:21:20 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Sat Dec 20 06:21:00 2003 Subject: [Python-Dev] Capabilities - published interfaces Message-ID: <20031220112120.GE1933@lkcl.net> following on, with a change of subject, from the rexec / acls discussions, i believe that i have finally worked out why acls are not the preferred way forward, leaving speed disadvantages aside. ACLs need to be applied subtly and explicitly on every important type of object. if they're not designed in at the time they are a PAIN to retro-fit. ACLs need to be applied subtly because if they are applied unsubtly you end up with more gates than you do green pastures - more roundabouts than roads (viz, milton keynes). ACLs need to be applied explicitly because of their nature: you have something (it could be the right to "see" something) and you try do _do_ something and are explicitly told whether it worked or not (or are explicitly fooled into believing that something cannot be "seen). from a code security review perspective, it's actually easier to review ACL-restricted code than it is capabilities-based code, but i'll expand on that later. Capabilities are like the difference between python 0.0 and python 2.5. python 0.0 is "capable" of doing certain things - however python 2.5 is "capable" of doing a heck of a lot more, simply because the functionality in python 0.0 doesn't _exist_. The principle behind Capabilities is therefore to simply _not give_ python 2.5 users _any_ means to use, view, see, certain functionality i.e. make it look like it doesn't exist. in other words [and this is the bit that i could _really_ have done with, two days ago], when a python user accessess __globals__, __builtins__, __locals__, an object's __class__, they find that the functionality they may be looking for literally _isn't there_. it's as if someone had recompiled python 2.5 #ifdeffing out a whole boat-load of code and forgot to include some modules. the advantage of Capabilities over ACLs is that there's no chance of making a mistake on the ACLs, because access to something that, as far as the user is concerned, doesn't _exist_, is just a non-issue. the other advantage is speed. the _dis_advantage of Capabilities over ACLs comes when an attempt to combine unrestricted Capability code with restricted Capability code in the same codebase is made, and a security audit is called. the nature of restricted Capabilities is such that it is "by omission" that you achieve security. how the heck are you going to do a security audit looking for things that are omitted, rather than included, especially in amongst a full set of code that has the full functionality? by contrast, with ACLs, you can check everything (in a regimented fashion) and if it don't have an ACL, you can at least ask _why_ it don't have an ACL. so, with that explanation in mind, i'm going to write up a separate message outlining some of the ways in which i believe the capabilities restriction system that is already in python can be improved. l. From lkcl at lkcl.net Sat Dec 20 07:19:42 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Sat Dec 20 07:19:23 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220112120.GE1933@lkcl.net> References: <20031220112120.GE1933@lkcl.net> Message-ID: <20031220121942.GF1933@lkcl.net> capabilities: the means to make code disappear. thing is that there is a lot of code. some of the boundaries have already been highlighted where the "Capabilities" have not sufficiently been restricted. object.__class__ is a good example. Access to __class__ ------------------- the obvious solution to fix that particular problem is to turn the access to __class__ around. the problem is that by creating objects, as many __class__es are created as there are objects. considering restricting access to all of those __class__es is totally daunting and impractical. ... but look at it closely: they're all the same function name, so it's not necessarily the individual __class__es that could be restricted but the method by which __class__ is accessed. if you "feed" access to __class__ via a single function, you have a bottleneck again which can be replaced. so, the proposal is simple: create an __builtin__ function called __get_class__ through which access to __class__ is made. it takes, as a first argument, the class itself. obj.__class__ is redirected to it (__builtin__.__get_class__(obj)) to maintain code compatibility. then it becomes possible to over-ride the __get_class__ function and to "vet" it in a restricted execution mode. again, it's worth reiterating here that if it wasn't for the fact that python is run "initiated" in "unrestricted" mode and "jumps down" to "restricted" execution, this wouldn't be a problem. the reason for that is because if you ran python initiated in a file-restricted mode from startup, no unrestricted file objects with write permission would be created, such that there would be zero risk of anyone _getting_ an unrestricted file object, through which the file.__class__.__new__(file.__class__, ....) method could be called. anyway. if that is not practical (for whatever reason), then you leave it alone, and consider the function __new__ instead, which i believe is the _real_ reason why access to __class__ is considered bad. again, turning it round by creating a function __builtin__.__new__() is a way forward because that function can be restricted. Access to __builtins__ and __globals__ from the stack. ----------------------------------------------------- i don't know if this is the case, but it's worth mentioning, as background, to avoid potential confusion. there's the concept of globals, locals and there's __builtins__ in the python language. in the _implementation_ of the python language, there are c-variables which may be global, may be local: possible conceptual confusion between the implementation and the language need to be avoided! my question is: _is_ it possible to make a function call in python such that that function's "view" of what the globals, locals and __builtins__ are are COMPLETELY different from what the CALLER of that function's view of globals, locals and __builtins__? in other words, is globals() global in the python implementation, or is globals() copied on the function-call stack frame in the python implementation? [or other] what i am getting at is that it needs to be possible, if capabilities are to work, to be able to make a transition in the python language from a full-featured mode in one function call into a totally restricted mode in another function call, and to be able to come back again. _without_ the entry into the restricted mode having any damaging consequences on the full-featured mode. i note, for example, in the rexec.py code, that only self.locals is modified in the runcode() example RestrictedConsole class. why is only self.locals['__builtins__'] restricted, not self.globals? am i missing something here? surely the globals should be modified too? what happens if self.globals is reassigned? Access to interfaces -------------------- one of the things that is strangely lacking in python is the ability to restrict access to python objects, a la public, protected and private from c++. perl users find this to be utterly incomprehensible and reprehensible, especially the bits where conventions are obeyed - and followed! - about putting underscores in front of function names. from a restricted execution perspective, this is not really okay. what i propose is that two additional per-class functions (or lists) be added (two more functions later) which can ONLY be called from __init__: they are called __set_public_interface__() and __set_protected_interface__(). these functions take, as arguments, a list - or simpler yet, two variables are added which can only be initialised in __init__: __public_interface__ and __protected_interface__. basically they consist of lists of functions that can be accessed publicly and by inheriting classes respectively. the reason for not specifying a list of private functions is because it then becomes necessary to explicitly search through that list excluding things from external access, which is not the way "Capabilities" are done. if the lists are None, there are no restrictions. if the lists are empty, no functions are accessible - which is different. what i also propose is that it be possible to make a copy of a class and then _remove_ items from these two lists. it must _not_ be possible to either add to or to replace the list items or the lists themselves. there's quite a lot here. i hope you find it useful. l. From aahz at pythoncraft.com Sat Dec 20 09:08:43 2003 From: aahz at pythoncraft.com (Aahz) Date: Sat Dec 20 09:08:55 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220121942.GF1933@lkcl.net> References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> Message-ID: <20031220140842.GA5627@panix.com> On Sat, Dec 20, 2003, Luke Kenneth Casson Leighton wrote: > > one of the things that is strangely lacking in python is the ability > to restrict access to python objects, a la public, protected and > private from c++. > > perl users find this to be utterly incomprehensible and reprehensible, > especially the bits where conventions are obeyed - and followed! - > about putting underscores in front of function names. > > from a restricted execution perspective, this is not really okay. Yup. That really is the fundamental issue. Python has been deliberately designed to let programmers do whatever they want; it's a "let the grownups play" perspective. That's part of the reason restricted execution hasn't received more attention: it's fundamentally opposed to Python's design. Another major reason is because restricted execution from within Python can never completely solve the problem unless we devote vast amounts of effort. Consider the following two snippets of code: 100 ** 100 ** 100 [None] * (10 ** 10) The first chews up CPU; the second chews up memory. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From lkcl at lkcl.net Sat Dec 20 09:39:10 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Sat Dec 20 09:38:47 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220140842.GA5627@panix.com> References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> <20031220140842.GA5627@panix.com> Message-ID: <20031220143910.GA19472@lkcl.net> On Sat, Dec 20, 2003 at 09:08:43AM -0500, Aahz wrote: > On Sat, Dec 20, 2003, Luke Kenneth Casson Leighton wrote: > > > > one of the things that is strangely lacking in python is the ability > > to restrict access to python objects, a la public, protected and > > private from c++. > > > > perl users find this to be utterly incomprehensible and reprehensible, > > especially the bits where conventions are obeyed - and followed! - > > about putting underscores in front of function names. > > > > from a restricted execution perspective, this is not really okay. > > Yup. That really is the fundamental issue. Python has been deliberately > designed to let programmers do whatever they want; it's a "let the > grownups play" perspective. as python moves into a more mainstream acceptance, it becomes more of an issue to let the kiddies bash themselves with rubber hammers. [i mention this because a few years ago, a six year old bashed his dad over the head with a hammer after watching tom and jerry beat the stuffing out of each other on telly.] > Python's design. Another major reason is because restricted execution > from within Python can never completely solve the problem unless we > devote vast amounts of effort. Consider the following two snippets of > code: > > 100 ** 100 ** 100 > [None] * (10 ** 10) > > The first chews up CPU; the second chews up memory. i infer from these two observations that some of the requirements of restricted execution could be to conserve CPU and memory? l. From aahz at pythoncraft.com Sat Dec 20 10:16:29 2003 From: aahz at pythoncraft.com (Aahz) Date: Sat Dec 20 10:16:33 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220143910.GA19472@lkcl.net> References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> <20031220140842.GA5627@panix.com> <20031220143910.GA19472@lkcl.net> Message-ID: <20031220151629.GA24651@panix.com> On Sat, Dec 20, 2003, Luke Kenneth Casson Leighton wrote: > On Sat, Dec 20, 2003 at 09:08:43AM -0500, Aahz wrote: >> On Sat, Dec 20, 2003, Luke Kenneth Casson Leighton wrote: >>> >>> one of the things that is strangely lacking in python is the ability >>> to restrict access to python objects, a la public, protected and >>> private from c++. >>> >>> from a restricted execution perspective, this is not really okay. >> >> Yup. That really is the fundamental issue. Python has been deliberately >> designed to let programmers do whatever they want; it's a "let the >> grownups play" perspective. > > as python moves into a more mainstream acceptance, it becomes more > of an issue to let the kiddies bash themselves with rubber hammers. That's an assertion. I think to a certain extent you'll need to prove your assertion. >> Another major reason is because restricted execution from within >> Python can never completely solve the problem unless we devote vast >> amounts of effort. Consider the following two snippets of code: >> >> 100 ** 100 ** 100 >> [None] * (10 ** 10) >> >> The first chews up CPU; the second chews up memory. > > i infer from these two observations that some of the requirements > of restricted execution could be to conserve CPU and memory? Exactly. From my observations of these discussions, there are essentially only two reasons for restricted execution: * To simplify things by reducing the potential solution space * To protect a system against a hostile attacker There are already many ways to achieve the first objective in Python without restricted execution. For the second, there's really no way to succeed without help from the operating system, unless you run a server with a limited set of capabilities that does *not* include random code. There are already plenty of servers written in Python.... Supposedly there's a middle ground of untrusted but non-hostile code, but what's the point of providing support for that? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From lkcl at lkcl.net Sat Dec 20 10:30:22 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Sat Dec 20 10:30:00 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220151629.GA24651@panix.com> References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> <20031220140842.GA5627@panix.com> <20031220143910.GA19472@lkcl.net> <20031220151629.GA24651@panix.com> Message-ID: <20031220153022.GB19472@lkcl.net> On Sat, Dec 20, 2003 at 10:16:29AM -0500, Aahz wrote: > > as python moves into a more mainstream acceptance, it becomes more > > of an issue to let the kiddies bash themselves with rubber hammers. > > That's an assertion. I think to a certain extent you'll need to prove > your assertion. 'm a bit worried that such proofs would involve painting large metal hammers and covering them in plastic to make them look less real. the resultant cracked skills could possibly land someone in jail. > Supposedly there's a middle ground of untrusted but non-hostile code, > but what's the point of providing support for that? the example that i gave that was because i wanted to offer a subset of python functionality to end-users such that they could run DNS lookups, pings, check a web page existed, telnet to a box, run commands and check the output. so it's running a user's python code on a server where there is a networked host being analysed by the user. what i _didn't_ want to happen was for that user to run code on the server that could damage the server and interfere with the 100 or so _other_ programs running to analyse 100 _other_ hosts. the whole point of using python was to avoid having to write an new programming language. what i came up with was very very useful. to some extent, i didn't care about things like __class__ because 1) the users weren't that bright. 2) the user's weren't that hostile. rexec fitted the requirements perfectly - and it still does: it's just been disabled and also changed into something that stops even the library functions from writing to log files. i couldn't even use the MySQLdb module which was kinda critical to the database-driven backend. l. From mwh at python.net Sat Dec 20 10:51:34 2003 From: mwh at python.net (Michael Hudson) Date: Sat Dec 20 10:51:39 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "20 Dec 2003 11:15:21 +0100") References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> Message-ID: <2mwu8rebcp.fsf@starship.python.net> martin@v.loewis.de (Martin v. L?wis) writes: > Jeff Epler writes: > >> > Can you find out what $$ is, and what the PIDs and thread IDs of all >> > participating threads are? >> >> I'm not sure what all information I should try to gather for you. Let me >> know if you think this is enough to file a bug report with... I changed >> the example to make it clearer that it's the subprocess ignoring the >> signal that is the problem, not anything in Python that is taking time >> to notice the death of a child process. > > That is an important observation; signals that are blocked in the > parent process will be blocked in the child process as well. > > I'm not sure what to do about this: We apparently *want* the signals > blocked in the thread, but we don't want them to be blocked in the > process invoked through system(). Proposals are welcome. Does pthread_atfork() help? Cheers, mwh -- We've had a lot of problems going from glibc 2.0 to glibc 2.1. People claim binary compatibility. Except for functions they don't like. -- Peter Van Eynde, comp.lang.lisp From aahz at pythoncraft.com Sat Dec 20 10:55:48 2003 From: aahz at pythoncraft.com (Aahz) Date: Sat Dec 20 10:55:53 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220153022.GB19472@lkcl.net> References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> <20031220140842.GA5627@panix.com> <20031220143910.GA19472@lkcl.net> <20031220151629.GA24651@panix.com> <20031220153022.GB19472@lkcl.net> Message-ID: <20031220155548.GA7420@panix.com> On Sat, Dec 20, 2003, Luke Kenneth Casson Leighton wrote: > On Sat, Dec 20, 2003 at 10:16:29AM -0500, Aahz wrote: >> >> Supposedly there's a middle ground of untrusted but non-hostile code, >> but what's the point of providing support for that? > > the example that i gave that was because i wanted to offer a subset > of python functionality to end-users such that they could run > DNS lookups, pings, check a web page existed, telnet to a box, > run commands and check the output. > > to some extent, i didn't care about things like __class__ because > 1) the users weren't that bright. > 2) the user's weren't that hostile. Yup. By "what's the point?" I didn't mean that there were no use cases; the problem is that such cases are not frequent enough to justify the effort. > rexec fitted the requirements perfectly - and it still does: it's > just been disabled and also changed into something that stops even > the library functions from writing to log files. > i couldn't even use the MySQLdb module which was kinda critical to > the database-driven backend. Well, you're free to maintain rexec as a separate project (or borrow from the still-maintained Zope system). But anything shipped as part of Python can't afford to assume your points 1) and 2). -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From lkcl at lkcl.net Sat Dec 20 11:35:51 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Sat Dec 20 11:35:28 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220155548.GA7420@panix.com> References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> <20031220140842.GA5627@panix.com> <20031220143910.GA19472@lkcl.net> <20031220151629.GA24651@panix.com> <20031220153022.GB19472@lkcl.net> <20031220155548.GA7420@panix.com> Message-ID: <20031220163551.GC19472@lkcl.net> On Sat, Dec 20, 2003 at 10:55:48AM -0500, Aahz wrote: > > to some extent, i didn't care about things like __class__ because > > 1) the users weren't that bright. > > 2) the user's weren't that hostile. > > Yup. By "what's the point?" I didn't mean that there were no use cases; > the problem is that such cases are not frequent enough to justify the > effort. ... which is why i made some recommendations to add in the concept of run-time-defineable public and protected class interfaces. such a concept 1) fits with the principle of capabilities 2) is an enhancement that goes beyond the small requirements of restricted execution 3) offers a means through which rexec can be implemented. > > rexec fitted the requirements perfectly - and it still does: it's > > just been disabled and also changed into something that stops even > > the library functions from writing to log files. > > i couldn't even use the MySQLdb module which was kinda critical to > > the database-driven backend. > > Well, you're free to maintain rexec as a separate project (or borrow > from the still-maintained Zope system). But anything shipped as part of > Python can't afford to assume your points 1) and 2). i appreciate that. so it turns into a wishlist: a class named RExecDontUseThisItsBrokenForMostPeople and a class named RExec which simply pulls that exception. l. From aahz at pythoncraft.com Sat Dec 20 11:39:31 2003 From: aahz at pythoncraft.com (Aahz) Date: Sat Dec 20 11:39:35 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220163551.GC19472@lkcl.net> References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> <20031220140842.GA5627@panix.com> <20031220143910.GA19472@lkcl.net> <20031220151629.GA24651@panix.com> <20031220153022.GB19472@lkcl.net> <20031220155548.GA7420@panix.com> <20031220163551.GC19472@lkcl.net> Message-ID: <20031220163931.GA20862@panix.com> On Sat, Dec 20, 2003, Luke Kenneth Casson Leighton wrote: > On Sat, Dec 20, 2003 at 10:55:48AM -0500, Aahz wrote: >>Luke: >>> >>> to some extent, i didn't care about things like __class__ because >>> 1) the users weren't that bright. >>> 2) the user's weren't that hostile. >> >> Yup. By "what's the point?" I didn't mean that there were no use cases; >> the problem is that such cases are not frequent enough to justify the >> effort. > > ... which is why i made some recommendations to add in the concept > of run-time-defineable public and protected class interfaces. > > such a concept 1) fits with the principle of capabilities 2) > is an enhancement that goes beyond the small requirements of > restricted execution 3) offers a means through which rexec > can be implemented. Sure. *If* we decide to implement restricted execution. As I made clear with my comments about CPU and memory usage, there's still no consensus that adding a restricted execution environment to Python represents a good use of community resources. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From guido at python.org Fri Dec 19 13:15:24 2003 From: guido at python.org (Guido van Rossum) Date: Sat Dec 20 12:15:31 2003 Subject: [Python-Dev] Last chance! In-Reply-To: Your message of "Fri, 19 Dec 2003 18:49:15 +0100." <3FE33A1B.5030903@tismer.com> References: <3FDE6D35.3090100@tismer.com> <3FE2C5B5.8080208@tismer.com> <200312191531.hBJFVgw27430@c-24-5-183-134.client.comcast.net> <3FE33A1B.5030903@tismer.com> Message-ID: <200312191815.hBJIFP528036@c-24-5-183-134.client.comcast.net> > > I *did* notice at least one case where using f_tstate might actually > > be a mistake: theoretically it's possible that two or more threads > > alternate calling next() on a generator (if they wrap it in a critical > > section); AFAICT the f_tstate is never updated. > > Right, f_tstate is never updated. > I think there is another inconsistent > situation, which can be created easily. > If a generator is run by a different thread > than it's creator, then the frame is run in that > other thread. eval_fame correctly uses tstate, > but if tracing is activated, call_trace uses > frame->f_tstate for no obvious reason, which > will probably mess up the tracing flags of the wrong > thread. Right. Could you dig through CVS logs to find out when f_tstate was first introduced? Maybe there's a clue about why there. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Sat Dec 20 11:27:34 2003 From: skip at pobox.com (Skip Montanaro) Date: Sat Dec 20 12:18:27 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: References: <200312191539.hBJFdjC27495@c-24-5-183-134.client.comcast.net> Message-ID: <16356.30838.750692.236147@montanaro.dyndns.org> Tim> [Guido] >> I would expect that a lot of our code assumes 8-bit characters, and I >> personally wouldn't mind if Python was limited to such platforms. Tim> Fine by me too. Then how about adding #if UCHAR_MAX != 255 #error "Python's source code currently assumes 8-bit characters." #endif right after the HAVE_LIMITS_H test? Skip From python at rcn.com Sat Dec 20 13:47:56 2003 From: python at rcn.com (Raymond Hettinger) Date: Sat Dec 20 13:48:26 2003 Subject: [Python-Dev] Last chance! In-Reply-To: <200312191815.hBJIFP528036@c-24-5-183-134.client.comcast.net> Message-ID: <000201c3c729$c8ffcde0$e841fea9@oemcomputer> > > > I *did* notice at least one case where using f_tstate might actually > > > be a mistake: theoretically it's possible that two or more threads > > > alternate calling next() on a generator (if they wrap it in a critical > > > section); AFAICT the f_tstate is never updated. > > > > Right, f_tstate is never updated. > > I think there is another inconsistent > > situation, which can be created easily. > > If a generator is run by a different thread > > than it's creator, then the frame is run in that > > other thread. eval_fame correctly uses tstate, > > but if tracing is activated, call_trace uses > > frame->f_tstate for no obvious reason, which > > will probably mess up the tracing flags of the wrong > > thread. > > Right. > > Could you dig through CVS logs to find out when f_tstate was first > introduced? Maybe there's a clue about why there. Here is a head start on the research. The ceval.c use of tstate goes back to the introduction of generators in 2001. The use in traceback.c and sysmodule.c goes back to 1997 when per-thread globals were factored into a structure to support separate thread-state management. Prior to that, the history is more diffuse and harder to follow. Raymond Hettinger --------------------------------------------------------------------- Ceval.c -------- 2.252 (nascheme 21-Jun-01): PyThreadState *tstate = 2.252 (nascheme 21-Jun-01): * necessarily their creator. */ 2.255 (tim_one 23-Jun-01): Py_XINCREF(tstate->frame); 2.252 (nascheme 21-Jun-01): assert(f->f_back == NULL); 2.252 (nascheme 21-Jun-01): f->f_back = tstate->frame; 2.252 (nascheme 21-Jun-01): revision 2.255 date: 2001/06/23 05:47:56; author: tim_one; state: Exp; lines: +2 -2 gen_iternext(): Don't assume that the current thread state's frame is not NULL. I don't think it can be NULL from Python code, but if using generators via the C API I expect a NULL frame is possible. revision 2.252 date: 2001/06/21 02:41:10; author: nascheme; state: Exp; lines: +27 -14 Try to avoid creating reference cycles involving generators. Only keep a reference to f_back when its really needed. Do a little whitespace normalization as well. This whole file is a big war between tabs and spaces but now is probably not the time to reindent everything. traceback.c ------------ 2.22 (guido 05-May-97): PyThreadState *tstate = frame->f_tstate; 2.22 (guido 05-May-97): tracebackobject *oldtb = (tracebackobjec revision 2.22 date: 1997/05/05 20:56:15; author: guido; state: Exp; lines: +6 -30 Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. The story for their life a globals pre-dates what I can sysmodule.c ------------ 2.45 (guido 02-Aug-97): PyThreadState *tstate = PyThreadState_Ge 2.45 (guido 02-Aug-97): PyObject *sd = tstate->interp->sysdict; 2.57 (guido 05-Oct-99): if (sd == NULL) 2.41 (guido 05-May-97): PyThreadState *tstate; 2.41 (guido 05-May-97): tstate = PyThreadState_Get(); 2.41 (guido 05-May-97): return Py_BuildValue( 2.41 (guido 05-May-97): "(OOO)", 2.41 (guido 05-May-97): tstate->exc_type != NULL ? tstat 2.41 (guido 05-May-97): tstate->exc_value != NULL ? tsta 2.41 (guido 05-May-97): tstate->exc_traceback != NULL ? 2.41 (guido 05-May-97): tstate->exc_traceback : 2.41 (guido 05-May-97): } 2.41 (guido 05-May-97): revision 2.41 date: 1997/05/05 20:56:11; author: guido; state: Exp; lines: +30 -9 Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. Details of the 2.41 checkin ---------------------------- Index: sysmodule.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Python/sysmodule.c,v retrieving revision 2.40 retrieving revision 2.41 diff -c -r2.40 -r2.41 *** sysmodule.c 29 Apr 1997 20:42:30 -0000 2.40 --- sysmodule.c 5 May 1997 20:56:11 -0000 2.41 *************** *** 48,56 **** #include "osdefs.h" - PyObject *_PySys_TraceFunc, *_PySys_ProfileFunc; - int _PySys_CheckInterval = 10; - #if HAVE_UNISTD_H #include #endif --- 48,53 ---- *************** *** 98,103 **** --- 95,119 ---- } static PyObject * + sys_exc_info(self, args) + PyObject *self; + PyObject *args; + { + PyThreadState *tstate; + if (!PyArg_Parse(args, "")) + return NULL; + tstate = PyThreadState_Get(); + if (tstate == NULL) + Py_FatalError("sys.exc_info(): no thread state"); + return Py_BuildValue( + "(OOO)", + tstate->exc_type != NULL ? tstate->exc_type : Py_None, + tstate->exc_value != NULL ? tstate->exc_value : Py_None, + tstate->exc_traceback != NULL ? + tstate->exc_traceback : Py_None); + } + + static PyObject * sys_exit(self, args) PyObject *self; PyObject *args; *************** *** 112,123 **** PyObject *self; PyObject *args; { if (args == Py_None) args = NULL; else Py_XINCREF(args); ! Py_XDECREF(_PySys_TraceFunc); ! _PySys_TraceFunc = args; Py_INCREF(Py_None); return Py_None; } --- 128,140 ---- PyObject *self; PyObject *args; { + PyThreadState *tstate = PyThreadState_Get(); if (args == Py_None) args = NULL; else Py_XINCREF(args); ! Py_XDECREF(tstate->sys_tracefunc); ! tstate->sys_tracefunc = args; Py_INCREF(Py_None); return Py_None; } *************** *** 127,138 **** PyObject *self; PyObject *args; { if (args == Py_None) args = NULL; else Py_XINCREF(args); ! Py_XDECREF(_PySys_ProfileFunc); ! _PySys_ProfileFunc = args; Py_INCREF(Py_None); return Py_None; } --- 144,156 ---- PyObject *self; PyObject *args; { + PyThreadState *tstate = PyThreadState_Get(); if (args == Py_None) args = NULL; else Py_XINCREF(args); ! Py_XDECREF(tstate->sys_profilefunc); ! tstate->sys_profilefunc = args; Py_INCREF(Py_None); return Py_None; } *************** *** 142,148 **** PyObject *self; PyObject *args; { ! if (!PyArg_ParseTuple(args, "i", &_PySys_CheckInterval)) return NULL; Py_INCREF(Py_None); return Py_None; --- 160,167 ---- PyObject *self; PyObject *args; { ! PyThreadState *tstate = PyThreadState_Get(); ! if (!PyArg_ParseTuple(args, "i", &tstate->sys_checkinterval)) return NULL; Py_INCREF(Py_None); return Py_None; *************** *** 202,207 **** --- 221,227 ---- static PyMethodDef sys_methods[] = { /* Might as well keep this in alphabetic order */ + {"exc_info", sys_exc_info, 0}, {"exit", sys_exit, 0}, #ifdef COUNT_ALLOCS {"getcounts", sys_getcounts, 0}, *************** *** 232,238 **** if (list == NULL) return NULL; for (i = 0; _PyImport_Inittab[i].name != NULL; i++) { ! PyObject *name = PyString_FromString(_PyImport_Inittab[i].name); if (name == NULL) break; PyList_Append(list, name); --- 252,259 ---- if (list == NULL) return NULL; for (i = 0; _PyImport_Inittab[i].name != NULL; i++) { ! PyObject *name = PyString_FromString( ! _PyImport_Inittab[i].name); if (name == NULL) break; PyList_Append(list, name); From raymond.hettinger at verizon.net Fri Dec 19 13:13:15 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sat Dec 20 14:00:11 2003 Subject: [Python-Dev] Liceinses in the Docs Message-ID: <006f01c3c65b$c61b3020$e841fea9@oemcomputer> I would like to add an section to the Library Reference to fulfill license requirements for code included in Python. For example, the license for underlying the Mersenne Twister code is already embedded in the source but it requests to be included in the docs as well. Any thoughts or objections? Raymond Hettinger -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031219/fb5e107d/attachment.html From zack at codesourcery.com Sat Dec 20 14:02:27 2003 From: zack at codesourcery.com (Zack Weinberg) Date: Sat Dec 20 14:02:34 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "20 Dec 2003 11:31:18 +0100") References: <87y8tbezcq.fsf@codesourcery.com> <87wu8sy91n.fsf@egil.codesourcery.com> Message-ID: <87r7yzuxbw.fsf@egil.codesourcery.com> martin@v.loewis.de (Martin v. L?wis) writes: > "Zack Weinberg" writes: > >> It does, however, lead to a different problem: >> >> if (condition) >> Py_DECREF (foo); >> else // oops, syntax error here > > How so? This expands to > > if(condition) > if(cond2)action1; > else action2; > else No, it expands to if(condition) if(cond2) action1; else action2;; else -- note the extra semicolon, which provokes a syntax error. zw From fdrake at acm.org Sat Dec 20 14:20:06 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sat Dec 20 14:20:15 2003 Subject: [Python-Dev] Liceinses in the Docs In-Reply-To: <006f01c3c65b$c61b3020$e841fea9@oemcomputer> References: <006f01c3c65b$c61b3020$e841fea9@oemcomputer> Message-ID: <16356.41190.525408.280943@sftp.fdrake.net> Raymond Hettinger writes: > I would like to add an section to the Library Reference to fulfill > license requirements for code included in Python. > > For example, the license for underlying the Mersenne Twister code is > already embedded in the source but it requests to be included in the > docs as well. > > Any thoughts or objections? It should be added to Doc/commontex/license.tex with an explanation of what it pertains to. Thanks! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From pf_moore at yahoo.co.uk Sat Dec 20 07:02:20 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Sat Dec 20 14:23:26 2003 Subject: [Python-Dev] Re: Relative import References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> Message-ID: Jack Diederich writes: > I like syntax that reads most important left-to-right, so what about > from MODULE import NAMES as RENAME searching HOW > or > import NAMES as RENAME from MODULE searching HOW I like this. It feels much more Pythonic than the various suggestions around magic dots in the name. It's explicit, easy to understand, and easy to extend (extra possibilities for HOW could be added with little difficulty). Paul. -- This signature intentionally left blank From tim.one at comcast.net Sat Dec 20 14:35:34 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Dec 20 14:35:42 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <87r7yzuxbw.fsf@egil.codesourcery.com> Message-ID: [Zack Weinberg] >>> It does, however, lead to a different problem: >>> >>> if (condition) >>> Py_DECREF (foo); >>> else // oops, syntax error here [Martin v. L?wis] >> How so? This expands to >> >> if(condition) >> if(cond2)action1; >> else action2; >> else [Zack] > No, it expands to > > if(condition) > if(cond2) action1; > else action2;; > else > > -- note the extra semicolon, which provokes a syntax error. Zack, which version of Python are you using? The current definition of Py_DECREF doesn't end with a semicolon: #define Py_DECREF(op) \ if (_Py_DEC_REFTOTAL _Py_REF_DEBUG_COMMA \ --(op)->ob_refcnt != 0) \ _Py_CHECK_REFCNT(op) \ else \ _Py_Dealloc((PyObject *)(op)) The expansion of _Py_CHECK_REFCNT(op) always ends with a semicolon (and may be nothing but a semicolon, depending on build type), but that's internal to the stuff Py_DECREF generates, and the leading underscore in _Py_CHECK_REFCNT's name makes it off-limits (by convention) for direct use by user-written code (_Py_CHECK_REFCNT is internal to Python's implementation, and Python internals guarantee to use it correctly). The expansion of Py_DECREF's else branch never ends with a semicolon, and Martin's explanation is correct. Same deal with Py_INCREF, Py_XINCREF, and Py_XDECREF. As a pragmatic matter, it's very unusual to need to do if (whatever) Py_DECREF(op); The functionality is common enough when "whatever" is "op != NULL", but accomplishing "decref if not NULL" is Py_XDECREF's purpose -- there's no need to code your own if-block in that case, and it's bad style (because non-idiomatic) to do so. From tim.one at comcast.net Sat Dec 20 14:41:47 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Dec 20 14:41:51 2003 Subject: [Python-Dev] Liceinses in the Docs In-Reply-To: <006f01c3c65b$c61b3020$e841fea9@oemcomputer> Message-ID: [Raymond Hettinger] > I would like to add an section to the Library Reference to fulfill > license requirements for code included in Python. > > For example, the license for underlying the Mersenne Twister code is > already embedded in the source but it requests to be included in the > docs as well. > > Any thoughts or objections? +1. Even in cases where it's not legally required, I think it's plain good manners to acknowledge external work Python incorporates, from bsddb to zlib. From zack at codesourcery.com Sat Dec 20 14:42:51 2003 From: zack at codesourcery.com (Zack Weinberg) Date: Sat Dec 20 14:42:56 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: (Tim Peters's message of "Sat, 20 Dec 2003 14:35:34 -0500") References: Message-ID: <87n09nuvgk.fsf@egil.codesourcery.com> "Tim Peters" writes: > [Zack Weinberg] >>>> It does, however, lead to a different problem: >>>> >>>> if (condition) >>>> Py_DECREF (foo); >>>> else // oops, syntax error here > > [Martin v. L?wis] >>> How so? This expands to >>> >>> if(condition) >>> if(cond2)action1; >>> else action2; >>> else > > [Zack] >> No, it expands to >> >> if(condition) >> if(cond2) action1; >> else action2;; >> else >> >> -- note the extra semicolon, which provokes a syntax error. > > Zack, which version of Python are you using? The current definition of > Py_DECREF doesn't end with a semicolon: I thought you might come back at me with that. Yes, you can avoid the syntax error by carefully leaving off the semicolon here. I don't consider this to be good style, though. It's easy to forget, and it's easy for someone to come along later and be confused and check in an 'obvious' patch that adds a semicolon - and it might not even break anything... then. I'm really not speaking about the python source code in specific. I'm speaking about general style principles, and I really do think it is good style *always* to wrap complex macros in do { } while (0) whether or not they need it. zw From tim.one at comcast.net Sat Dec 20 15:06:51 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Dec 20 15:06:56 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <16356.30838.750692.236147@montanaro.dyndns.org> Message-ID: [Skip Montanaro] > Then how about adding > > #if UCHAR_MAX != 255 > #error "Python's source code currently assumes 8-bit characters." > #endif > > right after the HAVE_LIMITS_H test? I wouldn't object. It should probably then also have #ifndef UCHAR_MAX #error ... #endif right before it, and stringobject.c's #if !defined(HAVE_LIMITS_H) && !defined(UCHAR_MAX) #define UCHAR_MAX 255 #endif should go away. From anthony at python.org Fri Dec 19 21:12:09 2003 From: anthony at python.org (Anthony Baxter) Date: Sat Dec 20 15:12:30 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (final) Message-ID: <200312200212.hBK2C9Ni011271@localhost.localdomain> On behalf of the Python development team and the Python community, I'm happy to announce the release of Python 2.3.3 (final). Python 2.3.3 is a bug-fix release of Python 2.3. A couple of serious bugs related to weakrefs and the cyclic garbage collector have been fixed, along with a number of bugs in the standard library. See the release notes on the web page for more details. For more information on Python 2.3.3, including download links for various platforms, release notes, and known issues, please see http://www.python.org/2.3.3 Highlights of this new release include: - A couple of serious bugs in the interactions of weakrefs and cyclic GC have been squashed. - At shutdown, the second call to the cyclic garbage collector has been removed. This caused more problems than it solved. - The xml.parsers.expat module now provides Expat 1.95.7. - Bug #823328: urllib2's HTTP Digest Auth support works again. - See http://www.python.org/2.3.3/NEWS.html for other bugfixes. Highlights of the previous major Python release (2.3) are available from the Python 2.3 page, at http://www.python.org/2.3/highlights.html Enjoy the new release, Anthony Anthony Baxter anthony@python.org Python 2.3.x Release Manager (on behalf of the entire python-dev team) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 226 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20031220/1b7d245a/attachment.bin From tismer at tismer.com Fri Dec 19 14:49:38 2003 From: tismer at tismer.com (Christian Tismer) Date: Sat Dec 20 15:51:56 2003 Subject: [Python-Dev] Last chance! In-Reply-To: <200312191815.hBJIFP528036@c-24-5-183-134.client.comcast.net> References: <3FDE6D35.3090100@tismer.com> <3FE2C5B5.8080208@tismer.com> <200312191531.hBJFVgw27430@c-24-5-183-134.client.comcast.net> <3FE33A1B.5030903@tismer.com> <200312191815.hBJIFP528036@c-24-5-183-134.client.comcast.net> Message-ID: <3FE35652.5000709@tismer.com> Guido van Rossum wrote: ... > Could you dig through CVS logs to find out when f_tstate was first > introduced? Maybe there's a clue about why there. Yeehaa! Quite some ride... Well, f_tstate was introduced here, in frameobject.c: ----------------------------------------------------- Revision : 2.29 Date : 1997/5/5 20:55:52 Author : 'guido' State : 'Exp' Lines : +13 -2 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. For frameobject.h, it was this version: --------------------------------------- Revision : 2.23 Date : 1997/5/5 20:55:43 Author : 'guido' State : 'Exp' Lines : +3 -1 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. Especially interesting: There is no single hint mentioning f_tstate. But, the diff shows PyObject *f_exc_type, *f_exc_value, *f_exc_traceback; PyThreadState *f_tstate; added at the same time. This was Python 1.5, I remember very well, the time where threading became the standard for Python. ceval.c was at this revision: ----------------------------- Revision : 2.118 Date : 1997/5/5 20:55:58 Author : 'guido' State : 'Exp' Lines : +263 -151 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. I fetched ceval 2.117 and 2.118 and compared them. This was exactly the time when the thread state was introduced. There were exactly two references to f_tstate: The most interesting diff for this question: PyObject * PyEval_SaveThread() { #ifdef WITH_THREAD if (interpreter_lock) { - PyObject *res; - res = (PyObject *)current_frame; - current_frame = NULL; + PyThreadState *tstate = PyThreadState_Swap(NULL); + PyObject *res = tstate ? (PyObject *) (tstate->frame) : NULL; release_lock(interpreter_lock); return res; } #endif return NULL; } void PyEval_RestoreThread(x) PyObject *x; { #ifdef WITH_THREAD if (interpreter_lock) { int err; err = errno; acquire_lock(interpreter_lock, 1); errno = err; - current_frame = (PyFrameObject *)x; + PyThreadState_Swap(x ? ((PyFrameObject *)x)->f_tstate : NULL); } #endif } PyEval_SaveThread has changed a bit meanwhile, no longer return the frame, but tstate. semantics are similar. The two functions clearly show hwo the transition from a global current_frame variable was made to the tstate mechanism. f_tstate was the source for switching the thread state. The current ceval version looks like this: void PyEval_RestoreThread(PyThreadState *tstate) { if (tstate == NULL) Py_FatalError("PyEval_RestoreThread: NULL tstate"); #ifdef WITH_THREAD if (interpreter_lock) { int err = errno; PyThread_acquire_lock(interpreter_lock, 1); errno = err; } #endif PyThreadState_Swap(tstate); } That means, tstate is not fetched from the frame, but given as a parameter. I checked all places where this function gets called, the tstate parameter never comes from a frame. As a conclusion, f_tstate appears to be an intermediate step during the move to threading. It was an easy place to keep the thread state. Its use was reduced to very few later on. call_trace introduced today's slightly false behavior exactly in that 2.118 version. It looks very much like an optimization, since there was no macro for fetching tstate at that time. This was perfectly fine, until generators were introduced. Until then, a frame was always bound to its thread. cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tim.one at comcast.net Sat Dec 20 16:07:15 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Dec 20 16:07:21 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <87n09nuvgk.fsf@egil.codesourcery.com> Message-ID: [Zack Weinberg] > I thought you might come back at me with that. Yes, you can avoid the > syntax error by carefully leaving off the semicolon here. I don't > consider this to be good style, though. It's easy to forget, and it's > easy for someone to come along later and be confused and check in an > 'obvious' patch that adds a semicolon - and it might not even break > anything... then. Zack, you have to be kidding. Martin was clearly talking about the expansion of Py_DECREF specifically, not "macros in general", as also was the poster who started this thread. > I'm really not speaking about the python source code in specific. That's now obvious . > I'm speaking about general style principles, and I really do think it is > good style *always* to wrap complex macros in do { } while (0) whether > or not they need it. Py_DECREF doesn't need it; there may be some point to it if doing so shuts up gcc nuisance complaints, and we can be sure that the extra complexity of the expanded code doesn't provoke optimization bugs in any of the C compilers used to compile Python. Writing non-loops *as* loops is a good way to tickle endcase bugs in sophisticated optimizers -- it's not without risk, and there are several thousand instances of Py_DECREF and Py_XDECREF just in the core. Also because it's used a lot, in a debug build such a change would generate a ton of useless extra instructions (at least MSVC really does generate code to create a zero, test it against zero, and jump back to the top of "the loop" if zero doesn't equal zero; I expect that most C compilers would). From zack at codesourcery.com Sat Dec 20 16:17:50 2003 From: zack at codesourcery.com (Zack Weinberg) Date: Sat Dec 20 16:17:56 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: (Tim Peters's message of "Sat, 20 Dec 2003 16:07:15 -0500") References: Message-ID: <87iskbur29.fsf@egil.codesourcery.com> "Tim Peters" writes: > [Zack Weinberg] >> I thought you might come back at me with that. Yes, you can avoid the >> syntax error by carefully leaving off the semicolon here. I don't >> consider this to be good style, though. It's easy to forget, and it's >> easy for someone to come along later and be confused and check in an >> 'obvious' patch that adds a semicolon - and it might not even break >> anything... then. > > Zack, you have to be kidding. Martin was clearly talking about the > expansion of Py_DECREF specifically, not "macros in general", as also was > the poster who started this thread. I, however, was never speaking about Py_DECREF specifically. I thought this was clear from previous statements but I apologize for the confusion. >> I'm speaking about general style principles, and I really do think it is >> good style *always* to wrap complex macros in do { } while (0) whether >> or not they need it. > > Py_DECREF doesn't need it; there may be some point to it if doing so shuts > up gcc nuisance complaints, and we can be sure that the extra complexity of > the expanded code doesn't provoke optimization bugs in any of the C > compilers used to compile Python. Writing non-loops *as* loops is a good > way to tickle endcase bugs in sophisticated optimizers -- it's not without > risk, and there are several thousand instances of Py_DECREF and Py_XDECREF > just in the core. Also because it's used a lot, in a debug build such a > change would generate a ton of useless extra instructions (at least MSVC > really does generate code to create a zero, test it against zero, and jump > back to the top of "the loop" if zero doesn't equal zero; I expect that most > C compilers would). I doubt both statements just because this is a standard C idiom. Recognizing it and generating no extra code falls naturally out of basic constant-condition detection, which I would expect to be present in any modern compiler, - and done even in "debug builds" because it can't hurt and in fact usually makes the compiler run faster (less junk intermediat code to process). I especially doubt that this construct will tickle bugs in sophisticated optimizers -- it would be harder to get it wrong than to get it right. I am not disputing that MSVC does this silly thing you describe, but MSVC has a long track record of bugs and poor code generation. GCC does not generate junk code even at -O0. zw From tim.one at comcast.net Sat Dec 20 17:13:34 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Dec 20 17:13:39 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <87iskbur29.fsf@egil.codesourcery.com> Message-ID: [Zack Weinberg] > I doubt both statements just because this is a standard C idiom. I worked in compiler development for 15 years, so I'll have to trust my experience on this. The union of MSVC and gcc accounts for the vast bulk of Python installations, but only a fraction of the total number of compilers in use. Many optimizers know nothing at all about C specifically, as they're a common back end for many front-end intermediate code generators, for many languages. It's unusual for a compiler to do any sort of optimization unless optimizations are specifically requested. I don't have any gripe with gcc doing some regardless, but gcc isn't the universe. > Recognizing it and generating no extra code falls naturally out > of basic constant-condition detection, No, changing the flow graph is a serious and delicate program transformation. A C compiler front end doing it more-than-less "mindlessly", as part of special-casing the goofy idioms C encourages, has a fine shot at never screwing up, because it's doing it then before flow graph construction. Not all compilers do it that way. > which I would expect to be present in any modern compiler, In one way or another, yes, at least optimizing compilers. Not all compilers optimize, though. A possibly important example for Python in the near future is that the "standard edition" retail .NET compilers have no optimization capability, and neither does the freely downloadable .NET C/C++ compiler. > ... > I am not disputing that MSVC does this silly thing you describe, but > MSVC has a long track record of bugs That doesn't match our experience. I don't believe Python code has ever tickled an MSVC codegen or optimization bug, neither any in gcc-for-x86. We've certainly tickled codegen and optimization bugs in gcc for other platforms, and *many* such bugs in vendor compilers (SGI's native compilers probably hold the record for sheer quantity of codegen and optimization bugs when compiling Python). At Dragon Systems we were using C++ rather than C, and I think MSVC and gcc were blamed for about an equal number of codegen and optimization bugs. > and poor code generation. This also doesn't match our experience. Release-build Python compiled under MSVC for x86 has been faster than release-build Python compiled under gcc for x86 each time I've seen them compared, and the same was true for Dragon's software. Maybe gcc is getting better on x86; I don't know; for a long time it was at a systematic disadvantage because it was originally designed for RISC-ish architectures with a large, uniform register set, and x86 is about as from that model as things get. MSVC had many years' advantage in dealing with x86's many architecural quirks. > GCC does not generate junk code even at -O0. It certainly generates needless code even at -O0. If it didn't, there would be no reason to have a higher optimization level than -O0 . From skip at pobox.com Sat Dec 20 20:07:32 2003 From: skip at pobox.com (Skip Montanaro) Date: Sat Dec 20 20:08:21 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: References: <16356.30838.750692.236147@montanaro.dyndns.org> Message-ID: <16356.62036.22726.310077@montanaro.dyndns.org> >> #if UCHAR_MAX != 255 >> #error "Python's source code currently assumes 8-bit characters." >> #endif >> >> right after the HAVE_LIMITS_H test? Tim> I wouldn't object. It should probably then also have Tim> #ifndef UCHAR_MAX Tim> #error ... Tim> #endif Isn't that supposed to always be defined in limits.h or is UCHAR_MAX not a standard macro? Tim> right before it, and stringobject.c's Tim> #if !defined(HAVE_LIMITS_H) && !defined(UCHAR_MAX) Tim> #define UCHAR_MAX 255 Tim> #endif Tim> should go away. Sounds like a plan. I modify my local source and see if it affects anything. Skip From tim.one at comcast.net Sat Dec 20 21:52:03 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Dec 20 21:52:09 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <16356.62036.22726.310077@montanaro.dyndns.org> Message-ID: [Skip Montanaro] > Isn't that supposed to always be defined in limits.h or is UCHAR_MAX > not a standard macro? Yes, it's supposed to be there. OTOH, so is limits.h, i.e. the HAVE_LIMITS_H test shouldn't be necessary either. So they're just sanity checks. But you're right, if UCHAR_MAX isn' defined, I was thinking the preprocessor would expand #if UCHAR_MAX != 255 to #if != 255 and then the error message would be incomprehensible. But unknown names in preprocessor conditionals actually get replaced by 0, and #if 0 != 255 does just as well. > Sounds like a plan. I modify my local source and see if it affects > anything. It should work fine. From anthony at interlink.com.au Sat Dec 20 22:14:03 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat Dec 20 22:14:41 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include patchlevel.h, 2.74.4.9, 2.74.4.10 In-Reply-To: Message-ID: <200312210314.hBL3E3ZU017703@localhost.localdomain> >>> fdrake@users.sourceforge.net wrote > Update of /cvsroot/python/python/dist/src/Include > In directory sc8-pr-cvs1:/tmp/cvs-serv22509 > > Modified Files: > Tag: release23-maint > patchlevel.h > Log Message: > Post-release version bumpage. Argh! Nonononononono! Please leave the branch at 2.3.3 for Jack to cut the Mac versions! Anthony -- Anthony Baxter It's never too late to have a happy childhood. From fdrake at acm.org Sun Dec 21 00:12:00 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sun Dec 21 00:12:16 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include patchlevel.h, 2.74.4.9, 2.74.4.10 In-Reply-To: <200312210314.hBL3E3ZU017703@localhost.localdomain> References: <200312210314.hBL3E3ZU017703@localhost.localdomain> Message-ID: <16357.11168.459937.182871@sftp.fdrake.net> Anthony Baxter writes: > Argh! Nonononononono! Please leave the branch at 2.3.3 for Jack to cut > the Mac versions! Oops, sorry; forgot about that. It was triggered by a change that Raymond made to the docs, properly removing material that leaked from the 2.4 tutorial. I've outdated the change to correct the version numbers. Not sure what the right thing to do about the docs is; let's just leave it alone, since it was a valuable correction. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tismer at tismer.com Sun Dec 21 03:45:30 2003 From: tismer at tismer.com (Christian Tismer) Date: Sun Dec 21 04:08:19 2003 Subject: [Python-Dev] Last chance! In-Reply-To: <200312191815.hBJIFP528036@c-24-5-183-134.client.comcast.net> References: <3FDE6D35.3090100@tismer.com> <3FE2C5B5.8080208@tismer.com> <200312191531.hBJFVgw27430@c-24-5-183-134.client.comcast.net> <3FE33A1B.5030903@tismer.com> <200312191815.hBJIFP528036@c-24-5-183-134.client.comcast.net> Message-ID: <3FE55DAA.2020808@tismer.com> Guido van Rossum wrote: ... > Could you dig through CVS logs to find out when f_tstate was first > introduced? Maybe there's a clue about why there. Yeehaa! Quite some ride... Well, f_tstate was introduced here, in frameobject.c: ----------------------------------------------------- Revision : 2.29 Date : 1997/5/5 20:55:52 Author : 'guido' State : 'Exp' Lines : +13 -2 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. For frameobject.h, it was this version: --------------------------------------- Revision : 2.23 Date : 1997/5/5 20:55:43 Author : 'guido' State : 'Exp' Lines : +3 -1 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. Especially interesting: There is no single hint mentioning f_tstate. But, the diff shows PyObject *f_exc_type, *f_exc_value, *f_exc_traceback; PyThreadState *f_tstate; added at the same time. This was Python 1.5, I remember very well, the time where threading became the standard for Python. ceval.c was at this revision: ----------------------------- Revision : 2.118 Date : 1997/5/5 20:55:58 Author : 'guido' State : 'Exp' Lines : +263 -151 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. I fetched ceval 2.117 and 2.118 and compared them. This was exactly the time when the thread state was introduced. There were exactly two references to f_tstate: The most interesting diff for this question: PyObject * PyEval_SaveThread() { #ifdef WITH_THREAD if (interpreter_lock) { - PyObject *res; - res = (PyObject *)current_frame; - current_frame = NULL; + PyThreadState *tstate = PyThreadState_Swap(NULL); + PyObject *res = tstate ? (PyObject *) (tstate->frame) : NULL; release_lock(interpreter_lock); return res; } #endif return NULL; } void PyEval_RestoreThread(x) PyObject *x; { #ifdef WITH_THREAD if (interpreter_lock) { int err; err = errno; acquire_lock(interpreter_lock, 1); errno = err; - current_frame = (PyFrameObject *)x; + PyThreadState_Swap(x ? ((PyFrameObject *)x)->f_tstate : NULL); } #endif } PyEval_SaveThread has changed a bit meanwhile, no longer return the frame, but tstate. semantics are similar. The two functions clearly show hwo the transition from a global current_frame variable was made to the tstate mechanism. f_tstate was the source for switching the thread state. The current ceval version looks like this: void PyEval_RestoreThread(PyThreadState *tstate) { if (tstate == NULL) Py_FatalError("PyEval_RestoreThread: NULL tstate"); #ifdef WITH_THREAD if (interpreter_lock) { int err = errno; PyThread_acquire_lock(interpreter_lock, 1); errno = err; } #endif PyThreadState_Swap(tstate); } That means, tstate is not fetched from the frame, but given as a parameter. I checked all places where this function gets called, the tstate parameter never comes from a frame. As a conclusion, f_tstate appears to be an intermediate step during the move to threading. It was an easy place to keep the thread state. Its use was reduced to very few later on. call_trace introduced today's slightly false behavior exactly in that 2.118 version. It looks very much like an optimization, since there was no macro for fetching tstate at that time. This was perfectly fine, until generators were introduced. Until then, a frame was always bound to its thread. cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at tismer.com Sun Dec 21 03:41:54 2003 From: tismer at tismer.com (Christian Tismer) Date: Sun Dec 21 04:08:32 2003 Subject: [Python-Dev] Last chance! In-Reply-To: <000201c3c729$c8ffcde0$e841fea9@oemcomputer> References: <000201c3c729$c8ffcde0$e841fea9@oemcomputer> Message-ID: <3FE55CD2.9040707@tismer.com> Raymond Hettinger wrote: ... > Here is a head start on the research. > > The ceval.c use of tstate goes back to the introduction of generators in > 2001. > > The use in traceback.c and sysmodule.c goes back to 1997 when > per-thread globals were factored into a structure to support separate > thread-state management. > > Prior to that, the history is more diffuse and harder to follow. Thanks a lot. I did a similar research on Friday, but for some reason it was not accepted by python.org, as it appears. (why, only 4 k, yours was 10?) I will try to send it again. My summary: f_tstate not needed! ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From martin at v.loewis.de Sun Dec 21 05:27:45 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sun Dec 21 05:27:59 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <2mwu8rebcp.fsf@starship.python.net> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> Message-ID: Michael Hudson writes: > > I'm not sure what to do about this: We apparently *want* the signals > > blocked in the thread, but we don't want them to be blocked in the > > process invoked through system(). Proposals are welcome. > > Does pthread_atfork() help? Most likely. system(3) is specified as being implemented through fork()/exec(), so an atfork handler should be invoked in any compliant implementation. We could install a child handler, which unblocks the signals we don't want to be blocked. Now, the question is: What signals precisely we don't want to be blocked? I think the answer is "All signals that have not explicitly been blocked by the application". OTOH, we already have PyOS_AfterFork, which could be used instead of pthread_atfork. Jeff, would you like to add some code there, to set all signal handlers into default for which Handlers lists that the default handling should occur? Regards, Martin From ncoghlan at iinet.net.au Sun Dec 21 08:16:58 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sun Dec 21 08:17:03 2003 Subject: [Python-Dev] Relative import In-Reply-To: <20031219213031.GD1246@wopr> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> Message-ID: <3FE59D4A.8050403@iinet.net.au> Jack Diederich wrote: > I like syntax that reads most important left-to-right, so what about > from MODULE import NAMES as RENAME searching HOW > or > import NAMES as RENAME from MODULE searching HOW This would work really well for any uses cases I can imagine having. It also has the following benefits: 1. No magic punctuation 2. People who don't need it, just never use a 'searching' clause 3. People encountering it for the first time have a clear flag letting them know something fancy is going on 4. These people have a word they can look up in the Python docs I hadn't even thought of point 4 prior to considering Jack's proposal. If I encounter "import .fred.mary" in some python I didn't write, how the heck do I find out what it means? [...] > ps, I like the second version better but it is less like the current version, > which is a drawback. To head a little further down this sidetrack, the from version of 'import' has puzzled me for a while: A. from mypkg import foo B. import mypkg.foo as foo What, if any, are the semantic differences between A & B? Is it only that A2 works, and B2 doesn't? A2. from mymodule import * B2. import mymodule.* Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From aahz at pythoncraft.com Sun Dec 21 12:30:15 2003 From: aahz at pythoncraft.com (Aahz) Date: Sun Dec 21 12:30:18 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <200312181750.hBIHof425371@c-24-5-183-134.client.comcast.net> References: <200312181750.hBIHof425371@c-24-5-183-134.client.comcast.net> Message-ID: <20031221173014.GA20633@panix.com> On Thu, Dec 18, 2003, Guido van Rossum wrote: > > OK. Wanna write a PEP? Just kidding, I know you have less time than > I do for it. So... *anybody* wanna write a PEP about this? (The > parentheses proposal should also be folded in.) Since nobody else has volunteered, I guess I'll write the PEP. Expect it sometime in the next couple of weeks. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From tim.one at comcast.net Sun Dec 21 13:29:06 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 21 13:29:11 2003 Subject: [Python-Dev] Last chance! In-Reply-To: <3FE55CD2.9040707@tismer.com> Message-ID: [Christian Tismer] > I did a similar research on Friday, but for some reason it > was not accepted by python.org, as it appears. > (why, only 4 k, yours was 10?) Your original was accepted: http://mail.python.org/pipermail/python-dev/2003-December/041287.html I think there may have been a long delay in sending it out, though, perhaps related to the python.org domain transfer, or to hardware fiddling recently done at mail.python.org's physical site. Whichever, it's in the archive, and I got it via email too. > My summary: f_tstate not needed! More, it seems conceptually flawed, albeit subtly. Python's runtime is, in effect, simulating thread-local storage by hand, and the current value of _PyThreadState_Current always points to a PyThreadState struct holding the currently-executing thread's *conceptual* thread-local storage. If we were able to use honest-to-goodness TLS instead, it wouldn't have been possible (well, not w/o ugly hacks) for a frame to use the values of these guys associated with any thread other than the one currently executing. Or, IOW, the existence of f_tstate creates a possibility for inter-thread mixups. Still, the possibility to switch threads across generator resumptions seems darned hard to view as "a feature". I'd advise people not to rely on it . From skip at pobox.com Sun Dec 21 15:03:54 2003 From: skip at pobox.com (Skip Montanaro) Date: Sun Dec 21 15:03:59 2003 Subject: [Python-Dev] make test failing in a spectacular way in test_codeccallbacks Message-ID: <16357.64682.286170.38911@montanaro.dyndns.org> Fresh cvs up (no local changes) on Mac OS X 10.2.8 yields this while running test_codeccallbacks (non-framework build, configured only using --with-pydebug): malloc[20582]: protecting edges malloc[20582]: recording stacks using standard recorder malloc[20582]: enabling scribbling to detect mods to free blocks malloc[20582]: checks heap after 1th operation and each 1000 operations malloc[20582]: environment variables that can be set for debug: - MallocGuardEdges to add 2 guard pages for each large block - MallocDoNotProtectPrelude to disable protection (when previous flag set) - MallocDoNotProtectPostlude to disable protection (when previous flag set) - MallocStackLogging to record all stacks. Tools like leaks can then be applied - MallocStackLoggingNoCompact to record all stacks. Needed for malloc_history - MallocScribble to detect writing on free blocks: 0x55 is written upon free - MallocCheckHeapStart to check the heap from time to time after operations - MallocHelp - this help! MallocCheckHeap: PASSED check at 1th operation MallocCheckHeap: PASSED check at 1001th operation MallocCheckHeap: PASSED check at 2001th operation MallocCheckHeap: PASSED check at 3001th operation test_backslashescape (__main__.CodecCallbackTest) ... ok test_badandgoodbackslashreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodignoreexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodstrictexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodxmlcharrefreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badhandlerresults (__main__.CodecCallbackTest) ... ok test_badregistercall (__main__.CodecCallbackTest) ... ok test_bug828737 (__main__.CodecCallbackTest) ... ok test_callbacks (__main__.CodecCallbackTest) ... ok test_charmapencode (__main__.CodecCallbackTest) ... ok test_decodehelper (__main__.CodecCallbackTest) ... ok MallocCheckHeap: PASSED check at 4001th operation test_encodehelper (__main__.CodecCallbackTest) ... ok Debug memory block at address p=0x7cb138: 7032 bytes originally requested The 4 pad bytes at p-4 are FORBIDDENBYTE, as expected. The 4 pad bytes at tail=0x7cccb0 are not all FORBIDDENBYTE (0xfb): at tail+0: 0x33 *** OUCH at tail+1: 0x35 *** OUCH at tail+2: 0x34 *** OUCH at tail+3: 0x3b *** OUCH The block was made by call #639840562 to debug malloc/realloc. Data at p: 00 00 00 00 00 00 00 00 ... 33 35 34 3b 26 23 31 32 Fatal Python error: bad trailing pad byte Abort trap The "Data at p" hex bytes map to ['3', '5', '4', ';', '&', '#', '1', '2']. The Malloc messages are all from Apple's malloc() pkg with a bunch of debug environment variables set. I haven't dug into this any further. I applied a new OS X security update yesterday. That's the only recent change to the system I'm aware of. GCC is [GCC 3.3 20030304 (Apple Computer, Inc. build 1493)] on darwin as reported by the Python interpreter at startup. Anyone else seen this? Skip From raymond.hettinger at verizon.net Sun Dec 21 17:15:03 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sun Dec 21 17:15:32 2003 Subject: [Python-Dev] Vinay Sajip Message-ID: <000601c3c80f$e2a5fee0$0dae2c81@oemcomputer> FYI, Vinay Sajip has been granted CVS commit privileges for the purpose of maintaining and extending his logging package. Raymond Hettinger -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031221/67ed5d5c/attachment.html From jepler at unpythonic.net Sun Dec 21 18:07:04 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Sun Dec 21 18:07:09 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> Message-ID: <20031221230655.GA4417@unpythonic.net> On Sun, Dec 21, 2003 at 11:27:45AM +0100, Martin v. L?wis wrote: > OTOH, we already have PyOS_AfterFork, which could be used instead of > pthread_atfork. Jeff, would you like to add some code there, to set > all signal handlers into default for which Handlers lists that the > default handling should occur? When using pthread_atfork, os.system never triggers my code. However, reimplementing os.system in terms of os.fork+os.execv, it does. I don't know if this is right or wrong according to pthread, but since it doesn't work on my platform the question is academic for me. Wouldn't the PyOS_AfterFork approach also require python to provide its own versions of any POSIX APIs that would typically be implemented in terms of fork (system(), popen(), and spawn*() come to mind)? Jeff Index: Python/thread_pthread.h =================================================================== RCS file: /cvsroot/python/python/dist/src/Python/thread_pthread.h,v retrieving revision 2.48 diff -u -r2.48 thread_pthread.h --- Python/thread_pthread.h 19 Nov 2003 22:52:22 -0000 2.48 +++ Python/thread_pthread.h 21 Dec 2003 23:03:52 -0000 @@ -143,6 +143,17 @@ * Initialization. */ +static void PyThread__fork_child(void) { + /* Mask all signals in the current thread before creating the new + * thread. This causes the new thread to start with all signals + * blocked. + */ + sigset_t childmask; + sigfillset(&childmask); + SET_THREAD_SIGMASK(SIG_UNBLOCK, &childmask, NULL); + fprintf(stderr, "PyThread__fork_child()\n"); fflush(stderr); +} + #ifdef _HAVE_BSDI static void _noop(void) @@ -157,6 +168,7 @@ pthread_t thread1; pthread_create(&thread1, NULL, (void *) _noop, &dummy); pthread_join(thread1, NULL); + pthread_atfork(NULL, NULL, PyThread__fork_child); } #else /* !_HAVE_BSDI */ @@ -167,6 +179,7 @@ #if defined(_AIX) && defined(__GNUC__) pthread_init(); #endif + pthread_atfork(NULL, NULL, PyThread__fork_child); } #endif /* !_HAVE_BSDI */ From guido at python.org Sun Dec 21 19:03:50 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 21 19:03:57 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Sun, 21 Dec 2003 12:30:15 EST." <20031221173014.GA20633@panix.com> References: <200312181750.hBIHof425371@c-24-5-183-134.client.comcast.net> <20031221173014.GA20633@panix.com> Message-ID: <200312220003.hBM03ph08816@c-24-5-183-134.client.comcast.net> > Since nobody else has volunteered, I guess I'll write the PEP. Expect > it sometime in the next couple of weeks. Thanks, Aahz! --Guido van Rossum (home page: http://www.python.org/~guido/) From tismer at tismer.com Sun Dec 21 19:23:51 2003 From: tismer at tismer.com (Christian Tismer) Date: Sun Dec 21 19:23:37 2003 Subject: [Python-Dev] Last chance! In-Reply-To: References: Message-ID: <3FE63997.4000600@tismer.com> Tim Peters wrote: ... > Your original was accepted: > > http://mail.python.org/pipermail/python-dev/2003-December/041287.html Yup, I wasn't aware of the DNS change. >>My summary: f_tstate not needed! > > More, it seems conceptually flawed, albeit subtly. ... Very much agreed. I always felt that f_tstate was not quite ok. You spelled it and supplied more reason. > Still, the possibility to switch threads across generator resumptions seems > darned hard to view as "a feature". I'd advise people not to rely on it > . Well, this might only work for generators if it is guaranteed that the thread switch doesn't happen while the generator is executing. I think this will give an exception. But most probably the result will be hard to predict. Other for my "tasklet" tiny threads. Their first purpose is to have concurrent small tasks without interaction (or if with interaction, then via channels and blocking). A tasklet being interrupted by a thread switch will not be auto-schedulable until the thread is resumed, but other tasklets might be, and they don't need to care about which real thread is actually in charge. This is one of my reasons why I want to get rid of f_tstate -- it does not really apply to most of my frames. Being tied to a thread is a property of the tasklet, and it does not apply all the time, only when I have to keep C state. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From jcarlson at uci.edu Sun Dec 21 19:41:45 2003 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun Dec 21 19:43:37 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: <200312220003.hBM03ph08816@c-24-5-183-134.client.comcast.net> References: <20031221173014.GA20633@panix.com> <200312220003.hBM03ph08816@c-24-5-183-134.client.comcast.net> Message-ID: <20031221163032.10E2.JCARLSON@uci.edu> Hey, I'm a new lurker (hoping to change that) here on the python-dev list, and I thought I would post a message about a new PEP I recently submitted entitled "A Case for Some". The PEP talks about how there exists None, that is false, and that is smaller than any other object ( min(None, a) -> None ). However, there doesn't exist an equivalent true value that is larger than any other number, like ( max(Some, a) -> Some ). Personally, I think the introduction of Some offers a nice compliment to None, especially because it allows algorithms that require an initialization of infinity to not require large numbers. Some would be sufficient. It also has the side benefit of being funny. I just wanted to get the discussion going. How does everyone feel about Some? Thanks, - Josiah Carlson From guido at python.org Sun Dec 21 19:49:54 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 21 19:50:02 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: Your message of "Sun, 21 Dec 2003 16:41:45 PST." <20031221163032.10E2.JCARLSON@uci.edu> References: <20031221173014.GA20633@panix.com> <200312220003.hBM03ph08816@c-24-5-183-134.client.comcast.net> <20031221163032.10E2.JCARLSON@uci.edu> Message-ID: <200312220049.hBM0nsU09360@c-24-5-183-134.client.comcast.net> > I'm a new lurker (hoping to change that) here on the python-dev list, > and I thought I would post a message about a new PEP I recently > submitted entitled "A Case for Some". > > The PEP talks about how there exists None, that is false, and that is > smaller than any other object ( min(None, a) -> None ). However, there > doesn't exist an equivalent true value that is larger than any other > number, like ( max(Some, a) -> Some ). > > Personally, I think the introduction of Some offers a nice compliment to > None, especially because it allows algorithms that require an > initialization of infinity to not require large numbers. Some would be > sufficient. It also has the side benefit of being funny. > > I just wanted to get the discussion going. How does everyone feel about > Some? -1. Depending on your algorithm you can already use sys.maxint, or a large float (e.g. 1e300), or you can write 1e1000 (or 1e300**2) which on most systems evaluates to a floating point Infinity. And "Some" sounds like a really strange spelling of "Infinity". There's also a PEP (PEP 754) to add a proper way to spell floating point Infinity. Or you could reverse signs and use min(None, -a). --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Sun Dec 21 20:07:59 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 21 20:08:03 2003 Subject: [Python-Dev] make test failing in a spectacular way intest_codeccallbacks In-Reply-To: <16357.64682.286170.38911@montanaro.dyndns.org> Message-ID: [Skip] > Fresh cvs up (no local changes) on Mac OS X 10.2.8 yields this while > running test_codeccallbacks (non-framework build, configured only > using --with-pydebug): > ... > Anyone else seen this? Just tried on Windows. test_codeccallbacks crashes here in release or debug builds. Here from a debug build: C:\Code\python\PCbuild>python_d ../lib/test/test_codeccallbacks.py test_backslashescape (__main__.CodecCallbackTest) ... ok test_badandgoodbackslashreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodignoreexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodstrictexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodxmlcharrefreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badhandlerresults (__main__.CodecCallbackTest) ... ok test_badregistercall (__main__.CodecCallbackTest) ... ok test_bug828737 (__main__.CodecCallbackTest) ... ok test_callbacks (__main__.CodecCallbackTest) ... ok test_charmapencode (__main__.CodecCallbackTest) ... ok test_decodehelper (__main__.CodecCallbackTest) ... ok test_encodehelper (__main__.CodecCallbackTest) ... ok test_longstrings (__main__.CodecCallbackTest) ... C:\Code\python\PCbuild>python_d ../lib/test/test_codeccallbacks.py test_backslashescape (__main__.CodecCallbackTest) ... ok test_badandgoodbackslashreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodignoreexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodstrictexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodxmlcharrefreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badhandlerresults (__main__.CodecCallbackTest) ... ok test_badregistercall (__main__.CodecCallbackTest) ... ok test_bug828737 (__main__.CodecCallbackTest) ... ok test_callbacks (__main__.CodecCallbackTest) ... ok test_charmapencode (__main__.CodecCallbackTest) ... ok test_decodehelper (__main__.CodecCallbackTest) ... ok test_encodehelper (__main__.CodecCallbackTest) ... ok test_longstrings (__main__.CodecCallbackTest) ... and at that point it dies with a segfault. It's really hosed at that point -- I can't get into the debugger. If I start the test under the debugger, the debugger craps out. So no easy clues from me either. Just to cheer things up , the bsddb3 tests are in much worse shape now on Win98SE than they are in 2.3.3 (about 50 more errors under current CVS). I expect that 2.3.3 will be the last Python release . BTW, when I cvs up'ed and recompiled to run the above, the only file I noticed getting recompiled was unicodeobject.c. From martin at v.loewis.de Sun Dec 21 20:08:34 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Sun Dec 21 20:09:38 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <20031221230655.GA4417@unpythonic.net> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> Message-ID: <3FE64412.3010500@v.loewis.de> Jeff Epler wrote: > When using pthread_atfork, os.system never triggers my code. However, > reimplementing os.system in terms of os.fork+os.execv, it does. I don't > know if this is right or wrong according to pthread, but since it doesn't > work on my platform the question is academic for me. Interesting. I'd be curious to find out why this fails - it may be a bug in your system, in which case I'd say "tough luck, complain to the system vendor" (for Redhat 9, I'm tempted to say that anyway ...) Looking at what likely is the source of your system(3) implementation (glibc 2.3.2, sysdeps/unix/sysv/linux/i386/system.c), I see that the fork used inside system(3) is # define FORK() \ INLINE_SYSCALL (clone, 3, CLONE_PARENT_SETTID | SIGCHLD, 0, &pid) Atleast, this is the fork being used if __ASSUME_CLONE_THREAD_FLAGS is defined, which is the case for Linux >2.5.50. With this fork() implementation, atfork handlers won't be invoked, which clearly looks like a bug to me. You might want to upgrade glibc to glibc-2.3.2-27.9.7.i686.rpm and nptl-devel-2.3.2-27.9.7.i686.rpm. In this version, the definition of FORK is changed to #if defined __ASSUME_CLONE_THREAD_FLAGS && !defined FORK # define FORK() \ INLINE_SYSCALL (clone, 3, CLONE_PARENT_SETTID | SIGCHLD, 0, &pid) #endif which might actually do the right thing, assuming FORK is already defined to one that calls the atfork handlers. > Wouldn't the PyOS_AfterFork approach also require python to provide its > own versions of any POSIX APIs that would typically be implemented in > terms of fork (system(), popen(), and spawn*() come to mind)? You are right. system(3) won't call our version of fork, so PyOS_AfterFork won't be invoked. So forget about this approach. Regards, Martin From perky at i18n.org Sun Dec 21 20:39:37 2003 From: perky at i18n.org (Hye-Shik Chang) Date: Sun Dec 21 20:39:43 2003 Subject: [Python-Dev] make test failing in a spectacular way intest_codeccallbacks In-Reply-To: References: <16357.64682.286170.38911@montanaro.dyndns.org> Message-ID: <20031222013937.GA92298@i18n.org> On Sun, Dec 21, 2003 at 08:07:59PM -0500, Tim Peters wrote: > [Skip] > > Fresh cvs up (no local changes) on Mac OS X 10.2.8 yields this while > > running test_codeccallbacks (non-framework build, configured only > > using --with-pydebug): > > ... > > Anyone else seen this? > > Just tried on Windows. test_codeccallbacks crashes here in release or debug > builds. Here from a debug build: > [snip] > and at that point it dies with a segfault. It's really hosed at that > point -- I can't get into the debugger. If I start the test under the > debugger, the debugger craps out. So no easy clues from me either. Just fixed in unicodeobject.c 2.206. I'm sorry for breaking the regrtest. I guess that I ran regrtest with built previous for checking sanity before the commit. (./python versus python) Hye-Shik From jepler at unpythonic.net Sun Dec 21 21:07:37 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Sun Dec 21 21:07:36 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <3FE64412.3010500@v.loewis.de> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> Message-ID: <20031222020728.GA15895@unpythonic.net> On Mon, Dec 22, 2003 at 02:08:34AM +0100, Martin v. Loewis wrote: > With this fork() implementation, atfork handlers won't be invoked, > which clearly looks like a bug to me. You might want to upgrade glibc > to glibc-2.3.2-27.9.7.i686.rpm and nptl-devel-2.3.2-27.9.7.i686.rpm. > In this version, the definition of FORK is changed to I also tested a fedora machine with glibc-2.3.2-101.1 nptl-devel-2.3.2-101.1 and the pthread_atfork handler is still not called for system(). Opengroup's documentation for system() says [...] The environment of the executed command shall be as if a child process were created using fork(), and the child process invoked the sh utility using execl() [...] http://www.opengroup.org/onlinepubs/007904975/functions/system.html so this looks like a glibc/nptl bug. I filed it in redhat's bugzilla: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=112517 Given all this, does it make sense to adopt a patch similar to the one I posted earlier, and ignore the bug in system() on any particular OS? system() and popen() are easy to replace in Python if anybody is really bothered by the problem. Jeff From tim.one at comcast.net Sun Dec 21 21:29:22 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 21 21:29:27 2003 Subject: [Python-Dev] make test failing in a spectacular wayintest_codeccallbacks In-Reply-To: <20031222013937.GA92298@i18n.org> Message-ID: [Hye-Shik Chang] > Just fixed in unicodeobject.c 2.206. Worked for me -- thanks! > I'm sorry for breaking the regrtest. I guess that I ran regrtest > with built previous for checking sanity before the commit. (./python > versus python) That's OK. Don't make the same mistake again, and even Guido will forgive you . From tjreedy at udel.edu Sun Dec 21 21:36:44 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Sun Dec 21 21:36:46 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> Message-ID: "Luke Kenneth Casson Leighton" wrote in message news:20031220121942.GF1933@lkcl.net... > if you "feed" access to __class__ via a single function, you have > a bottleneck again which can be replaced. > > so, the proposal is simple: create an __builtin__ function called > __get_class__ through which access to __class__ is made. > I wonder if it would not be better to turn __class__ into a property with a get function that conditionally gives out the real thing. TJR From tim.one at comcast.net Sun Dec 21 21:52:48 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 21 21:52:53 2003 Subject: [Python-Dev] Last chance! In-Reply-To: <3FE63997.4000600@tismer.com> Message-ID: [Tim] >> Still, the possibility to switch threads across generator >> resumptions seems darned hard to view as "a feature". I'd advise >> people not to rely on it . [Christian Tismer] > Well, this might only work for generators if it is guaranteed > that the thread switch doesn't happen while the generator > is executing. I think this will give an exception. But most > probably the result will be hard to predict. I'm not sure I'm following, but nothing in Python now *stops* you from resuming a suspended generator from another thread. Here's an extreme example of that: """ import threading def myid(): while True: yield threading.currentThread() next = myid().next mut = threading.Lock() class Worker(threading.Thread): def run(self): while True: mut.acquire() print next() mut.release() for i in range(3): Worker().start() """ Then the body of the single generator is executed by 3 different threads, more or less randomly, although never by the thread that created the generator. If the body of the generator were relying on thread-local state (like, e.g., the mass of settings in the decimal arithmetic package's per-thread default "numeric context" object), the result would sure be unpredictable. It wouldn't crash Python, it would just be a bitch to debug. Then again, lots of bad threading practices lead to bitch-to-debug code, so I'm not really inclined to offer extra protection here. > Other for my "tasklet" tiny threads. Their first purpose is > to have concurrent small tasks without interaction (or if > with interaction, then via channels and blocking). A tasklet > being interrupted by a thread switch will not be auto-schedulable > until the thread is resumed, but other tasklets might be, and > they don't need to care about which real thread is actually in > charge. This is one of my reasons why I want to get rid of > f_tstate -- it does not really apply to most of my frames. > Being tied to a thread is a property of the tasklet, and it > does not apply all the time, only when I have to keep C state. If a tasklet can be affected by thread-local state (like numeric context -- Python has little of this nature now, but it's bound to increase), then a tasklet will also start to care "who's in charge". So I expect a patch to reintroduce f_tstate in about a year . From martin at v.loewis.de Sun Dec 21 21:46:14 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Sun Dec 21 22:52:44 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <20031222020728.GA15895@unpythonic.net> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> Message-ID: <3FE65AF6.6010403@v.loewis.de> Jeff Epler wrote: > Given all this, does it make sense to adopt a patch similar to the one > I posted earlier, and ignore the bug in system() on any particular OS? > system() and popen() are easy to replace in Python if anybody is really > bothered by the problem. I'd think so, yes. The patch needs to be elaborated, and I'd delay integration to wait for a Redhat response. We should also restrict this for 2.4 at the moment, to find out how other pthread implementations react to pthread_atfork. Regards, Martin From guido at python.org Sun Dec 21 23:48:01 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 21 23:48:09 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Fri, 19 Dec 2003 16:30:31 EST." <20031219213031.GD1246@wopr> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> Message-ID: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> > I like syntax that reads most important left-to-right, so what about > from MODULE import NAMES as RENAME searching HOW > or > import NAMES as RENAME from MODULE searching HOW > > searching could be 'absolute' 'relative' or any of the other > suggested words. If you wanted to get fancy it could be a list, if > it isn't a list people who truly care could cascade try/except on > ImportError. > > -jackdied > > ps, I like the second version better but it is less like the current version, > which is a drawback. This got several encouraging responses, so I have to consider it. Unfortunately it looks like 'searching' would have to become a reserved word; if we use the same trick we used to avoid making 'as' a reserved word, that would be too confusing to decode in the code generator. The HOW could be just an identifier then. I'm not sure how well it'll read in actual examples. Let's try. Adapting an example from Opal that was used here before... from COGS import generate searching relative To me, that looks a lot like someone left the commas out of from COGS import generate, searching, relative In general I'm not a big fan of "wordy" clauses like this -- they work better in languages like SQL or COBOL because those are case-sensitive so you can use a difference in capitalization there; there the former would be FROM COGS IMPORT generate SEARCHING RELATIVE (using imaginary syntax) while the latter would be FROM COGS IMPORT generate, searching, relative while in Python the long sequence of lowercase words becomes a blur. So if the two alternatives are simply 'searching absolute' and 'searching relative', with the default being 'searching absolute' in Python 3.0 (but the current ambiguous search in Python 2.x) I'd still prefer making the distinction with "from ...MODULE" vs. "from MODULE" rather than with a searching clause. And I still prefer the single-dot syntax over three dots, because it can be used to spell importing from the parent or grandparent package explicitly. A separate suggestion is to switch from "from X import Y" to "import Y from X". This seems a gratuitous change even if it reads or edits marginally better. Now's not the time to tinker with marginal stuff -- that should've been done 10 years ago. --Guido van Rossum (home page: http://www.python.org/~guido/) From jack at performancedrivers.com Mon Dec 22 02:52:19 2003 From: jack at performancedrivers.com (Jack Diederich) Date: Mon Dec 22 02:52:24 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> Message-ID: <20031222075219.GA3426@performancedrivers.com> On Sun, Dec 21, 2003 at 08:48:01PM -0800, Guido van Rossum wrote: > > I like syntax that reads most important left-to-right, so what about > > from MODULE import NAMES as RENAME searching HOW > > or > > import NAMES as RENAME from MODULE searching HOW > > > > searching could be 'absolute' 'relative' or any of the other > > suggested words. If you wanted to get fancy it could be a list, if > > it isn't a list people who truly care could cascade try/except on > > ImportError. > > Unfortunately it looks like 'searching' would have to become a > reserved word; if we use the same trick we used to avoid making 'as' a > reserved word, that would be too confusing to decode in the code > generator. The HOW could be just an identifier then. Too bad the decorator PEP (PEP 318) seems to favor 'as' in the syntax, import foo [relative] is more explicit than the dot syntax and has a nice visual seperator while avoiding the wordiness problem of 'searching' > I'm not sure how well it'll read in actual examples. Let's try. > Adapting an example from Opal that was used here before... > > from COGS import generate searching relative > > To me, that looks a lot like someone left the commas out of > > from COGS import generate, searching, relative moving the 'import' to always be first would alleviate this a bit because 'searching' would always follow a non-list. import generate from COGS searching relative or if we decorate import generate from COGS [relative] from COGS import generate [relative] > In general I'm not a big fan of "wordy" clauses like this -- they work > better in languages like SQL or COBOL because those are case-sensitive > so you can use a difference in capitalization there [snip] > while in Python the long sequence of lowercase words becomes a blur. as above, moving import to be first helps but doesn't eliminate the wordiness problem[*]. 'as' and 'from' are short enough to be easilly seen as keywords, 'find' or 'look' reads less like english but would be easier to pick out visually than 'searching' > So if the two alternatives are simply 'searching absolute' and > 'searching relative' Did someone mention 'searching scan' in dusty corner somewhere? One drawback to the dot syntax is if there can be more than two ways to do the search. Adding a third way is easier w/ the searching clause instead of adding a 'bang' syntax to the 'dot' syntax . The decorator would do one better by being a placeholder for anything that modifies the import. > A separate suggestion is to switch from "from X import Y" to "import Y > from X". This seems a gratuitous change even if it reads or edits > marginally better. Now's not the time to tinker with marginal stuff > -- that should've been done 10 years ago. Damn, I really liked that part. I don't get tripped up on from X import Y anymore, but I did when I started w/ python. benign-user-for-life-ly, -jackdied [*] I use an emacs hack to display 'lambda' as 'L' to avoid the same keyword-reads-like-a-variable effect. From jcarlson at uci.edu Mon Dec 22 03:13:14 2003 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon Dec 22 03:15:46 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: <20031222075219.GA3426@performancedrivers.com> References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> Message-ID: <20031222000819.10EC.JCARLSON@uci.edu> I just realized I've been forgetting to CC the list. It is a bit long, but I'll cut out the non-relevant portions that everyone has seen before. In chronological order below. I ask the rest of you, does Some sound so crazy? - Josiah *** message from Josiah Carlson on Sun, 21 Dec 2003 19:05:45 -0800 *** > -1. Depending on your algorithm you can already use sys.maxint, or a > large float (e.g. 1e300), or you can write 1e1000 (or 1e300**2) which > on most systems evaluates to a floating point Infinity. And "Some" > sounds like a really strange spelling of "Infinity". There's also a > PEP (PEP 754) to add a proper way to spell floating point Infinity. > Or you could reverse signs and use min(None, -a). > > --Guido van Rossum (home page: http://www.python.org/~guido/) I've read PEP 754, and yes, the author talks about having a proper Infinity, -Infinity and Not a Number (PosInf, NegInf and NaN respectively), but considering that it has been in the 'draft' state since March 28, has not been updated, and it doesn't seem like Python is any closer to having proper definitions now than it was ~9 months ago, I'm not going to put my eggs in the IEEE 754 Infinity basket. In terms of having something useful on a platform, there is only so useful that infinity can be. The below is from the windows verstion of Python 2.3.2 >>> 1e309 1.#INF >>> -1e309 -1.#INF >>> 1.#INF 1.0 >>> max(1e309, 0) 1.#INF >>> min(1e309, 10**309) Traceback (most recent call last): File "", line 1, in ? OverflowError: long int too large to convert to float The real trick is that it is not so much about being able to have max(Some, a) -> Some, it is about being able to have min(Some, a) -> a. Then you don't need to worry about sys.maxint not being big enough (because there are larger numbers), nor do you need to worry about float casts of large integers producing OverflowErrors. As for an application, many path searches in graphs start out with initializing distances to infinity, Some would be suitable for this. Other algorithms require a negative infinity for this, None has been working fine. As an initialization value, I believe that Some would be quite a useful object. Certainly Some seems like a strange way to spell infinity, but it is a logical infinity, one that you can compare things against, anything, not just Python floats or things that can be cast into Python floats. In terms if PEP 754, if it were actually implemented, certainly there would be some question as to whether Some would be equal to PosInf. If so, then would -Some be equal to NegInf? And if so, would None be equal to NegInf? Maybe such things would require a float cast. I talk about all this in my PEP submission, which isn't up yet. I suppose I probably should have given it a few days before discussing it, but yeah. - Josiah *** end message from Josiah *** *** message from Guido van Rossum on Sun, 21 Dec 2003 20:45:34 -0800 *** > I've read PEP 754, and yes, the author talks about having a proper > Infinity, -Infinity and Not a Number (PosInf, NegInf and NaN > respectively), but considering that it has been in the 'draft' state > since March 28, has not been updated, and it doesn't seem like Python is > any closer to having proper definitions now than it was ~9 months ago, > I'm not going to put my eggs in the IEEE 754 Infinity basket. I expect your PEP will fare no better, and given its weirdness, probably much worse. Do you know of any language that has a similar concept? --Guido van Rossum (home page: http://www.python.org/~guido/) *** end message from Guido *** *** message from Josiah Carlson on Sun, 21 Dec 2003 21:47:34 -0800 *** > I expect your PEP will fare no better, and given its weirdness, > probably much worse. Do you know of any language that has a similar > concept? Is None so weird? All I am proposing is a symmetric value to None. Just like there is a -1 and 1, -Infinity and Infinity, None logically should have Some. Maybe not so much -Some == None, but as I propose in the PEP, possibly Some == not None. - Josiah *** end message from Josiah *** From lists at hlabs.spb.ru Mon Dec 22 06:45:14 2003 From: lists at hlabs.spb.ru (Dmitry Vasiliev) Date: Mon Dec 22 03:45:11 2003 Subject: [Python-Dev] Importing packages from command line In-Reply-To: <1071839460.3fe2f8e4dbbad@mcherm.com> References: <1071839460.3fe2f8e4dbbad@mcherm.com> Message-ID: <3FE6D94A.2060904@hlabs.spb.ru> Michael Chermside wrote: >>Just an idea... >> >>1. python -p package_name >> >> Equivalent to: >> >> import package_name > > (1) Sourceforge is a great place to request feature enhancements. > Suggestions made on this mailing list is likely to be forgotten > sooner or later. Yes, I know, but the request is not formed yet. :-) > > (2) Can you explain WHY you would want this feature? Is there some > use case? I would prefer NOT to have this, because right now > if I'm reading code and it uses "package_name.someFunction()" > I can scan upward for "package_name" to find out what it's > using. With command line imports, I couldn't do that. So unless > you've got a good reason for it, I personally wouldn't want > this feature. Sorry for description absentees. The main idea is to treating package as a program and run package initialization code from command line. The advantage is zipping all program modules in one zip archive and running the program from command line without unzipping it, like Java's jar's. But this idea need more thoughts however... -- Dmitry Vasiliev (dima at hlabs.spb.ru) From troy at gci.net Mon Dec 22 04:03:08 2003 From: troy at gci.net (Troy Melhase) Date: Mon Dec 22 04:03:19 2003 Subject: [Python-Dev] Relative import In-Reply-To: <20031222075219.GA3426@performancedrivers.com> References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> Message-ID: <200312220003.08810.troy@gci.net> On Sunday 21 December 2003 10:52 pm, Jack Diederich wrote: > On Sun, Dec 21, 2003 at 08:48:01PM -0800, Guido van Rossum wrote: > > > I like syntax that reads most important left-to-right, so what about > > > from MODULE import NAMES as RENAME searching HOW > > > import NAMES as RENAME from MODULE searching HOW I like Jack's original idea and I've been chewing on it for a few days. His was: import NAMES as RENAME from MODULE searching HOW My humble suggestion: import NAMES import NAMES in MODULE [in HOW] import NAME, OTHER in FOO in RELATIVE import NAME as RENAME in MODULE in ABSOLUTE Playing with it: import sys import path, stat in os import AClass, AValue in AModule in __search__ import AType in Package.Library in __absolute__ It emphasizes the name that's imported and it reduces+reuses a keyword. Reusing "in" is a stretch, of course, and it reading it twice in the same statement might be confusing for some. I don't really like the dot/triple dot notation... the leading punctuation smells perl-ish. :) Only-a-random-two-cents-ly yours, -- Troy Melhase, troy@gci.net -- Merry Christmas! From tismer at tismer.com Mon Dec 22 05:36:07 2003 From: tismer at tismer.com (Christian Tismer) Date: Mon Dec 22 05:34:34 2003 Subject: [Python-Dev] Last chance! In-Reply-To: References: Message-ID: <3FE6C917.6080107@tismer.com> Tim Peters wrote: > [Tim] > >>>Still, the possibility to switch threads across generator >>>resumptions seems darned hard to view as "a feature". I'd advise >>>people not to rely on it . > > > [Christian Tismer] > >>Well, this might only work for generators if it is guaranteed >>that the thread switch doesn't happen while the generator >>is executing. I think this will give an exception. But most >>probably the result will be hard to predict. Explaining that: I was talking about freely using the generator in another thread, which would most likely resume it while it is active, and that's an exception. > I'm not sure I'm following, but nothing in Python now *stops* you from > resuming a suspended generator from another thread. Here's an extreme > example of that: [XTreme example] > Then the body of the single generator is executed by 3 different threads, > more or less randomly, although never by the thread that created the > generator. > > If the body of the generator were relying on thread-local state (like, e.g., > the mass of settings in the decimal arithmetic package's per-thread default > "numeric context" object), the result would sure be unpredictable. It > wouldn't crash Python, it would just be a bitch to debug. Then again, lots > of bad threading practices lead to bitch-to-debug code, so I'm not really > inclined to offer extra protection here. I agree, sure, cross-thread generators need to be designed for that. [tasklets] > If a tasklet can be affected by thread-local state (like numeric context -- > Python has little of this nature now, but it's bound to increase), then a > tasklet will also start to care "who's in charge". So I expect a patch to > reintroduce f_tstate in about a year . It is already there, not in the frames, but in the tasklet. A tasklet can be thread-bound or not. In that case, it keeps a reference to the thread state. But only once, since that information is not local for frames. Being thead bound is either enforced in the case that the tasklet is carrying non-trivial C state, otherwise it can be set as a user option (conservative default, of course). "Nomad threads" might be nice for some background monitoring. This is all theory, I haven't tried any real threads with Stackless so far. But people are mixing tasklets with threads already, so I have to care *somehow*. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From ncoghlan at iinet.net.au Mon Dec 22 05:50:01 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Mon Dec 22 05:50:05 2003 Subject: [Python-Dev] Importing packages from command line In-Reply-To: <3FE6D94A.2060904@hlabs.spb.ru> References: <1071839460.3fe2f8e4dbbad@mcherm.com> <3FE6D94A.2060904@hlabs.spb.ru> Message-ID: <3FE6CC59.10100@iinet.net.au> Dmitry Vasiliev wrote: > The main idea is to treating package as a program and run package > initialization code from command line. The advantage is zipping all > program modules in one zip archive and running the program from command > line without unzipping it, like Java's jar's. But this idea need more > thoughts however... > python runzip.py mypkg.zip where runzip.py is a pure Python script that works as you suggest. I'm fairly sure zipimport would allow you to write the program you describe in pure Python. I've obviously never written such a program, though. It's certainly an interesting packaging idea for small Python programs with several files, but for which a full-blown installer would be overkill. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From paul-python at svensson.org Mon Dec 22 06:48:30 2003 From: paul-python at svensson.org (Paul Svensson) Date: Mon Dec 22 06:48:38 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> Message-ID: <20031222060114.K47423@familjen.svensson.org> Summarizing what I've seen so far: Status quo: [from NAME] import NAME [, NAME] [from NAME] import NAME as NAME There also seems to be consensus on adding: [from NAME] import (NAME [, NAME]) To this, we want to as a way to spell out explicitly how the module is found; for example: -a- by searching sys.path -b- in the current package -c- in the current package or its parent package, recursively -d- among the python standard library modules (and nowhere else) -e- in the current working directory -f- in a specific directory -g- combinations of the above Jack Diederich proposed: >> from MODULE import NAMES as RENAME searching HOW >> or >> import NAMES as RENAME from MODULE searching HOW Guido van Rossum oppined: > from COGS import generate searching relative > >To me, that looks a lot like someone left the commas out of > > from COGS import generate, searching, relative (---) >A separate suggestion is to switch from "from X import Y" to "import Y >from X". This seems a gratuitous change even if it reads or edits >marginally better. Now's not the time to tinker with marginal stuff >-- that should've been done 10 years ago. Not only a gratuitous change; it was always my impression that "from" was at the beginning of the statement for exactly the reason Guido is hesitant about adding "searching" at the end of it. With this in mind, my suggestion is to add the searching syntax should before the "import" keyword rather than at the end of the statement. Reusing an existing keyword, we get: [from NAMES] [in WHERE] import ... WHERE would then be an object, or a list of objects (-g-) specifying the import mechanism: -d- __stdlib__ or maybe stdlib -b- __package__ -c- __search__ -e- '.' -f- '/home/bellman/python' -a- sys.path Without any "from" clause, "in WHERE import ..." reads a little strangely, but I think it should be possible to get used to it, just as we've gotten used to the current "from NAME import ..." Would this work ? /Paul From skip at pobox.com Mon Dec 22 08:08:04 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 22 08:08:34 2003 Subject: [Python-Dev] make test failing in a spectacular wayintest_codeccallbacks In-Reply-To: References: <20031222013937.GA92298@i18n.org> Message-ID: <16358.60596.317938.185310@montanaro.dyndns.org> Tim> [Hye-Shik Chang] >> Just fixed in unicodeobject.c 2.206. Tim> Worked for me -- thanks! Me also... Skip From mcherm at mcherm.com Mon Dec 22 08:33:45 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Mon Dec 22 08:33:47 2003 Subject: [Python-Dev] Got None. Maybe Some? Message-ID: <1072100025.3fe6f2b90ea70@mcherm.com> [Josiah proposes a Some object to be the counterpart of None] [Guido seems reluctant] Josiah: I have a counterproposal. The good news is, it's REALLY easy -- you won't need a PEP, and I am sure that you'll get no objections from anyone on this list. And it'll provide you with a "Some" object that acts like you describe (except the one mention you made of "Some == not None"... that one's a bad idea since right now "True == not None"). Here's the idea: Create a file called "Some.py": -------- Some.py ------------ class _SomeType(object): def __lt__(self, other): return False # never less than anything def __le__(self, other): if other is Some: return True # equal to itself else: return False # less than anything else def __gt__(self, other): if other is Some: return False # equal to itself else: return True # greater than anything else def __ge__(self, other): return True # greater than or equal to anything def __repr__(self): return 'Some' Some = _SomeType() # singleton instance ---------- end -------------- Place this in your python path, then add this line to the top of your scripts: "from Some import Some". If you really want to, you could put code in your site.py file so that you don't even need this import line, but I think in the long run you'll prefer using the import line. I think that this meets your expectations (if I understood them properly), and if not, it can probably be adjusted so that it does. The one thing it doesn't do is that it isn't part of the Standard Library. But that doesn't mean it can't be part of YOUR standard library... something you use all the time all over the place. I think that most people on this list (myself included) would say that the uses of this object are sufficiently esoteric and the naming of it is sufficiently quirky that they wouldn't want it in the language itself, but *everyone* here would stand up for your right to create and use such an object yourself... something which would never be possible in (for instance) Java. Once-we've-got-Some-perhaps-we-should-consider-a-builtin-Milk lly yours, -- Michael Chermside From jepler at unpythonic.net Mon Dec 22 09:03:17 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Mon Dec 22 09:03:21 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <3FE65AF6.6010403@v.loewis.de> References: <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> Message-ID: <20031222140255.GH12586@unpythonic.net> On Mon, Dec 22, 2003 at 03:46:14AM +0100, Martin v. Loewis wrote: > I'd think so, yes. The patch needs to be elaborated, and I'd delay > integration to wait for a Redhat response. Well, unlike the last two bugs I filed in redhat bugzilla, this one has already seen some activity. I hope they don't take the line that this is not a bug :( ------- Additional Comments From jakub@redhat.com 2003-12-22 03:53 ------- Why? There is no word about system() in http://www.opengroup.org/onlinepubs/007904975/functions/pthread_atfork.html and there doesn't seem to be anything in http://www.opengroup.org/onlinepubs/007904975/functions/system.html that would mandate system() to be implemented using fork() function. ------- Additional Comments From jepler@unpythonic.net 2003-12-22 08:01 ------- I base my claim that pthread_atfork() should be called by system() from reading http://www.opengroup.org/onlinepubs/007904975/functions/system.html The third paragraph under the heading "description" says this: "[CX] The environment of the executed command shall be as if a child process were created using fork(), and the child process invoked the sh utility using execl() as follows [...]" The rationale section expands on what the "environment of the executed command" is: "IEEE Std 1003.1-2001 places additional restrictions on system(). It requires that if there is a command language interpreter, the environment must be as specified by fork() and exec. This ensures, for example, that close-on- exec works, that file locks are not inherited, and that the process ID is different." The pthread_atfork web page says it "provides multi-threaded application programs with a standard mechanism for protecting themselves from fork() calls in a library routine or the application itself." I believe that system() is one such library routine, since it is specified to create an environment "as specified by fork()". This issue arose in the context of Python. For various reasons, signals are blocked in all threads besides the main thread. This means that in processes generated by fork() from subthreads, all signals are blocked, which leads to undesired behavior. Using the child argument of pthread_atfork() allows signals to be unblocked when using fork()+exec() to execute an external program, but not when system() is used. From barry at python.org Mon Dec 22 09:55:16 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 22 09:55:42 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312220003.08810.troy@gci.net> References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> Message-ID: <1072104915.18828.47.camel@anthem> On Mon, 2003-12-22 at 04:03, Troy Melhase wrote: > I don't really like the dot/triple dot notation... the leading punctuation > smells perl-ish. :) I don't either, although my biggest problem with it is that it isn't clear just by looking what the punctuation actually means. I just fear this will be a permanent head scratcher. Here's my entry in the sure-to-be-shot-down-by-the-BDFL-alternative -syntax game: import foo by absolute import foo.bar by relative as baz import foo by searching as bar import foo by absolute from bar import foo as bar by absolute from baz IOW, 'by' is a reserved word in the import statement context, and it specifies how to search for modules. The question then is whether 'absolute', 'relative', and 'searching' are also reserved words or built-ins and how those names are equated with the module location algorithms. -Barry From guido at python.org Mon Dec 22 10:24:30 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 10:27:43 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: Your message of "Sun, 21 Dec 2003 21:47:34 PST." <20031221212332.10E9.JCARLSON@uci.edu> References: <20031221183714.10E6.JCARLSON@uci.edu> <200312220445.hBM4jYS17286@c-24-5-183-134.client.comcast.net> <20031221212332.10E9.JCARLSON@uci.edu> Message-ID: <200312221527.hBMFRYt17987@c-24-5-183-134.client.comcast.net> > Is None so weird? All I am proposing is a symmetric value to None. > Just like there is a -1 and 1, -Infinity and Infinity, None logically > should have Some. Maybe not so much -Some == None, but as I propose in > the PEP, possibly Some == not None. None's purposr is not to be less than all numbers (in fact in the past it wasn't); it's to be "not any object", like nil or NULL in other languages. I don't see a similar broad role for Some. And I repeat my question: which other language has that concept? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 22 10:36:09 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 10:36:14 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Mon, 22 Dec 2003 09:55:16 EST." <1072104915.18828.47.camel@anthem> References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> <1072104915.18828.47.camel@anthem> Message-ID: <200312221536.hBMFa9S18026@c-24-5-183-134.client.comcast.net> > I don't either, although my biggest problem with it is that it isn't > clear just by looking what the punctuation actually means. I just > fear this will be a permanent head scratcher. This argument by itself has very little value. The question isn't whether it's self-describing (very few features beyond decimal integers are); the question is whether it's easy to remember. There are two subclasses of that: easy to remember what it means when you next see it, and easy to remember how to spell it when you next need it. IMO the dot syntax does fine here. > Here's my entry in the sure-to-be-shot-down-by-the-BDFL-alternative > -syntax game: There have been so many alternatives proposed this morning that I have to spend some time comparing them all... --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Mon Dec 22 10:44:46 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 22 10:44:55 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312221536.hBMFa9S18026@c-24-5-183-134.client.comcast.net> References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> <1072104915.18828.47.camel@anthem> <200312221536.hBMFa9S18026@c-24-5-183-134.client.comcast.net> Message-ID: <1072107885.18828.76.camel@anthem> On Mon, 2003-12-22 at 10:36, Guido van Rossum wrote: > > I don't either, although my biggest problem with it is that it isn't > > clear just by looking what the punctuation actually means. I just > > fear this will be a permanent head scratcher. > > This argument by itself has very little value. The question isn't > whether it's self-describing (very few features beyond decimal > integers are); the question is whether it's easy to remember. There > are two subclasses of that: easy to remember what it means when you > next see it, and easy to remember how to spell it when you next need > it. IMO the dot syntax does fine here. Sorry, it was a pre-coffee argument. To me, the dot-syntax doesn't really tell me whether it forces a relative search or an absolute search. The dot tells me that something different is happening, so maybe when absolute is the default, it tells me there's a switch to relative searching going on. I'm not sure though, so I do think it may potentially fail the "easy to remember what it means when you next see it" test. OTOH, it would be nice to see a Greg Wilson-style newbie poll. > > Here's my entry in the sure-to-be-shot-down-by-the-BDFL-alternative > > -syntax game: > > There have been so many alternatives proposed this morning that I have > to spend some time comparing them all... Hopefully, this stuff will make it into the PEP so you don't have to troll through suggestions buried in endless email threads. -Barry From guido at python.org Mon Dec 22 11:29:31 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 11:29:35 2003 Subject: [Python-Dev] OpenBSD anyone? Message-ID: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> John Draper (the famous Capt'n Crunch) reports that on his OpenBSD box, he can't build a debug version of Python 2.3(.3c1, but I doubt that matters) unless he tweaks the configure file to add -O3 to the C flags. That sounds fishy to me, but he insists that's how it is, and he's reproduced it from a clean tarball. Debugging it further doesn't really seem to be within John's power; the -O3 trick was found by his partner Steve who's temporarily unavailable. The symptom is that without -O3, a "python" binary gets built but its use to run setup.py segfaults. Is there anyone here with access to an OpenBSD box who could try to reproduce this? --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Mon Dec 22 11:46:01 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 22 11:46:04 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: <200312221527.hBMFRYt17987@c-24-5-183-134.client.comcast.net> Message-ID: [Guido] > ... > And I repeat my question: which other language has that concept? Variations on None were debated pretty exhaustively during Python's first year (yes, that was a while ago ). Some obscure functional languages have "top" and "bottom" values. There's a partial ordering among values, and "top" and "bottom" are added to make a lattice out of that. "bottom" is usually identified as "the error value", and f(..., bottom, ...) is defined to return bottom for all functions f then. That's not really like None, it's more like 754's NaN (although NaN**0.0 in C99 is defined to return 1, not NaN, under the theory that it doesn't matter if an error occurred during the computation of the base, because base**0 is 1 no matter what base should have been). Aaron Watters also identified several distinct detailed semantics for "missing value" markers in assorted database languages, which are closer to what Python's None usually intends ("I'm not really here, but *something* has to be here so I'm it"). If all that's really wanted is an object that compares larger than anything else, it's easy to define such a beast (and I see Michael Chermside posted code for that already). If more than that is wanted, then it becomes more a collection of arbitrary behaviors, and then different apps will want different arbitrary behaviors. None is handy and I like it, but it's arbitrary enough in some cases that it takes time to get used to its quirks (e.g., why is a value that's "not really there" considered to be false? why not? sometimes that's what you want; sometimes you'd rather get an exception; None picks an arbitrary answer and sticks to it), so I don't want see more things like it proliferate (one is enough -- but not too many ). From skip at pobox.com Mon Dec 22 11:50:24 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 22 11:50:52 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: References: <16356.62036.22726.310077@montanaro.dyndns.org> Message-ID: <16359.8400.767266.692477@montanaro.dyndns.org> >> Sounds like a plan. I modify my local source and see if it affects >> anything. Tim> It should work fine. Done. UCHAR_MAX is now required in Python.h, and it's required to be 255. That may open up a couple (small) optimization opportunities (places where gcc 3.3 says explicit tests against char values will always be true or false). I'll leave it for others to make those tweaks if they so desire. Note that some of those gcc messages are probably related to larger types (e.g. unsigned short) or depend on the presence or absence of other macro definitions (e.g. Py_UNICODE_WIDE). Should you wish to code any of these speedups, make sure you get the cpp tests right. Guido won't forgive you if you don't. Skip From blunck at gst.com Mon Dec 22 12:02:57 2003 From: blunck at gst.com (Christopher Blunck) Date: Mon Dec 22 12:03:01 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> Message-ID: <20031222170257.GA31352@homer.gst.com> On Mon, Dec 22, 2003 at 08:29:31AM -0800, Guido van Rossum wrote: > John Draper (the famous Capt'n Crunch) reports > that on his OpenBSD box, he can't build a debug version of Python > 2.3(.3c1, but I doubt that matters) unless he tweaks the configure > file to add -O3 to the C flags. That sounds fishy to me, but he > insists that's how it is, and he's reproduced it from a clean tarball. > Debugging it further doesn't really seem to be within John's power; > the -O3 trick was found by his partner Steve who's temporarily > unavailable. > > The symptom is that without -O3, a "python" binary gets built but its > use to run setup.py segfaults. Is there anyone here with access to an > OpenBSD box who could try to reproduce this? OpenBSD 3.3 builds both HEAD and r233c1 fine for me. This is code out of the anoncvs repository btw. -c -- 12:00:00 up 21 days, 1:44, 12 users, load average: 0.01, 0.07, 0.09 From aahz at pythoncraft.com Mon Dec 22 12:07:02 2003 From: aahz at pythoncraft.com (Aahz) Date: Mon Dec 22 12:07:06 2003 Subject: [Python-Dev] Relative import In-Reply-To: <1072107885.18828.76.camel@anthem> References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> <1072104915.18828.47.camel@anthem> <200312221536.hBMFa9S18026@c-24-5-183-134.client.comcast.net> <1072107885.18828.76.camel@anthem> Message-ID: <20031222170702.GA11581@panix.com> On Mon, Dec 22, 2003, Barry Warsaw wrote: > On Mon, 2003-12-22 at 10:36, Guido van Rossum wrote: >>> >>> Here's my entry in the sure-to-be-shot-down-by-the-BDFL-alternative >>> -syntax game: >> >> There have been so many alternatives proposed this morning that I have >> to spend some time comparing them all... > > Hopefully, this stuff will make it into the PEP so you don't have to > troll through suggestions buried in endless email threads. That's the plan; I expect that people will review the PEP to make sure their pet proposal is listed. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From aleaxit at yahoo.com Mon Dec 22 12:07:14 2003 From: aleaxit at yahoo.com (Alex Martelli) Date: Mon Dec 22 12:07:20 2003 Subject: [Python-Dev] Importing packages from command line In-Reply-To: <3FE6D94A.2060904@hlabs.spb.ru> References: <1071839460.3fe2f8e4dbbad@mcherm.com> <3FE6D94A.2060904@hlabs.spb.ru> Message-ID: <200312221807.14352.aleaxit@yahoo.com> On Monday 22 December 2003 12:45 pm, Dmitry Vasiliev wrote: ... > >>1. python -p package_name > >> > >> Equivalent to: > >> > >> import package_name ... > The main idea is to treating package as a program and run package > initialization code from command line. The advantage is zipping all > program modules in one zip archive and running the program from command > line without unzipping it, like Java's jar's. But this idea need more > thoughts however... Couldn't you use: python -c "import package_name" for this purpose even today? Alex From tjreedy at udel.edu Mon Dec 22 12:20:19 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Mon Dec 22 12:20:25 2003 Subject: [Python-Dev] Re: Got None. Maybe Some? References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net><20031222075219.GA3426@performancedrivers.com> <20031222000819.10EC.JCARLSON@uci.edu> Message-ID: "Josiah Carlson" wrote in message news:20031222000819.10EC.JCARLSON@uci.edu... > Is None so weird? All I am proposing is a symmetric value to None. To me that would be All of AllType. Some is in between. Can you think of any uses for All other than being max element? TJR From perky at i18n.org Mon Dec 22 12:26:25 2003 From: perky at i18n.org (Hye-Shik Chang) Date: Mon Dec 22 12:26:33 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <16359.8400.767266.692477@montanaro.dyndns.org> References: <16356.62036.22726.310077@montanaro.dyndns.org> <16359.8400.767266.692477@montanaro.dyndns.org> Message-ID: <20031222172625.GA9189@i18n.org> On Mon, Dec 22, 2003 at 10:50:24AM -0600, Skip Montanaro wrote: > >> Sounds like a plan. I modify my local source and see if it affects > >> anything. > > Tim> It should work fine. > > Done. UCHAR_MAX is now required in Python.h, and it's required to be 255. Thanks! > That may open up a couple (small) optimization opportunities (places where > gcc 3.3 says explicit tests against char values will always be true or > false).I'll leave it for others to make those tweaks if they so desire. > Note that some of those gcc messages are probably related to larger types > (e.g. unsigned short) or depend on the presence or absence of other macro > definitions (e.g. Py_UNICODE_WIDE). Should you wish to code any of these > speedups, make sure you get the cpp tests right. Guido won't forgive you if > you don't. I see. Hehe :-) BTW, I wonder whether we have any good way to get C integer types with explicitly defined size such as u_int16_t or int64_t of POSIX. I got a report from an OpenBSD/sparc64 user that he want to change an unsigned int variable to int type which is used for socket.inet_aton. (Modules/socketmodule.c:2887). But I think int type isn't appropriate due to fearing its being 64-bits on some 64bit platforms. Also in_addr_t, POSIX return type of inet_addr(3), is not so portable. If we have types like u_int32_t, it will be very helpful to cope with these sort of issues. Hye-Shik From tjreedy at udel.edu Mon Dec 22 12:29:47 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Mon Dec 22 12:29:48 2003 Subject: [Python-Dev] Re: Relative import References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net><20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> Message-ID: "Troy Melhase" wrote in message news:200312220003.08810.troy@gci.net... > My humble suggestion: > > import NAMES > import NAMES in MODULE [in HOW] > import NAME, OTHER in FOO in RELATIVE > import NAME as RENAME in MODULE in ABSOLUTE I had same idea of reusing 'in', but with relative, absolute changed to perhaps RelMode, AbsMode so as to grammatically flow better. 'in NAME' could then indicate place in package to begin search. 'in Package' and 'in Path' are clearer (more explicit) alternatives to relative and absolute. Terry J. Reedy From guido at python.org Mon Dec 22 12:33:55 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 12:34:01 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: Your message of "Mon, 22 Dec 2003 11:46:01 EST." References: Message-ID: <200312221733.hBMHXtZ18242@c-24-5-183-134.client.comcast.net> > None is handy and I like it, but it's arbitrary enough in some cases > that it takes time to get used to its quirks (e.g., why is a value > that's "not really there" considered to be false? why not? > sometimes that's what you want; sometimes you'd rather get an > exception; None picks an arbitrary answer and sticks to it), so I > don't want see more things like it proliferate (one is enough -- but > not too many ). None inherits most of its semantics from C's NULL -- that's where None being false comes from. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 22 12:37:29 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 12:37:37 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Your message of "Mon, 22 Dec 2003 12:02:57 EST." <20031222170257.GA31352@homer.gst.com> References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> Message-ID: <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> > > The symptom is that without -O3, a "python" binary gets built but its > > use to run setup.py segfaults. Is there anyone here with access to an > > OpenBSD box who could try to reproduce this? > > OpenBSD 3.3 builds both HEAD and r233c1 fine for me. Thanks! That's what I suspected, but it's always good to have confirmation. :) > This is code out of the anoncvs repository btw. Shouldn't matter. If anybody *does* see the bogus behavior and feels like debugging it, I'd appreciate it! --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 22 12:39:00 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 12:39:04 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Mon, 22 Dec 2003 12:07:02 EST." <20031222170702.GA11581@panix.com> References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> <1072104915.18828.47.camel@anthem> <200312221536.hBMFa9S18026@c-24-5-183-134.client.comcast.net> <1072107885.18828.76.camel@anthem> <20031222170702.GA11581@panix.com> Message-ID: <200312221739.hBMHd0G18293@c-24-5-183-134.client.comcast.net> > > Hopefully, this stuff will make it into the PEP so you don't have to > > troll through suggestions buried in endless email threads. > > That's the plan; I expect that people will review the PEP to make sure > their pet proposal is listed. Thanks -- I'll stop tracking this thread closely then until I see the PEP. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Mon Dec 22 12:50:24 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 22 12:50:29 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: <200312221733.hBMHXtZ18242@c-24-5-183-134.client.comcast.net> Message-ID: [Guido] > None inherits most of its semantics from C's NULL -- that's where None > being false comes from. I know that's where it came from, but Python adds its own twists. If it really wanted to act like C's NULL, then print >> None, "oops" should segfault . There's nothing "wrong" about None evaluating to false in a Boolean context, it's simply one choice that *could* have made -- and better than most. From tim.one at comcast.net Mon Dec 22 13:10:36 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 22 13:10:41 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <20031222172625.GA9189@i18n.org> Message-ID: [Hye-Shik Chang] > BTW, I wonder whether we have any good way to get C integer types > with explicitly defined size such as u_int16_t or int64_t of POSIX. We do not, and C doesn't guarantee that any such types exist. For example, while the Crays we recently talked about do support the illusion of 8-bit char, they have no 16-bit or 32-bit integral types (just 8 and 64); that's fine by the C standard. C99 defines a large pile of names, for things like "exactly 16 bits *if* such a thing exists", "smallest integral type holding at least 16 bits (this must exist)", and "fastest integral type holding at least 16 bits (which also must exist)". Since not all C compilers support these names yet, they can't be used directly in Python. If one is needed, it's intended to be added, in pyport.h, following this comment: /* typedefs for some C9X-defined synonyms for integral types. * * The names in Python are exactly the same as the C9X names, except * with a Py_ prefix. Until C9X is universally implemented, this is the * only way to ensure that Python gets reliable names that don't * conflict with names in non-Python code that are playing their own * tricks to define the C9X names. * * NOTE: don't go nuts here! Python has no use for *most* of the C9X * integral synonyms. Only define the ones we actually need. */ > ... > If we have types like u_int32_t, it will be very helpful to cope > with these sort of issues. Not really -- any use of an "exactly N bits" name will make the code uncompilable on some platform. From kbk at shore.net Mon Dec 22 13:57:18 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Mon Dec 22 13:57:23 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Mon, 22 Dec 2003 09:37:29 -0800") References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: > If anybody *does* see the bogus behavior and feels like debugging it, > I'd appreciate it! I've been running cvs Python but not previously with debug switched on. I get a core when using OPT=-g. (Note that I'm running Patch 850977 "Modify setup.py to detect Tcl/Tk on BSD", though I wouldn't think it would cause this, and I doubt he's using it.) Investigating. Hints appreciated! -- KBK Script started on Mon Dec 22 13:30:26 2003 hydra /home/kbk/PYTHON/python/dist/src$ uname -a OpenBSD hydra.localdomain 3.3 GENERIC_RAID#0 i386 hydra /home/kbk/PYTHON/python/dist/src$ hydra /home/kbk/PYTHON/python/dist/src$ make clean find . -name '*.o' -exec rm -f {} ';' find . -name '*.s[ol]' -exec rm -f {} ';' find . -name '*.py[co]' -exec rm -f {} ';' hydra /home/kbk/PYTHON/python/dist/src$ make OPT=-g gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Modules/python.o Modules/python.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/acceler.o Parser/acceler.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/grammar1.o Parser/grammar1.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/listnode.o Parser/listnode.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/node.o Parser/node.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/parser.o Parser/parser.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/parsetok.o Parser/parsetok.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/bitset.o Parser/bitset.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/metagrammar.o Parser/metagrammar.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/firstsets.o Parser/firstsets.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/grammar.o Parser/grammar.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/pgen.o Parser/pgen.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/myreadline.o Parser/myreadline.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/tokenizer.o Parser/tokenizer.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/abstract.o Objects/abstract.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/boolobject.o Objects/boolobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/bufferobject.o Objects/bufferobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/cellobject.o Objects/cellobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/classobject.o Objects/classobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/cobject.o Objects/cobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/complexobject.o Objects/complexobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/descrobject.o Objects/descrobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/enumobject.o Objects/enumobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/fileobject.o Objects/fileobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/floatobject.o Objects/floatobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/frameobject.o Objects/frameobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/funcobject.o Objects/funcobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/intobject.o Objects/intobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/iterobject.o Objects/iterobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/listobject.o Objects/listobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/longobject.o Objects/longobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/dictobject.o Objects/dictobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/methodobject.o Objects/methodobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/moduleobject.o Objects/moduleobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/object.o Objects/object.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/obmalloc.o Objects/obmalloc.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/rangeobject.o Objects/rangeobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/setobject.o Objects/setobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/sliceobject.o Objects/sliceobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/stringobject.o Objects/stringobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/structseq.o Objects/structseq.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/tupleobject.o Objects/tupleobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/typeobject.o Objects/typeobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/weakrefobject.o Objects/weakrefobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/unicodeobject.o Objects/unicodeobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/unicodectype.o Objects/unicodectype.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/bltinmodule.o Python/bltinmodule.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/exceptions.o Python/exceptions.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/ceval.o Python/ceval.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/mysnprintf.o Python/mysnprintf.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/tokenizer_pgen.o Parser/tokenizer_pgen.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/printgrammar.o Parser/printgrammar.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/pgenmain.o Parser/pgenmain.c gcc -pthread -g Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o Python/mysnprintf.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/pgenmain.o -lpthread -lutil -o Parser/pgen Parser/pgen ./Grammar/Grammar ./Include/graminit.h ./Python/graminit.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/compile.o Python/compile.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/codecs.o Python/codecs.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/errors.o Python/errors.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/frozen.o Python/frozen.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/frozenmain.o Python/frozenmain.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/future.o Python/future.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/getargs.o Python/getargs.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/getcompiler.o Python/getcompiler.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/getcopyright.o Python/getcopyright.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/getmtime.o Python/getmtime.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -DPLATFORM='"openbsd3"' -o Python/getplatform.o ./Python/getplatform.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/getversion.o Python/getversion.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/graminit.o Python/graminit.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/import.o Python/import.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -I. -o Python/importdl.o ./Python/importdl.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/marshal.o Python/marshal.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/modsupport.o Python/modsupport.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/mystrtoul.o Python/mystrtoul.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/pyfpe.o Python/pyfpe.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/pystate.o Python/pystate.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/pythonrun.o Python/pythonrun.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/structmember.o Python/structmember.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/symtable.o Python/symtable.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/sysmodule.o Python/sysmodule.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/traceback.o Python/traceback.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/getopt.o Python/getopt.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/dynload_shlib.o Python/dynload_shlib.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/thread.o Python/thread.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Modules/config.o Modules/config.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -DPYTHONPATH='":plat-openbsd3:lib-tk"' -DPREFIX='"/usr/local"' -DEXEC_PREFIX='"/usr/local"' -DVERSION='"2.4"' -DVPATH='""' -o Modules/getpath.o ./Modules/getpath.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Modules/main.o Modules/main.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Modules/gcmodule.o Modules/gcmodule.c gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/threadmodule.c -o Modules/threadmodule.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/signalmodule.c -o Modules/signalmodule.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/posixmodule.c -o Modules/posixmodule.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/errnomodule.c -o Modules/errnomodule.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/_sre.c -o Modules/_sre.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/_codecsmodule.c -o Modules/_codecsmodule.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/zipimport.c -o Modules/zipimport.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/symtablemodule.c -o Modules/symtablemodule.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/xxsubtype.c -o Modules/xxsubtype.o if test -f buildno; then expr `cat buildno` + 1 >buildno1; mv -f buildno1 buildno; else echo 1 >buildno; fi gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -DBUILD=`cat buildno` -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c rm -f libpython2.4.a ar cr libpython2.4.a Modules/getbuildinfo.o ar cr libpython2.4.a Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Parser/myreadline.o Parser/tokenizer.o ar cr libpython2.4.a Objects/abstract.o Objects/boolobject.o Objects/bufferobject.o Objects/cellobject.o Objects/classobject.o Objects/cobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/intobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/methodobject.o Objects/moduleobject.o Objects/object.o Objects/obmalloc.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/stringobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/weakrefobject.o Objects/unicodeobject.o Objects/unicodectype.o ar cr libpython2.4.a Python/bltinmodule.o Python/exceptions.o Python/ceval.o Python/compile.o Python/codecs.o Python/errors.o Python/frozen.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getmtime.o Python/getplatform.o Python/getversion.o Python/graminit.o Python/import.o Python/importdl.o Python/marshal.o Python/modsupport.o Python/mystrtoul.o Python/mysnprintf.o Python/pyfpe.o Python/pystate.o Python/pythonrun.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/traceback.o Python/getopt.o Python/dynload_shlib.o Python/thread.o ar cr libpython2.4.a Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o ar cr libpython2.4.a Modules/threadmodule.o Modules/signalmodule.o Modules/posixmodule.o Modules/errnomodule.o Modules/_sre.o Modules/_codecsmodule.o Modules/zipimport.o Modules/symtablemodule.o Modules/xxsubtype.o ranlib libpython2.4.a gcc -pthread -o python Modules/python.o libpython2.4.a -lpthread -lutil -lm ./Modules/posixmodule.c:5825: warning: tempnam() possibly used unsafely; consider using mkstemp() ./Modules/posixmodule.c:5872: warning: tmpnam() possibly used unsafely; consider using mkstemp() case $MAKEFLAGS in *-s*) CC='gcc -pthread' LDSHARED='ld -Bshareable' OPT='-g' ./python -E ./setup.py -q build;; *) CC='gcc -pthread' LDSHARED='ld -Bshareable' OPT='-g' ./python -E ./setup.py build;; esac Memory fault (core dumped) *** Error code 139 Stop in /home/kbk/PYTHON/python/dist/src (line 335 of Makefile). hydra /home/kbk/PYTHON/python/dist/src$ hydra /home/kbk/PYTHON/python/dist/src$ hydra /home/kbk/PYTHON/python/dist/src$ cvs st configure.in =================================================================== File: configure.in Status: Up-to-date Working revision: 1.442 Repository revision: 1.442 /cvsroot/python/python/dist/src/configure.in,v Sticky Tag: (none) Sticky Date: (none) Sticky Options: (none) hydra /home/kbk/PYTHON/python/dist/src$ Script done on Mon Dec 22 13:31:57 2003 Script started on Mon Dec 22 13:40:31 2003 hydra /home/kbk/PYTHON/python/dist/src$ hydra /home/kbk/PYTHON/python/dist/src$ hydra /home/kbk/PYTHON/python/dist/src$ # [re-build with make clean && make] hydra /home/kbk/PYTHON/python/dist/src$ ./python Python 2.4a0 (#35, Dec 22 2003, 13:38:03) [GCC 2.95.3 20010125 (prerelease, propolice)] on openbsd3 Type "help", "copyright", "credits" or "license" for more information. >>> hydra /home/kbk/PYTHON/python/dist/src$ Script done on Mon Dec 22 13:41:26 2003 From devin at whitebread.org Mon Dec 22 13:36:45 2003 From: devin at whitebread.org (Devin) Date: Mon Dec 22 14:11:48 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312221739.hBMHd0G18293@c-24-5-183-134.client.comcast.net> Message-ID: I've been monitoring all of this discussion about relative imports, and have my own humble idea about how to spell relative imports. The first part stems from Guido's idea about using a '.' at the beginning of a relative import. It seemed a bit weird at first, but I eventually became very comfortable with the idea after thinking about it in respect to absolute imports. I'll explain what I mean. If you currently want to import a module 'a.b.c.d', you write: import a.b.c.d Now, if you're in the module 'a.b', you could still write: import a.b.c.d ... but the 'a.b' part is redundant because you're already in the module 'a.b'. If you take away 'a.b', then you end up with: import .c.d This flowed well in my head, and looks natural. However, the suggestion about '.' or '..' referring to the parent looks _ugly and unnatural_. Let's say '.' was used to spell "parent module", and let's assume that the module 'a.b.c.d' wants to import 'a.b.e'. The absolute import would be spelled: import a.b.e ... and the relative import would be spelled: import .....e # .[parent].[parent].e Yuck! (no offense :) What's needed, I believe, is an identifier/keyword/symbol that spells 'parent' that isn't '.'. Assuming that symbol is represented by '[parent]', the above relative import would look like: import .[parent].[parent].e ... which is a hell lot more clear IMHO. My initial idea is to spell parent '__parent__', and to have '__parent__' be a module-level variable in each module that refers to the parent module (where a top-level module might have '__parent__' refer to itself, or to None). Continuing with the above example, the relative import would be spelled: import .__parent__.__parent__.e ... but I'm not completely convinced that there's not a better identifier/keyword/symbol that could be substituted for '[parent]'. If I'm completely off track, then tell me and I'll continue to lurk until I get a better feel for the happenings of python-dev. -- Devin devin@whitebread.org http://www.whitebread.org/ From blunck at gst.com Mon Dec 22 14:26:53 2003 From: blunck at gst.com (Christopher Blunck) Date: Mon Dec 22 14:26:56 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> Message-ID: <20031222192653.GA31766@homer.gst.com> On Mon, Dec 22, 2003 at 01:57:18PM -0500, Kurt B. Kaiser wrote: > Guido van Rossum writes: > > > If anybody *does* see the bogus behavior and feels like debugging it, > > I'd appreciate it! > > I've been running cvs Python but not previously with debug switched on. > > I get a core when using OPT=-g. > > (Note that I'm running Patch 850977 "Modify setup.py to detect Tcl/Tk on BSD", > though I wouldn't think it would cause this, and I doubt he's using it.) > > Investigating. Hints appreciated! Kurt- Interesting... When I do OPT=-g, I get a Memory fault (core dumped) when linking python. Is that when you receive the core dump? -c -- 14:10:00 up 21 days, 3:54, 12 users, load average: 0.00, 0.02, 0.00 From theller at python.net Mon Dec 22 14:28:24 2003 From: theller at python.net (Thomas Heller) Date: Mon Dec 22 14:28:27 2003 Subject: [Python-Dev] Improve Python for embedding Message-ID: <1xqwodnr.fsf@python.net> I find the current situation when 'embedding' Python rather unsatisfying. 1. Careful preparation (hacking?) of environment variables is needed to avoid that Py_Initialize() takes the default actions needed for the standard Python interpreter (for example, to avoid the default sys.path). 2. A suggestion from /F on c.l.p was to change pythonw.exe so that instead of hiding all output to stderr a console window would be opened to show the tracebacks. This requires to construct an object with a 'write' method doing all this magic, and it must be done after the call to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the code of Py_Main(), it seems impossible without rewriting most of the code. Possible improvements I currently can think of are: - Provide a function like Py_InitializeEx(), which accepts parameters specifying an initial sys.path, optimize flag, verbose flag, and so on. - Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), or let Py_Main() accept a callback function which will be called jsut after Py_Initialize() has been run. I'm not ready to write a PEP, but if there are people sharing this feeling I would suggest we create a wiki entry somewhere acting as a temporary brainstorming area. Thomas From jcarlson at uci.edu Mon Dec 22 14:25:32 2003 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon Dec 22 14:29:05 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: <1072100025.3fe6f2b90ea70@mcherm.com> References: <1072100025.3fe6f2b90ea70@mcherm.com> Message-ID: <20031222111017.0A6D.JCARLSON@uci.edu> > [Josiah proposes a Some object to be the counterpart of None] > > [Guido seems reluctant] > > Josiah: I have a counterproposal. The good news is, it's REALLY easy -- you > won't need a PEP, and I am sure that you'll get no objections from anyone > on this list. And it'll provide you with a "Some" object that acts like > you describe (except the one mention you made of "Some == not None"... > that one's a bad idea since right now "True == not None"). -- snip -- Yes, renaming the class and being more explicit about the comparisons (and tossing the hash function, which turns out to be unneeded) does pretty much what I want it to. But then again, I've had Some since I thought of it. I'm all about this free software idea, and I know that it could be useful to others. I suppose I'll just post information about Some and a sample implementation (combination of yours and mine) on my website or somesuch. - Josiah From guido at python.org Mon Dec 22 14:29:40 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 14:29:45 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Your message of "Mon, 22 Dec 2003 14:26:53 EST." <20031222192653.GA31766@homer.gst.com> References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <20031222192653.GA31766@homer.gst.com> Message-ID: <200312221929.hBMJTef18533@c-24-5-183-134.client.comcast.net> > Interesting... When I do OPT=-g, I get a Memory fault (core dumped) > when linking python. Is that when you receive the core dump? >From looking at his output, he gets it just after the linking, when using the freshly linked python executable to run the setup.py script. Are you saying you didn't try the debug build? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 22 14:33:29 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 14:33:35 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: Your message of "Mon, 22 Dec 2003 20:28:24 +0100." <1xqwodnr.fsf@python.net> References: <1xqwodnr.fsf@python.net> Message-ID: <200312221933.hBMJXTI18559@c-24-5-183-134.client.comcast.net> > I find the current situation when 'embedding' Python rather unsatisfying. I'm interested in improve it. Can you explain what exactly your 'embedding' requirements are? > 1. Careful preparation (hacking?) of environment variables is needed to > avoid that Py_Initialize() takes the default actions needed for the > standard Python interpreter (for example, to avoid the default sys.path). I guess it depends on the embedding needs. Some embedded uses would be quite happy with the default sys.path. > 2. A suggestion from /F on c.l.p was to change pythonw.exe so that > instead of hiding all output to stderr a console window would be opened > to show the tracebacks. This requires to construct an object with a > 'write' method doing all this magic, and it must be done after the call > to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the > code of Py_Main(), it seems impossible without rewriting most of the > code. I'm not sure what this has to do with embedding -- can you clarify? > Possible improvements I currently can think of are: > > - Provide a function like Py_InitializeEx(), which accepts parameters > specifying an initial sys.path, optimize flag, verbose flag, and so on. Except for sys.path, you can set most flags directly before calling Py_Initialize(), so I'm not sure if there's a need to pass these into Py_Initialize(). > - Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), > or let Py_Main() accept a callback function which will be called jsut > after Py_Initialize() has been run. OK, now I'm confused. If you're embedding Python, why would you be using Py_Main() at all? > I'm not ready to write a PEP, but if there are people sharing this > feeling I would suggest we create a wiki entry somewhere acting as a > temporary brainstorming area. Go ahead, but count me out of the wiki -- wikis don't work for discussions for me. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 22 14:35:42 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 14:35:48 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: Your message of "Mon, 22 Dec 2003 11:25:32 PST." <20031222111017.0A6D.JCARLSON@uci.edu> References: <1072100025.3fe6f2b90ea70@mcherm.com> <20031222111017.0A6D.JCARLSON@uci.edu> Message-ID: <200312221935.hBMJZgD18583@c-24-5-183-134.client.comcast.net> > But then again, I've had Some since I thought of it. I'm all about this > free software idea, and I know that it could be useful to others. I > suppose I'll just post information about Some and a sample > implementation (combination of yours and mine) on my website or somesuch. Just please, Please, PLEASE come up with a better name. Some is just too mysterious. Remember that it is *not* in most senses the opposite of None. And even if it was, Some would be a poor choice of name. --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh at python.net Mon Dec 22 14:41:26 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 22 14:41:30 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: (Tim Peters's message of "Sat, 20 Dec 2003 21:52:03 -0500") References: Message-ID: <2mptegej2x.fsf@starship.python.net> "Tim Peters" writes: > [Skip Montanaro] >> Isn't that supposed to always be defined in limits.h or is UCHAR_MAX >> not a standard macro? > > Yes, it's supposed to be there. OTOH, so is limits.h, i.e. the > HAVE_LIMITS_H test shouldn't be necessary either. As Martin regularly points out, we have rather too much of this kind of thing. Given that we demand ANSI C, we could surely lose HAVE_LIMITS_H, and probably much other cruft. People who have to contend with broken platforms might have a different view. I guess there's no real reason to churn, but when crap like this gets in the way we should strive to kill it. Cheers, mwh -- Any form of evilness that can be detected without *too* much effort is worth it... I have no idea what kind of evil we're looking for here or how to detect is, so I can't answer yes or no. -- Guido Van Rossum, python-dev From tree at basistech.com Mon Dec 22 14:40:12 2003 From: tree at basistech.com (Tom Emerson) Date: Mon Dec 22 14:42:28 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <1xqwodnr.fsf@python.net> References: <1xqwodnr.fsf@python.net> Message-ID: <16359.18588.450934.263664@magrathea.basistech.com> Thomas Heller writes: > I find the current situation when 'embedding' Python rather unsatisfying. Indeed. See patch 849278 for some of the work we did to deal with this. > 1. Careful preparation (hacking?) of environment variables is needed to > avoid that Py_Initialize() takes the default actions needed for the > standard Python interpreter (for example, to avoid the default sys.path). I've done this in the patches submitted (by a coworker) above. The idea is to separate the embedded interpreter *completely* from the environment. > 2. A suggestion from /F on c.l.p was to change pythonw.exe so that > instead of hiding all output to stderr a console window would be opened > to show the tracebacks. This requires to construct an object with a > 'write' method doing all this magic, and it must be done after the call > to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the > code of Py_Main(), it seems impossible without rewriting most of the > code. Yes, this is a problem with the embedded interpreter where errors may be written to stderr without regard to the calling application. Further, it is possible for the interpreter to call abort(), which is even worse. > Possible improvements I currently can think of are: > > - Provide a function like Py_InitializeEx(), which accepts parameters > specifying an initial sys.path, optimize flag, verbose flag, and so on. See the patch. > - Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), > or let Py_Main() accept a callback function which will be called jsut > after Py_Initialize() has been run. Yup. > I'm not ready to write a PEP, but if there are people sharing this > feeling I would suggest we create a wiki entry somewhere acting as a > temporary brainstorming area. Absolutely: count me in. -tree -- Tom Emerson Basis Technology Corp. Software Architect http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever" From kbk at shore.net Mon Dec 22 14:45:43 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Mon Dec 22 14:45:49 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312221907.hBMJ7hg18485@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Mon, 22 Dec 2003 11:07:43 -0800") References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <200312221907.hBMJ7hg18485@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: > Very interesting! Had you done a previous non-debug build in the same > directory? (Even if you did a "make clobber" before restarting -- you > never know what leave-behind could cause this.) Yes. I did do a make clobber && ./configure before building the debug version, but it's not a clean checkout. OK, just did a clean checkout, no patch, and got the same result. > The next thing I'd try would be to start the python executable that >was build under gdb and play around in it, like this: > > $ gdb ./python > [...] > (gdb) run > Starting program: /home/guido/projects/python/dist/src/debug/python > [New Thread 1074895104 (LWP 30404)] > Python 2.4a0 (#2, Dec 22 2003, 11:02:19) > [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> print 2+2 >>>> # try various things that don't need external modules except sys >>>> # define a function, etc... >>>> # if that doesn't segfault, try: >>>> from test import autotest > [...] > > Eventually I expect you'd get a segfault; at that point you can use > the gdb 'bt' command to get a stack trace. Hopefully it'll point to > some innocent C code that gets mistreated by the non-debug compiler... It doesn't get as far as the banner: Script started on Mon Dec 22 14:42:19 2003 hydra /home/kbk/proj/sandbox/python_clean$ gdb ./python GNU gdb 4.16.1 Copyright 1996 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-openbsd3.3"... (gdb) run Starting program: /home/kbk/proj/sandbox/python_clean/./python Program received signal SIGSEGV, Segmentation fault. 0x401900a0 in strchr () (gdb) quit The program is running. Quit anyway (and kill it)? (y or n) y hydra /home/kbk/proj/sandbox/python_clean$ Script done on Mon Dec 22 14:42:38 2003 Investigating. -- KBK From jcarlson at uci.edu Mon Dec 22 14:45:19 2003 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon Dec 22 14:47:58 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: <200312221524.hBMFOVj17969@c-24-5-183-134.client.comcast.net> References: <20031221212332.10E9.JCARLSON@uci.edu> <200312221524.hBMFOVj17969@c-24-5-183-134.client.comcast.net> Message-ID: <20031222114456.0A73.JCARLSON@uci.edu> > > Is None so weird? All I am proposing is a symmetric value to None. > > Just like there is a -1 and 1, -Infinity and Infinity, None logically > > should have Some. Maybe not so much -Some == None, but as I propose in > > the PEP, possibly Some == not None. > > None's purposr is not to be less than all numbers (in fact in > the past it wasn't); it's to be "not any object", like nil or NULL in > other languages. > > I don't see a similar broad role for Some. > > And I repeat my question: which other language has that concept? > > --Guido van Rossum (home page: http://www.python.org/~guido/) Certainly if one thinks of None as an object, or a possible pointer to an object, None does fit the role of nil or NULL. The fact is, NULL and nil, as pointers, are 0. None does not equal 0, it is a symbolic representation of the lack of any value (something I remember you saying before, but cannot find a quote), who'se original purpose was to be an equivalent to nil or NULL. As of right now, None has gone beyone its original purpose. As soon as comparisons were able to be done to it and reveal that None was smaller than anything else (even the floating point -infinity), it seemingly has become a more consistant minimum value. Some is just begging to be a more consistant maximum value. It is bigger than sys.maxint, float(1e309), and reveals that the current state is special. What other language has the concept? Python already has the concept. None is currently used as negative infinity in some projects, an actual negative infinity, one that doesn't crap out on integers that cannot be cast into floats. Some would become positive infinity. In terms of non-Python languages, I don't know of any, but that doesn't mean that they don't exist - it just means that I have limited knowledge of programming languages (I only know about 7 or so). - Josiah From mwh at python.net Mon Dec 22 14:48:29 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 22 14:48:33 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: (Tim Peters's message of "Sat, 20 Dec 2003 17:13:34 -0500") References: Message-ID: <2mllp4eir6.fsf@starship.python.net> "Tim Peters" writes: ['>>' is Zack] >> I am not disputing that MSVC does this silly thing you describe, but >> MSVC has a long track record of bugs > > That doesn't match our experience. I don't believe Python code has ever > tickled an MSVC codegen or optimization bug, neither any in gcc-for-x86. Some version of gcc for OpenBSD (on intel) miscompiles (w/optimization) the usual definition of Py_IS_OVERFLOW (or whatever that macro's called). Just nitpicking :-) >> GCC does not generate junk code even at -O0. I'm not sure I understand the context of this remark, but out of context it sure sounds implausible. Cheers, mwh -- The use of COBOL cripples the mind; its teaching should, therefore, be regarded as a criminal offence. -- Edsger W. Dijkstra, SIGPLAN Notices, Volume 17, Number 5 From kbk at shore.net Mon Dec 22 14:48:43 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Mon Dec 22 14:48:51 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <20031222192653.GA31766@homer.gst.com> (Christopher Blunck's message of "Mon, 22 Dec 2003 14:26:53 -0500") References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <20031222192653.GA31766@homer.gst.com> Message-ID: Christopher Blunck writes: > Interesting... When I do OPT=-g, I get a Memory fault (core dumped) > when linking python. Is that when you receive the core dump? Sounds like we're seeing the same thing. -- KBK From guido at python.org Mon Dec 22 14:54:01 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 14:54:07 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: Your message of "Mon, 22 Dec 2003 14:40:12 EST." <16359.18588.450934.263664@magrathea.basistech.com> References: <1xqwodnr.fsf@python.net> <16359.18588.450934.263664@magrathea.basistech.com> Message-ID: <200312221954.hBMJs1M18648@c-24-5-183-134.client.comcast.net> > Indeed. See patch 849278 for some of the work we did to deal with this. This mostly removes the use of the PYTHONXXX environment variables. Can you explain why you don't want these to work? Also note that you can already disable *all* use of the environment (not just in Py_Initialize()) by setting the global Py_IgnoreEnvironmentFlag to a non-zero value. I'd reject that part of the patch for that reason. Py_SetSysPaths() might be a valuable addition, but the game you play with the prefiX name is abject. :) > > 1. Careful preparation (hacking?) of environment variables is needed to > > avoid that Py_Initialize() takes the default actions needed for the > > standard Python interpreter (for example, to avoid the default sys.path). > > I've done this in the patches submitted (by a coworker) above. The > idea is to separate the embedded interpreter *completely* from the > environment. > > > 2. A suggestion from /F on c.l.p was to change pythonw.exe so that > > instead of hiding all output to stderr a console window would be opened > > to show the tracebacks. This requires to construct an object with a > > 'write' method doing all this magic, and it must be done after the call > > to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the > > code of Py_Main(), it seems impossible without rewriting most of the > > code. > > Yes, this is a problem with the embedded interpreter where errors may > be written to stderr without regard to the calling > application. Further, it is possible for the interpreter to call > abort(), which is even worse. What do you suggest the interpreter should do instead of calling abort()? This is only called in response to *fatal* errors -- situations where raising an exception is *not* an acceptable alternative for various reasons (maybe because it's in a piece of code that is part of the implementation of exceptions, or maybe because a corruption is detected in the very interpreter environment that's needed to raise an exception). Anout the only alternative I can think of would be a longjmp() call to a handler you have set up, but that can't be safe when the fatal error is discovered in a different thread. > > Possible improvements I currently can think of are: > > > > - Provide a function like Py_InitializeEx(), which accepts parameters > > specifying an initial sys.path, optimize flag, verbose flag, and so on. > > See the patch. > > > - Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), > > or let Py_Main() accept a callback function which will be called just > > after Py_Initialize() has been run. > > Yup. What's your embedding environment? Why do you want to use Py_Main() at all? --Guido van Rossum (home page: http://www.python.org/~guido/) From theller at python.net Mon Dec 22 14:55:38 2003 From: theller at python.net (Thomas Heller) Date: Mon Dec 22 14:55:43 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <200312221933.hBMJXTI18559@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Mon, 22 Dec 2003 11:33:29 -0800") References: <1xqwodnr.fsf@python.net> <200312221933.hBMJXTI18559@c-24-5-183-134.client.comcast.net> Message-ID: [Thomas Heller] >> I find the current situation when 'embedding' Python rather unsatisfying. > [Guido van Rossum] > I'm interested in improve it. Can you explain what exactly your > 'embedding' requirements are? see below. >> 1. Careful preparation (hacking?) of environment variables is needed to >> avoid that Py_Initialize() takes the default actions needed for the >> standard Python interpreter (for example, to avoid the default sys.path). > > I guess it depends on the embedding needs. Some embedded uses would > be quite happy with the default sys.path. Sure, but others not. >> Possible improvements I currently can think of are: >> >> - Provide a function like Py_InitializeEx(), which accepts parameters >> specifying an initial sys.path, optimize flag, verbose flag, and so on. > > Except for sys.path, you can set most flags directly before calling > Py_Initialize(), so I'm not sure if there's a need to pass these into > Py_Initialize(). I want to specify the Python path before calling Py_Initialize(), and I don't want it to use *any* environment variables which (again) set the flags. So I thought of Py_InitializeEx() which would take parameters specifying these things, and Py_Initialize() could be changed to call Py_InitializeEx(NULL), maybe. >> 2. A suggestion from /F on c.l.p was to change pythonw.exe so that >> instead of hiding all output to stderr a console window would be opened >> to show the tracebacks. This requires to construct an object with a >> 'write' method doing all this magic, and it must be done after the call >> to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the >> code of Py_Main(), it seems impossible without rewriting most of the >> code. > > I'm not sure what this has to do with embedding -- can you clarify? The Python interpreter (dll) embedded into pythonw.exe ;-) > >> - Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), >> or let Py_Main() accept a callback function which will be called jsut >> after Py_Initialize() has been run. > > OK, now I'm confused. If you're embedding Python, why would you be > using Py_Main() at all? I'm probably not good at explaining these things. I was thinking of pythonw.exe (WinMain.c). I want to create a Python object (an stderr writer), and assign this to sys.stderr, without changing too much code. Does it make more sense now? Thomas From mwh at python.net Mon Dec 22 14:56:32 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 22 14:56:36 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: (Terry Reedy's message of "Sun, 21 Dec 2003 21:36:44 -0500") References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> Message-ID: <2m65g8eidr.fsf@starship.python.net> "Terry Reedy" writes: > "Luke Kenneth Casson Leighton" > wrote in message > news:20031220121942.GF1933@lkcl.net... >> if you "feed" access to __class__ via a single > function, you have >> a bottleneck again which can be replaced. >> >> so, the proposal is simple: create an > __builtin__ function called >> __get_class__ through which access to __class__ > is made. >> > > I wonder if it would not be better to turn > __class__ into a property with a get function that > conditionally gives out the real thing. As much as it makes sense to say it, __class__ is a property -- really a getset descriptor, but that's just because it's in C. It would be a imple matter to add a call to PyEval_GetRestricted in object_get_class (not sure whether this is wise or helpful -- only been skimming this thread -- but it certainly ain't hard). Cheers, mwh -- I don't remember any dirty green trousers. -- Ian Jackson, ucam.chat From mwh at python.net Mon Dec 22 14:58:55 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 22 14:58:59 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <20031222192653.GA31766@homer.gst.com> (Christopher Blunck's message of "Mon, 22 Dec 2003 14:26:53 -0500") References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <20031222192653.GA31766@homer.gst.com> Message-ID: <2m1xqwei9s.fsf@starship.python.net> Christopher Blunck writes: > On Mon, Dec 22, 2003 at 01:57:18PM -0500, Kurt B. Kaiser wrote: >> Guido van Rossum writes: >> >> > If anybody *does* see the bogus behavior and feels like debugging it, >> > I'd appreciate it! >> >> I've been running cvs Python but not previously with debug switched on. >> >> I get a core when using OPT=-g. >> >> (Note that I'm running Patch 850977 "Modify setup.py to detect Tcl/Tk on BSD", >> though I wouldn't think it would cause this, and I doubt he's using it.) >> >> Investigating. Hints appreciated! > > Kurt- > > Interesting... When I do OPT=-g, I get a Memory fault (core dumped) > when linking python. Is that when you receive the core dump? What version of gcc? Anthony and I hacked around on HP's compile farm in the 2.3.2 timeframe and saw gcc problems which we hacked around, but nothing like this. Cheers, mwh -- I've even been known to get Marmite *near* my mouth -- but never actually in it yet. Vegamite is right out. UnicodeError: ASCII unpalatable error: vegamite found, ham expected -- Tim Peters, comp.lang.python From guido at python.org Mon Dec 22 14:59:43 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 14:59:48 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Your message of "Mon, 22 Dec 2003 14:45:43 EST." References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <200312221907.hBMJ7hg18485@c-24-5-183-134.client.comcast.net> Message-ID: <200312221959.hBMJxhY18697@c-24-5-183-134.client.comcast.net> > It doesn't get as far as the banner: > > Script started on Mon Dec 22 14:42:19 2003 > hydra /home/kbk/proj/sandbox/python_clean$ gdb ./python > GNU gdb 4.16.1 > Copyright 1996 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-unknown-openbsd3.3"... > (gdb) run > Starting program: /home/kbk/proj/sandbox/python_clean/./python > > Program received signal SIGSEGV, Segmentation fault. > 0x401900a0 in strchr () > (gdb) quit > The program is running. Quit anyway (and kill it)? (y or n) y > hydra /home/kbk/proj/sandbox/python_clean$ > Script done on Mon Dec 22 14:42:38 2003 > > Investigating. That suggests it's still in Py_Initialize(). What does the gdb command 'bt' say??? I'd also try another experiment: instead of "run" try "run -S". This passes the -S option to Python when it is started, so that it doesn't try to load site.py (which executes rather a lot of Python code). I'd be interested in seeing how much you can do interactively in that case, of if it still crashes in Py_Initialize(). --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Mon Dec 22 14:59:00 2003 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon Dec 22 15:00:56 2003 Subject: [Python-Dev] Re: Got None. Maybe Some? In-Reply-To: References: <20031222000819.10EC.JCARLSON@uci.edu> Message-ID: <20031222114715.0A76.JCARLSON@uci.edu> > "Josiah Carlson" wrote in > message > news:20031222000819.10EC.JCARLSON@uci.edu... > > Is None so weird? All I am proposing is a > symmetric value to None. > > To me that would be All of AllType. Some is in > between. Can you think of any uses for All other > than being max element? > > TJR Not really. However, I was thinking of Some in the context of graph and dynamic programming algorithms. When programming some algorithms in Python, you need to either pick some large constant (that you hope will be big enough), or have a test to see if a value is None. That is ugly and very not Pythonic. I do like All and AllType though, which gives a better name, as suggested by Guido. - Josiah From guido at python.org Mon Dec 22 15:03:40 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 15:03:47 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: Your message of "Mon, 22 Dec 2003 20:55:38 +0100." References: <1xqwodnr.fsf@python.net> <200312221933.hBMJXTI18559@c-24-5-183-134.client.comcast.net> Message-ID: <200312222003.hBMK3eH18732@c-24-5-183-134.client.comcast.net> > [Thomas Heller] > >> I find the current situation when 'embedding' Python rather unsatisfying. > > > > [Guido van Rossum] > > I'm interested in improve it. Can you explain what exactly your > > 'embedding' requirements are? > > see below. The only clue I find below is that you're trying to inmprove the error reporting of pythonw.exe, which is certainly laudable, but hardly qualifies as "embedding" IMO. > >> 1. Careful preparation (hacking?) of environment variables is needed to > >> avoid that Py_Initialize() takes the default actions needed for the > >> standard Python interpreter (for example, to avoid the default sys.path). > > > > I guess it depends on the embedding needs. Some embedded uses would > > be quite happy with the default sys.path. > > Sure, but others not. But the only use *you* are interested in is pythonw.exe? > >> Possible improvements I currently can think of are: > >> > >> - Provide a function like Py_InitializeEx(), which accepts parameters > >> specifying an initial sys.path, optimize flag, verbose flag, and so on. > > > > Except for sys.path, you can set most flags directly before calling > > Py_Initialize(), so I'm not sure if there's a need to pass these into > > Py_Initialize(). > > I want to specify the Python path before calling Py_Initialize(), and I > don't want it to use *any* environment variables which (again) set the > flags. So I thought of Py_InitializeEx() which would take parameters > specifying these things, and Py_Initialize() could be changed to call > Py_InitializeEx(NULL), maybe. If you don't want the environment to be used, set the global Py_IgnoreEnvironmentFlag before calling Py_Initialize(). > >> 2. A suggestion from /F on c.l.p was to change pythonw.exe so that > >> instead of hiding all output to stderr a console window would be opened > >> to show the tracebacks. This requires to construct an object with a > >> 'write' method doing all this magic, and it must be done after the call > >> to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the > >> code of Py_Main(), it seems impossible without rewriting most of the > >> code. > > > > I'm not sure what this has to do with embedding -- can you clarify? > > The Python interpreter (dll) embedded into pythonw.exe ;-) > > > > >> - Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), > >> or let Py_Main() accept a callback function which will be called jsut > >> after Py_Initialize() has been run. > > > > OK, now I'm confused. If you're embedding Python, why would you be > > using Py_Main() at all? > > I'm probably not good at explaining these things. > > I was thinking of pythonw.exe (WinMain.c). I want to create a Python > object (an stderr writer), and assign this to sys.stderr, without > changing too much code. > > Does it make more sense now? Not quite -- I don't understand why *you* don't want the environment variables to be used, if all you want is a better pythonw.exe. If that's your only experience with embedding Python, you can't really speak for other embedders. --Guido van Rossum (home page: http://www.python.org/~guido/) From pf_moore at yahoo.co.uk Mon Dec 22 15:04:24 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Mon Dec 22 15:04:19 2003 Subject: [Python-Dev] Re: Got None. Maybe Some? References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <20031222000819.10EC.JCARLSON@uci.edu> Message-ID: Josiah Carlson writes: > I ask the rest of you, does Some sound so crazy? To be honest, yes. To me, None is not the low end of a scale, rather it is a unique distinguished value, more related to the concept of "uninitialised" or "not used" than to "smallest". The fact that it compares less than every other value is to me incidental, and arbitrary. I see None as inhabiting the same sort of niche as the SQL NULL value. >From the above, you'll gather that from my viewpoint, there really is no "other extreme" from None, any more than there is from zero. Paul. -- This signature intentionally left blank From tree at basistech.com Mon Dec 22 15:06:42 2003 From: tree at basistech.com (Tom Emerson) Date: Mon Dec 22 15:08:10 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <200312221954.hBMJs1M18648@c-24-5-183-134.client.comcast.net> References: <1xqwodnr.fsf@python.net> <16359.18588.450934.263664@magrathea.basistech.com> <200312221954.hBMJs1M18648@c-24-5-183-134.client.comcast.net> Message-ID: <16359.20178.766911.979542@magrathea.basistech.com> Guido van Rossum writes: > > Indeed. See patch 849278 for some of the work we did to deal with this. > > This mostly removes the use of the PYTHONXXX environment variables. > Can you explain why you don't want these to work? If the user installs our application and has a separate Python installation on their machine for which they use those environment variables, bad things can happen. I do not want to be in a position where I'm told our app doesn't work because of environment variables we have no control over, and it is the Wrong Answer to tell the user to change their environment. > Also note that you can already disable *all* use of the environment > (not just in Py_Initialize()) by setting the global > Py_IgnoreEnvironmentFlag to a non-zero value. I'd reject that part of > the patch for that reason. I did not know about or notice this flag when I made the changes --- in all the places I looked at the time the environment variables were read without such a check, at least as I remember. It certainly makes more sense to use the built in support. > Py_SetSysPaths() might be a valuable addition, but the game you play > with the prefiX name is abject. :) Sure, whatever. These were made for our own use and later submitted, They can be cleaned up as needed. > What do you suggest the interpreter should do instead of calling > abort()? This is only called in response to *fatal* errors -- > situations where raising an exception is *not* an acceptable > alternative for various reasons (maybe because it's in a piece of code > that is part of the implementation of exceptions, or maybe because a > corruption is detected in the very interpreter environment that's > needed to raise an exception). During initialization if there is a fatal error I can just disable the functionality that uses the interpreter --- having my app abort because Python cannot initialize for whatever reason is a Bad Thing. Similarly with writing messages directly to stderr: I do not necessarily want our users to see these since they don't know what to do with them. > Anout the only alternative I can think of would be a longjmp() call > to a handler you have set up, but that can't be safe when the fatal > error is discovered in a different thread. In our use of Python we build without threads, so the longjmp() would work. In the case of threads all bets are off. > What's your embedding environment? Why do you want to use Py_Main() > at all? I don't use Py_Main() in my embedding (multiple Unix flavors, 32- and 64-bit, and Win32 and Win64)... I wasn't reading closely enough. -tree -- Tom Emerson Basis Technology Corp. Software Architect http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever" From logistix at cathoderaymission.net Mon Dec 22 15:18:24 2003 From: logistix at cathoderaymission.net (logistix) Date: Mon Dec 22 15:09:52 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Message-ID: <000001c3c8c8$c0c116c0$20bba8c0@XP> > -----Original Message----- > From: > python-dev-bounces+logistix=cathoderaymission.net@python.org > [mailto:python-dev-bounces+logistix=cathoderaymission.net@pyth > on.org] On Behalf Of Kurt B. Kaiser > Sent: Monday, December 22, 2003 1:57 PM > To: python-dev@python.org > Subject: Re: [Python-Dev] OpenBSD anyone? > > > Guido van Rossum writes: > > > If anybody *does* see the bogus behavior and feels like > debugging it, > > I'd appreciate it! > > I've been running cvs Python but not previously with debug > switched on. > > I get a core when using OPT=-g. > > (Note that I'm running Patch 850977 "Modify setup.py to > detect Tcl/Tk on BSD", though I wouldn't think it would cause > this, and I doubt he's using it.) > > Investigating. Hints appreciated! > > -- > KBK > > I saw this a while ago and filed bug 712056. In my case it was problems with unicode and the newly introduced iconv module. At the time, this and a few other bugs caused the developers to yank the iconv module from the release and things compiled properly. Could be completely unrelated but I thought I'd mention it. From pf_moore at yahoo.co.uk Mon Dec 22 15:08:54 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Mon Dec 22 15:10:30 2003 Subject: [Python-Dev] Re: Relative import References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> <1072104915.18828.47.camel@anthem> <200312221536.hBMFa9S18026@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: >> I don't either, although my biggest problem with it is that it isn't >> clear just by looking what the punctuation actually means. I just >> fear this will be a permanent head scratcher. > > This argument by itself has very little value. The question isn't > whether it's self-describing (very few features beyond decimal > integers are); the question is whether it's easy to remember. There > are two subclasses of that: easy to remember what it means when you > next see it, and easy to remember how to spell it when you next need > it. IMO the dot syntax does fine here. The biggest "clarity" issue I have with the dot notation is that there's no obvious keyword to look up when you want to find out what's going on. The best you could do is look up the "import" section of the manual. Heaven help you if you want to use Google. (Has anyone tried to get any meaningful information from Google on Microsoft's two technologies, COM and .NET? You'd almost think they'd done it deliberately... Seriously, the "what do I search for?" factor bugs me every time I hit it. Paul. -- This signature intentionally left blank From guido at python.org Mon Dec 22 15:13:55 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 15:14:01 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: Your message of "Mon, 22 Dec 2003 15:06:42 EST." <16359.20178.766911.979542@magrathea.basistech.com> References: <1xqwodnr.fsf@python.net> <16359.18588.450934.263664@magrathea.basistech.com> <200312221954.hBMJs1M18648@c-24-5-183-134.client.comcast.net> <16359.20178.766911.979542@magrathea.basistech.com> Message-ID: <200312222013.hBMKDt118815@c-24-5-183-134.client.comcast.net> > > This mostly removes the use of the PYTHONXXX environment variables. > > Can you explain why you don't want these to work? > > If the user installs our application and has a separate Python > installation on their machine for which they use those environment > variables, bad things can happen. I do not want to be in a position > where I'm told our app doesn't work because of environment variables > we have no control over, and it is the Wrong Answer to tell the user > to change their environment. Fair enough. (Though you could have built your app to work even if Python wrote some random stuff to stderr or optimized its bytecode.) > > Also note that you can already disable *all* use of the environment > > (not just in Py_Initialize()) by setting the global > > Py_IgnoreEnvironmentFlag to a non-zero value. I'd reject that part of > > the patch for that reason. > > I did not know about or notice this flag when I made the changes --- > in all the places I looked at the time the environment variables were > read without such a check, at least as I remember. It certainly makes > more sense to use the built in support. Perhaps it's easy to overlook that it's using Py_GETENV() rather than getenv(); this macro takes tests for Py_IgnoreEnvironmentFlag first. > > Py_SetSysPaths() might be a valuable addition, but the game you play > > with the prefiX name is abject. :) > > Sure, whatever. These were made for our own use and later submitted, > They can be cleaned up as needed. Please do. Add some docs too -- undocumented new code isn't very welcome, since we've already got enough of that. :-) > > What do you suggest the interpreter should do instead of calling > > abort()? This is only called in response to *fatal* errors -- > > situations where raising an exception is *not* an acceptable > > alternative for various reasons (maybe because it's in a piece of code > > that is part of the implementation of exceptions, or maybe because a > > corruption is detected in the very interpreter environment that's > > needed to raise an exception). > > During initialization if there is a fatal error I can just disable the > functionality that uses the interpreter --- having my app abort > because Python cannot initialize for whatever reason is a Bad > Thing. Similarly with writing messages directly to stderr: I do not > necessarily want our users to see these since they don't know what to > do with them. > > > Anout the only alternative I can think of would be a longjmp() call > > to a handler you have set up, but that can't be safe when the fatal > > error is discovered in a different thread. > > In our use of Python we build without threads, so the longjmp() would > work. In the case of threads all bets are off. So that's an indirect answer to my question: you'd be happy if instead of calling abort() it would longjmp() to a location you specified? I guess a better (more general) solution would be to have a callback function that's called instead of abort(), and to initialize this with something that calls abort() by default. That's a useful feature; if you upload it to SF with reference to this thread it ought to be accepted. --Guido van Rossum (home page: http://www.python.org/~guido/) From theller at python.net Mon Dec 22 15:15:08 2003 From: theller at python.net (Thomas Heller) Date: Mon Dec 22 15:15:27 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <200312222003.hBMK3eH18732@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Mon, 22 Dec 2003 12:03:40 -0800") References: <1xqwodnr.fsf@python.net> <200312221933.hBMJXTI18559@c-24-5-183-134.client.comcast.net> <200312222003.hBMK3eH18732@c-24-5-183-134.client.comcast.net> Message-ID: >> I want to specify the Python path before calling Py_Initialize(), and I >> don't want it to use *any* environment variables which (again) set the >> flags. So I thought of Py_InitializeEx() which would take parameters >> specifying these things, and Py_Initialize() could be changed to call >> Py_InitializeEx(NULL), maybe. > > If you don't want the environment to be used, set the global > Py_IgnoreEnvironmentFlag before calling Py_Initialize(). I didn't know that, thanks for the hint. [...] >> Does it make more sense now? > > Not quite -- I don't understand why *you* don't want the environment > variables to be used, if all you want is a better pythonw.exe. I did probably mix two different things here. > If that's your only experience with embedding Python, you can't really > speak for other embedders. That may be - my other experiences are py2exe and the embedding done in bdist_wininst. Now knowing Py_IgnoreEnvironmentFlag I would be satisfied with the addition of a PySys_SetPath function ;-). Thomas From guido at python.org Mon Dec 22 15:15:44 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 15:15:50 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Mon, 22 Dec 2003 20:08:54 GMT." References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> <1072104915.18828.47.camel@anthem> <200312221536.hBMFa9S18026@c-24-5-183-134.client.comcast.net> Message-ID: <200312222015.hBMKFi918839@c-24-5-183-134.client.comcast.net> > The biggest "clarity" issue I have with the dot notation is that > there's no obvious keyword to look up when you want to find out what's > going on. The best you could do is look up the "import" section of the > manual. Heaven help you if you want to use Google. (Has anyone tried > to get any meaningful information from Google on Microsoft's two > technologies, COM and .NET? You'd almost think they'd done it > deliberately... > > Seriously, the "what do I search for?" factor bugs me every time I hit > it. There's no deep reason why the language reference couldn't contain non-alphanumeric characters, although this won't help for Google. But in this case, "from" would be the thing to look up, so I'm not sure I see it as such a serious problem. --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh at python.net Mon Dec 22 15:24:12 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 22 15:24:17 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: (Tim Peters's message of "Mon, 22 Dec 2003 11:46:01 -0500") References: Message-ID: <2mwu8od2j7.fsf@starship.python.net> "Tim Peters" writes: > None is handy and I like it, but it's arbitrary enough in some cases > that it takes time to get used to its quirks At least we don't pretend it's the empty list as well... Cheers, mwh -- On the other hand, the following areas are subject to boycott in reaction to the rampant impurity of design or execution, as determined after a period of study, in no particular order: ... http://www.naggum.no/profile.html From tim.one at comcast.net Mon Dec 22 15:34:38 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 22 15:34:42 2003 Subject: [Python-Dev] Re: Got None. Maybe Some? In-Reply-To: Message-ID: [Paul Moore] > ... > The fact that [None] compares less than every other value is to me > incidental, and arbitrary. Arbitrary?! I clearly recall when the decision was made that None would compare less than anything else. I was in the same room with Guido, and this is a verbatim transcript of the debate: "As long as we're changing everything here anyway, maybe comparing with None should yield a consistent result." "Like what, maybe smaller than anything else?" "Sure. Or larger." "Ya, one of those. It's not worth it if it just ends up in the middle somewhere." "What does Python do now?" ... [head-scratching] ... "So what did it do before?" ... [more head-scratching] ... "Well, that's no help, then." "The Reference Manual doesn't promise anything about how None will compare, just the 'consistent total ordering' bit". "So that doesn't constrain us." "No, it's neutral." "So maybe it should just be smaller than everything else." "Sure. Or larger." "If it were smaller, then when you sorted a mixed-type list, all the Nones would float to the start." "Good point. If it were larger, they'd float to the end." "I wouldn't like that much." "Me neither." "OK, it's settled." Two lines of code later, the Only Objectively Correct result was achieved. history-is-more-impressive-in-textbooks-ly y'rs - tim From guido at python.org Mon Dec 22 16:04:11 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 16:04:17 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) Message-ID: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.net> New-style classes inherit a default __hash__ from object, but this is incorrect if the inheriting object defines __eq__ or __cmp__. This has been reported several times; the roll-up SF entry is 660098, which has the proposed patch attached as newpatch.txt. I've long pondered this, and it seems the best solution is to simply not define __hash__ on the base object class. After all, you don't inherit __eq__ or __cmp__ either; the default comparison behavior is supplied by intrinsic behavior of comparisons, and the default hash() behavior is also supplied by intrinsic behavior of hash(). Does anybody have any reason why I shouldn't check in this change in 2.4? There's no unit test that fails if I do this. It *is* a slight incompatibility though: if someone implemented their __hash__ in terms of object.__hash__, they would get a somewhat puzzling error message after the change, because object.__hash__ will refer to the type.__hash__ function bound to object, and refuse to take an argument: >>> object.__hash__() 135328576 >>> object.__hash__(42) Traceback (most recent call last): File "", line 1, in ? TypeError: expected 0 arguments, got 1 >>> Since the default hash simply takes id() of the object, it's easy to fix such code once the failure is understood though. --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis at bluewin.ch Mon Dec 22 16:23:26 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Mon Dec 22 16:20:00 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.ne t> Message-ID: <5.2.1.1.0.20031222221506.02f2ccf8@pop.bluewin.ch> At 13:04 22.12.2003 -0800, Guido van Rossum wrote: >Since the default hash simply takes id() of the object, it's easy to >fix such code once the failure is understood though. Jython 2.2a0 on java1.4.0_02 (JIT: null) Type "copyright", "credits" or "license" for more information. >>> class C: pass ... >>> id(C) 1 >>> hash(C) 9174506 >>> I see, but then there should be probably be a different way to spell the default hash, because id is not a sensible option for Jython etc From pje at telecommunity.com Mon Dec 22 16:19:10 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Dec 22 16:26:29 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.ne t> Message-ID: <5.1.1.6.0.20031222161608.02e98d50@telecommunity.com> At 01:04 PM 12/22/03 -0800, Guido van Rossum wrote: >New-style classes inherit a default __hash__ from object, but this >is incorrect if the inheriting object defines __eq__ or __cmp__. > >This has been reported several times; the roll-up SF entry is 660098, >which has the proposed patch attached as newpatch.txt. > >I've long pondered this, and it seems the best solution is to simply >not define __hash__ on the base object class. After all, you don't >inherit __eq__ or __cmp__ either; the default comparison behavior is >supplied by intrinsic behavior of comparisons, and the default hash() >behavior is also supplied by intrinsic behavior of hash(). > >Does anybody have any reason why I shouldn't check in this change in >2.4? There's no unit test that fails if I do this. Would this mean that instances of the following class: class Dummy(object): pass would no longer be usable as dictionary keys? I guess the parts I'm not clear on are 1) whether dictionaries call the equivalent of 'hash(key)' or 'key.__hash__()', and 2) whether 'hash(Dummy())' will do something meaningful. From guido at python.org Mon Dec 22 16:29:23 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 16:29:29 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: Your message of "Mon, 22 Dec 2003 16:19:10 EST." <5.1.1.6.0.20031222161608.02e98d50@telecommunity.com> References: <5.1.1.6.0.20031222161608.02e98d50@telecommunity.com> Message-ID: <200312222129.hBMLTNT19225@c-24-5-183-134.client.comcast.net> > Would this mean that instances of the following class: > > class Dummy(object): > pass > > would no longer be usable as dictionary keys? I guess the parts I'm > not clear on are 1) whether dictionaries call the equivalent of > 'hash(key)' or 'key.__hash__()', and 2) whether 'hash(Dummy())' will > do something meaningful. That class would still be usable. hash(Dummy()) sees that there's no __eq__ or __cmp__ override and then uses the default hash. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 22 16:32:15 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 16:32:21 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: Your message of "Mon, 22 Dec 2003 22:23:26 +0100." <5.2.1.1.0.20031222221506.02f2ccf8@pop.bluewin.ch> References: <5.2.1.1.0.20031222221506.02f2ccf8@pop.bluewin.ch> Message-ID: <200312222132.hBMLWFe19253@c-24-5-183-134.client.comcast.net> > >Since the default hash simply takes id() of the object, it's easy to > >fix such code once the failure is understood though. > > Jython 2.2a0 on java1.4.0_02 (JIT: null) > Type "copyright", "credits" or "license" for more information. > >>> class C: pass > ... > >>> id(C) > 1 > >>> hash(C) > 9174506 > >>> > > I see, but then there should be probably be a different way to spell > the default hash, because id is not a sensible option for Jython etc Ow, you're right. I bet this is why object.__hash__ was introduced in the first place. We're either back to square one, or we can add a __default_hash__ to object which has the default hash implementation -- this isn't very pretty but at least it works. (I should add that this topic rose to my attention when I had to debug the issue in some code of my own -- I had a class whose __eq__ was overridden but whose __hash__ wasn't, and it was being used as a dict key... It is indeed a pain to debug this. :-( ) --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis at bluewin.ch Mon Dec 22 16:43:40 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Mon Dec 22 16:40:25 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: <200312222132.hBMLWFe19253@c-24-5-183-134.client.comcast.ne t> References: <5.2.1.1.0.20031222221506.02f2ccf8@pop.bluewin.ch> Message-ID: <5.2.1.1.0.20031222224015.02f0a640@pop.bluewin.ch> At 13:32 22.12.2003 -0800, Guido van Rossum wrote: > > >Since the default hash simply takes id() of the object, it's easy to > > >fix such code once the failure is understood though. > > > > Jython 2.2a0 on java1.4.0_02 (JIT: null) > > Type "copyright", "credits" or "license" for more information. > > >>> class C: pass > > ... > > >>> id(C) > > 1 > > >>> hash(C) > > 9174506 > > >>> > > > > I see, but then there should be probably be a different way to spell > > the default hash, because id is not a sensible option for Jython etc > >Ow, you're right. I bet this is why object.__hash__ was introduced in >the first place. > >We're either back to square one, or we can add a __default_hash__ to >object which has the default hash implementation -- this isn't very >pretty but at least it works. yes, or hash should grow a keyword argument to get access to the default impl, which would be id for CPython but not for all Python impls in general. The problem is that in general it cannot be expected from Python implementations to implement the default hash in terms of id because the latter can be much more costly beacuse of its uniqueness constraints. regards. From kbk at shore.net Mon Dec 22 16:41:13 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Mon Dec 22 16:41:18 2003 Subject: [Python-Dev] OpenBSD anyone? References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <200312221907.hBMJ7hg18485@c-24-5-183-134.client.comcast.net> <200312221959.hBMJxhY18697@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: > That suggests it's still in Py_Initialize(). What does the gdb > command 'bt' say??? > > I'd also try another experiment: instead of "run" try "run -S". This > passes the -S option to Python when it is started, so that it doesn't > try to load site.py (which executes rather a lot of Python code). I'd > be interested in seeing how much you can do interactively in that > case, of if it still crashes in Py_Initialize(). Script started on Mon Dec 22 15:20:25 2003 hydra /home/kbk/proj/sandbox/python_clean$ gdb ./python GNU gdb 4.16.1 Copyright 1996 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-openbsd3.3"... (gdb) run -S Starting program: /home/kbk/proj/sandbox/python_clean/./python -S Program received signal SIGSEGV, Segmentation fault. 0x401900a0 in strchr () (gdb) bt #0 0x401900a0 in strchr () #1 0x1d11d in load_next (mod=0x10905c, altmod=0x10905c, p_name=0xcfbfd640, buf=0xcfbfd230 "__builtin__", p_buflen=0xcfbfd22c) at Python/import.c:2004 #2 0x1cc40 in import_module_ex (name=0x19786 "__builtin__", globals=0x0, locals=0x0, fromlist=0x0) at Python/import.c:1888 #3 0x1ce29 in PyImport_ImportModuleEx (name=0x19786 "__builtin__", globals=0x0, locals=0x0, fromlist=0x0) at Python/import.c:1922 #4 0x1dfe9 in PyImport_Import (module_name=0x115598) at Python/import.c:2333 #5 0x1caec in PyImport_ImportModule (name=0xb895e "__builtin__") at Python/import.c:1853 #6 0xb8b3d in _PyExc_Init () at Python/exceptions.c:1755 #7 0x25ad4 in Py_Initialize () at Python/pythonrun.c:205 #8 0x282f in Py_Main (argc=2, argv=0xcfbfd82c) at Modules/main.c:376 #9 0x17e3 in main (argc=2, argv=0xcfbfd82c) at Modules/python.c:23 (gdb) q The program is running. Quit anyway (and kill it)? (y or n) y hydra /home/kbk/proj/sandbox/python_clean$ Script done on Mon Dec 22 15:21:06 2003 Same result w/o -S ============================================================= Slightly later: (gdb) b import.c:2004 Breakpoint 1 at 0x1d10f: file Python/import.c, line 2004. (gdb) r Starting program: /home/kbk/proj/sandbox/python_clean/./python Breakpoint 1, load_next (mod=0xe98ec, altmod=0xe98ec, p_name=0xcfbfd88c, buf=0xcfbfd47c "", p_buflen=0xcfbfd478) at Python/import.c:2004 2004 char *dot = strchr(name, '.'); (gdb) p *p_name $1 = 0x19786 "__builtin__" (gdb) p name $2 = 0x19786 "__builtin__" (gdb) p strchr(name, '.') Program received signal SIGSEGV, Segmentation fault. OTOH, if I break at 2004 and then step once, I get by the strchr call OK. Also if I stepi through it. If I continue, it segfaults at the next execution of line 2004. Weird. Investigating. -- KBK From guido at python.org Mon Dec 22 16:43:09 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 16:43:16 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: Your message of "Mon, 22 Dec 2003 22:43:40 +0100." <5.2.1.1.0.20031222224015.02f0a640@pop.bluewin.ch> References: <5.2.1.1.0.20031222221506.02f2ccf8@pop.bluewin.ch> <5.2.1.1.0.20031222224015.02f0a640@pop.bluewin.ch> Message-ID: <200312222143.hBMLh9C19299@c-24-5-183-134.client.comcast.net> > > > I see, but then there should be probably be a different way to spell > > > the default hash, because id is not a sensible option for Jython etc > > > >Ow, you're right. I bet this is why object.__hash__ was introduced in > >the first place. > > > >We're either back to square one, or we can add a __default_hash__ to > >object which has the default hash implementation -- this isn't very > >pretty but at least it works. > > yes, or hash should grow a keyword argument to get access to the default > impl, which would be id for CPython but not for all Python impls in > general. The problem is that in general it cannot be expected from Python > implementations to implement the default hash in terms of id because the > latter can be much more costly beacuse of its uniqueness constraints. Hm, the keyword arg would be more work to implement in C, and I don't really like the ad-hoc extension of the __hash__ signature -- it may make subclasses think they have to support the keyword too. (Overriding the __default_hash__ would be possible, but silly, and I recommend not to support this; if this were Java I'd make it a final method. Can you make it final in Jython?) --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis at bluewin.ch Mon Dec 22 16:55:48 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Mon Dec 22 16:52:22 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: <200312222143.hBMLh9C19299@c-24-5-183-134.client.comcast.ne t> References: <5.2.1.1.0.20031222221506.02f2ccf8@pop.bluewin.ch> <5.2.1.1.0.20031222224015.02f0a640@pop.bluewin.ch> Message-ID: <5.2.1.1.0.20031222224820.02f31840@pop.bluewin.ch> At 13:43 22.12.2003 -0800, Guido van Rossum wrote: > > > > I see, but then there should be probably be a different way to spell > > > > the default hash, because id is not a sensible option for Jython etc > > > > > >Ow, you're right. I bet this is why object.__hash__ was introduced in > > >the first place. > > > > > >We're either back to square one, or we can add a __default_hash__ to > > >object which has the default hash implementation -- this isn't very > > >pretty but at least it works. > > > > yes, or hash should grow a keyword argument to get access to the default > > impl, which would be id for CPython but not for all Python impls in > > general. The problem is that in general it cannot be expected from Python > > implementations to implement the default hash in terms of id because the > > latter can be much more costly beacuse of its uniqueness constraints. > >Hm, the keyword arg would be more work to implement in C, and I don't >really like the ad-hoc extension of the __hash__ signature -- it may >make subclasses think they have to support the keyword too. > >(Overriding the __default_hash__ would be possible, but silly, and I >recommend not to support this; if this were Java I'd make it a final >method. Can you make it final in Jython?) internally yes. But I would probably hardcode is usage in hash, so that is just a way to access the default hash impl and nothing more. From guido at python.org Mon Dec 22 16:56:40 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 16:56:52 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Your message of "Mon, 22 Dec 2003 16:41:13 EST." References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <200312221907.hBMJ7hg18485@c-24-5-183-134.client.comcast.net> <200312221959.hBMJxhY18697@c-24-5-183-134.client.comcast.net> Message-ID: <200312222156.hBMLueA19333@c-24-5-183-134.client.comcast.net> (JD: yes, it's real!) > Script started on Mon Dec 22 15:20:25 2003 > hydra /home/kbk/proj/sandbox/python_clean$ gdb ./python > GNU gdb 4.16.1 > Copyright 1996 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-unknown-openbsd3.3"... > (gdb) run -S > Starting program: /home/kbk/proj/sandbox/python_clean/./python -S > > Program received signal SIGSEGV, Segmentation fault. > 0x401900a0 in strchr () > (gdb) bt > #0 0x401900a0 in strchr () > #1 0x1d11d in load_next (mod=0x10905c, altmod=0x10905c, p_name=0xcfbfd640, > buf=0xcfbfd230 "__builtin__", p_buflen=0xcfbfd22c) at Python/import.c:2004 > #2 0x1cc40 in import_module_ex (name=0x19786 "__builtin__", globals=0x0, locals=0x0, > fromlist=0x0) at Python/import.c:1888 > #3 0x1ce29 in PyImport_ImportModuleEx (name=0x19786 "__builtin__", globals=0x0, locals=0x0, > fromlist=0x0) at Python/import.c:1922 > #4 0x1dfe9 in PyImport_Import (module_name=0x115598) at Python/import.c:2333 > #5 0x1caec in PyImport_ImportModule (name=0xb895e "__builtin__") at Python/import.c:1853 > #6 0xb8b3d in _PyExc_Init () at Python/exceptions.c:1755 > #7 0x25ad4 in Py_Initialize () at Python/pythonrun.c:205 > #8 0x282f in Py_Main (argc=2, argv=0xcfbfd82c) at Modules/main.c:376 > #9 0x17e3 in main (argc=2, argv=0xcfbfd82c) at Modules/python.c:23 > (gdb) q > The program is running. Quit anyway (and kill it)? (y or n) y > hydra /home/kbk/proj/sandbox/python_clean$ > Script done on Mon Dec 22 15:21:06 2003 > > > Same result w/o -S > > ============================================================= > Slightly later: > > (gdb) b import.c:2004 > Breakpoint 1 at 0x1d10f: file Python/import.c, line 2004. > (gdb) r > Starting program: /home/kbk/proj/sandbox/python_clean/./python > > Breakpoint 1, load_next (mod=0xe98ec, altmod=0xe98ec, p_name=0xcfbfd88c, buf=0xcfbfd47c "", > p_buflen=0xcfbfd478) at Python/import.c:2004 > 2004 char *dot = strchr(name, '.'); > (gdb) p *p_name > $1 = 0x19786 "__builtin__" > (gdb) p name > $2 = 0x19786 "__builtin__" > (gdb) p strchr(name, '.') > > Program received signal SIGSEGV, Segmentation fault. > > OTOH, if I break at 2004 and then step once, I get by the strchr call > OK. Also if I stepi through it. If I continue, it segfaults at the > next execution of line 2004. Weird. > > Investigating. The most likely cause then is some kind of bug in the platform's strchr(). This could explain why -O3 fixes the issue: I think I've heard of GCC replacing calls to strchr(), strcpy() etc. with inline code, thereby avoiding the buggy library version (and explaining why the buggy code could persist undetected in the library -- most system code is of course compiled fully optimized). As to why stepi doesn't trigger the segfault: possibly it's a timing bug that doesn't occur when run one instruction at a time. This would even make it CPU dependent, which would explain that some folks didn't see this. I don't have the OpenBSD strchr.c source code online here so I'll stop speculating here... --Guido van Rossum (home page: http://www.python.org/~guido/) From blunck at gst.com Mon Dec 22 16:58:55 2003 From: blunck at gst.com (Christopher Blunck) Date: Mon Dec 22 16:58:59 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312221929.hBMJTef18533@c-24-5-183-134.client.comcast.net> References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <20031222192653.GA31766@homer.gst.com> <200312221929.hBMJTef18533@c-24-5-183-134.client.comcast.net> Message-ID: <20031222215855.GA32420@homer.gst.com> On Mon, Dec 22, 2003 at 11:29:40AM -0800, Guido van Rossum wrote: > > Interesting... When I do OPT=-g, I get a Memory fault (core dumped) > > when linking python. Is that when you receive the core dump? > > >From looking at his output, he gets it just after the linking, when > using the freshly linked python executable to run the setup.py script. > > Are you saying you didn't try the debug build? I received the dump when trying to run setup.py build. -c -- 16:55:00 up 21 days, 6:39, 12 users, load average: 1.30, 0.60, 0.24 From whisper at oz.net Mon Dec 22 15:43:09 2003 From: whisper at oz.net (David LeBlanc) Date: Mon Dec 22 16:59:31 2003 Subject: [Python-Dev] Re: Got None. Maybe Some? In-Reply-To: Message-ID: > [Paul Moore] > > ... > > The fact that [None] compares less than every other value is to me > > incidental, and arbitrary. > > Arbitrary?! I clearly recall when the decision was made that None would > compare less than anything else. I was in the same room with Guido, and > this is a verbatim transcript of the debate: > Two lines of code later, the Only Objectively Correct result was achieved. > > history-is-more-impressive-in-textbooks-ly y'rs - tim > How about "everything" or "all" as the obverse of "none", ordered higher than all other values? "Some" is far too indeterminate. Dave LeBlanc Seattle, WA USA From blunck at gst.com Mon Dec 22 16:59:54 2003 From: blunck at gst.com (Christopher Blunck) Date: Mon Dec 22 17:00:29 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <2m1xqwei9s.fsf@starship.python.net> References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <20031222192653.GA31766@homer.gst.com> <2m1xqwei9s.fsf@starship.python.net> Message-ID: <20031222215954.GB32420@homer.gst.com> On Mon, Dec 22, 2003 at 07:58:55PM +0000, Michael Hudson wrote: > What version of gcc? Anthony and I hacked around on HP's compile farm > in the 2.3.2 timeframe and saw gcc problems which we hacked around, > but nothing like this. 2.95.3. But I think I was mistaken in my last email. I get the dump when I run setup.py build, not when linking python. -c -- 16:55:00 up 21 days, 6:39, 12 users, load average: 1.30, 0.60, 0.24 From barry at python.org Mon Dec 22 17:10:07 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 22 17:10:20 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.net> References: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.net> Message-ID: <1072131005.13676.9.camel@anthem> On Mon, 2003-12-22 at 16:04, Guido van Rossum wrote: > >>> object.__hash__() > 135328576 > >>> object.__hash__(42) > Traceback (most recent call last): > File "", line 1, in ? > TypeError: expected 0 arguments, got 1 > >>> Would it be better if object.__hash__() raised a NotImplementedError? -Barry From jepler at unpythonic.net Mon Dec 22 17:10:58 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Mon Dec 22 17:11:03 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <20031222140255.GH12586@unpythonic.net> References: <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> Message-ID: <20031222221058.GD24252@unpythonic.net> On Mon, Dec 22, 2003 at 08:03:17AM -0600, Jeff Epler wrote: > Well, unlike the last two bugs I filed in redhat bugzilla, this one has > already seen some activity. I hope they don't take the line that this > is not a bug :( Unfortunately, this is redhat's position. ------- Additional Comments From roland@redhat.com 2003-12-22 16:37 ------- I think it is clear that the specification refers to the elements of the child process state that survive exec, so that the executed command can perceive them as part of its "environment". You could submit an interpretation request, but I think the committee would concur with my reading. The specification of pthread_atfork refers to calls to fork, not to other parts of the POSIX.1 implementation. If your application calls system, and not fork, those clauses do not apply to it. roland@redhat.com changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution| |NOTABUG Status|ASSIGNED |CLOSED From guido at python.org Mon Dec 22 17:13:02 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 17:13:15 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: Your message of "Mon, 22 Dec 2003 17:10:07 EST." <1072131005.13676.9.camel@anthem> References: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.net> <1072131005.13676.9.camel@anthem> Message-ID: <200312222213.hBMMD2m19449@c-24-5-183-134.client.comcast.net> > > >>> object.__hash__() > > 135328576 > > >>> object.__hash__(42) > > Traceback (most recent call last): > > File "", line 1, in ? > > TypeError: expected 0 arguments, got 1 > > >>> > > Would it be better if object.__hash__() raised a NotImplementedError? It can't -- it's type.__hash__(object). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 22 17:23:11 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 17:23:17 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: Your message of "Mon, 22 Dec 2003 16:10:58 CST." <20031222221058.GD24252@unpythonic.net> References: <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> <20031222221058.GD24252@unpythonic.net> Message-ID: <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> > Unfortunately, this is redhat's position. > > ------- Additional Comments From roland@redhat.com 2003-12-22 16:37 ------- > I think it is clear that the specification refers to the elements of > the child process state that survive exec, so that the executed > command can perceive them as part of its "environment". You could > submit an interpretation request, but I think the committee would > concur with my reading. The specification of pthread_atfork refers to > calls to fork, not to other parts of the POSIX.1 implementation. If > your application calls system, and not fork, those clauses do not > apply to it. How hard would it be to reimplement our own system() and popen() using only POSIX calls, for POSIX systems? I've always thought of these to be pretty simple combinations of fork() and exec(), with an assumption of a working /bin/sh. Without error checking: int system(char *cmd) { pid = fork(); if (!pid) { /* child */ execl("/bin/sh", "-c", cmd, NULL); } else { /* parent */ int sts; waitpid(pid, *sts, 0); return sts; } --Guido van Rossum (home page: http://www.python.org/~guido/) From jepler at unpythonic.net Mon Dec 22 17:38:20 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Mon Dec 22 17:38:26 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> References: <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> <20031222221058.GD24252@unpythonic.net> <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> Message-ID: <20031222223811.GA1685@unpythonic.net> On Mon, Dec 22, 2003 at 02:23:11PM -0800, Guido van Rossum wrote: > How hard would it be to reimplement our own system() and popen() using > only POSIX calls, for POSIX systems? I've always thought of these to > be pretty simple combinations of fork() and exec(), with an assumption > of a working /bin/sh. Without error checking: Yeah -- this is something we could do if necessary. the opengroup standard lists a few more things to "get right" in terms of signal handling for system(), but we can do all those in C or in Python.. Jeff From mwh at python.net Mon Dec 22 17:39:53 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 22 17:39:55 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <200312222013.hBMKDt118815@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Mon, 22 Dec 2003 12:13:55 -0800") References: <1xqwodnr.fsf@python.net> <16359.18588.450934.263664@magrathea.basistech.com> <200312221954.hBMJs1M18648@c-24-5-183-134.client.comcast.net> <16359.20178.766911.979542@magrathea.basistech.com> <200312222013.hBMKDt118815@c-24-5-183-134.client.comcast.net> Message-ID: <2msmjccw92.fsf@starship.python.net> Guido van Rossum writes: >> > (not just in Py_Initialize()) by setting the global >> > Py_IgnoreEnvironmentFlag to a non-zero value. I'd reject that part of >> > the patch for that reason. >> >> I did not know about or notice this flag when I made the changes --- >> in all the places I looked at the time the environment variables were >> read without such a check, at least as I remember. It certainly makes >> more sense to use the built in support. > > Perhaps it's easy to overlook that it's using Py_GETENV() rather than > getenv(); this macro takes tests for Py_IgnoreEnvironmentFlag first. Py_IgnoreEnvironmentFlag and the -E command line option are both /fairly/ new... 2.2 era? Cheers, mwh -- Worryingly, DEFUN appears to be a function that removes all the fun from something: after using it all your code is converted to C++. -- Tim Bradshaw, comp.lang.lisp From mwh at python.net Mon Dec 22 17:41:37 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 22 17:41:40 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Mon, 22 Dec 2003 14:23:11 -0800") References: <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> <20031222221058.GD24252@unpythonic.net> <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> Message-ID: <2moeu0cw66.fsf@starship.python.net> Guido van Rossum writes: >> Unfortunately, this is redhat's position. >> >> ------- Additional Comments From roland@redhat.com 2003-12-22 16:37 ------- >> I think it is clear that the specification refers to the elements of >> the child process state that survive exec, so that the executed >> command can perceive them as part of its "environment". You could >> submit an interpretation request, but I think the committee would >> concur with my reading. The specification of pthread_atfork refers to >> calls to fork, not to other parts of the POSIX.1 implementation. If >> your application calls system, and not fork, those clauses do not >> apply to it. > > How hard would it be to reimplement our own system() and popen() using > only POSIX calls, for POSIX systems? I've always thought of these to > be pretty simple combinations of fork() and exec(), with an assumption > of a working /bin/sh. Without error checking: I think it's a bit harder than what you post, but there's code in APUE for it... Cheers, mwh -- I'd certainly be shocked to discover a consensus. ;-) -- Aahz, comp.lang.python From jcarlson at uci.edu Mon Dec 22 17:41:29 2003 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon Dec 22 17:44:59 2003 Subject: [Python-Dev] Re: Got None. Maybe Some? In-Reply-To: References: Message-ID: <20031222144122.0A7F.JCARLSON@uci.edu> > > [Paul Moore] > > > ... > > > The fact that [None] compares less than every other value is to me > > > incidental, and arbitrary. > > > > Arbitrary?! I clearly recall when the decision was made that None would > > compare less than anything else. I was in the same room with Guido, and > > this is a verbatim transcript of the debate: > > > Two lines of code later, the Only Objectively Correct result was achieved. > > > > history-is-more-impressive-in-textbooks-ly y'rs - tim > > > How about "everything" or "all" as the obverse of "none", ordered higher > than all other values? "Some" is far too indeterminate. > > Dave LeBlanc > Seattle, WA USA All seems like a reasonable spelling for an object with the given behavior. I started with the name Some because the name made me smile. - Josiah From guido at python.org Mon Dec 22 17:45:04 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 17:45:13 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: Your message of "Mon, 22 Dec 2003 22:39:53 GMT." <2msmjccw92.fsf@starship.python.net> References: <1xqwodnr.fsf@python.net> <16359.18588.450934.263664@magrathea.basistech.com> <200312221954.hBMJs1M18648@c-24-5-183-134.client.comcast.net> <16359.20178.766911.979542@magrathea.basistech.com> <200312222013.hBMKDt118815@c-24-5-183-134.client.comcast.net> <2msmjccw92.fsf@starship.python.net> Message-ID: <200312222245.hBMMj4W19537@c-24-5-183-134.client.comcast.net> > > Perhaps it's easy to overlook that it's using Py_GETENV() rather than > > getenv(); this macro takes tests for Py_IgnoreEnvironmentFlag first. > > Py_IgnoreEnvironmentFlag and the -E command line option are both > /fairly/ new... 2.2 era? Yeah, but the patch he suggested was using Py_GETENV(), so it's not like it didn't exist when they did the patch; they must have missed its significance. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From tree at basistech.com Mon Dec 22 18:49:37 2003 From: tree at basistech.com (Tom Emerson) Date: Mon Dec 22 18:51:32 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <200312222245.hBMMj4W19537@c-24-5-183-134.client.comcast.net> References: <1xqwodnr.fsf@python.net> <16359.18588.450934.263664@magrathea.basistech.com> <200312221954.hBMJs1M18648@c-24-5-183-134.client.comcast.net> <16359.20178.766911.979542@magrathea.basistech.com> <200312222013.hBMKDt118815@c-24-5-183-134.client.comcast.net> <2msmjccw92.fsf@starship.python.net> <200312222245.hBMMj4W19537@c-24-5-183-134.client.comcast.net> Message-ID: <16359.33553.818193.723995@magrathea.basistech.com> Guido van Rossum writes: > > > Perhaps it's easy to overlook that it's using Py_GETENV() rather than > > > getenv(); this macro takes tests for Py_IgnoreEnvironmentFlag first. > > > > Py_IgnoreEnvironmentFlag and the -E command line option are both > > /fairly/ new... 2.2 era? > > Yeah, but the patch he suggested was using Py_GETENV(), so it's not > like it didn't exist when they did the patch; they must have missed > its significance. :-) Yes, *they* did miss the significance of the use of the macro. -- Tom Emerson Basis Technology Corp. Software Architect http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever" From martin at v.loewis.de Mon Dec 22 15:01:28 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 22 20:46:49 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <2mptegej2x.fsf@starship.python.net> References: <2mptegej2x.fsf@starship.python.net> Message-ID: <3FE74D98.1020307@v.loewis.de> Michael Hudson wrote: > As Martin regularly points out, we have rather too much of this kind > of thing. Given that we demand ANSI C, we could surely lose > HAVE_LIMITS_H, and probably much other cruft. Indeed, and I would encourage contributions in that direction. Make sure you put, in the commit message, an elaboration why it is safe to delete the code in question. Regards, Martin From martin at v.loewis.de Mon Dec 22 15:39:20 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 22 21:19:40 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <200312222003.hBMK3eH18732@c-24-5-183-134.client.comcast.net> References: <1xqwodnr.fsf@python.net> <200312221933.hBMJXTI18559@c-24-5-183-134.client.comcast.net> <200312222003.hBMK3eH18732@c-24-5-183-134.client.comcast.net> Message-ID: <3FE75678.6090508@v.loewis.de> Guido van Rossum wrote: > Not quite -- I don't understand why *you* don't want the environment > variables to be used, if all you want is a better pythonw.exe. I haven't embedded Python for a while, but I can sympathize with the desire to ignore environment variables. One approach to embedding is to provide a complete (stripped) copy of Python, inside the target directory of the application. You might even directly link with a static python23.lib, and freeze all modules, instead of linking with python23.dll, and providing a copy of all .py files. In such an environment, you don't want a regular Python installation to interfere with your embedded environment. For example, you may have incorporated the library Foo, but the target system has another (older) copy of Foo in PYTHONPATH. If your embedded application now picks up the older copy of Foo, it fails, and all your testing goes to hell (*) since arbitrary pieces of the library might get arbitrarily replaced with something else. So to allow reproducable results, you need a guarantee that nothing on the target system is incorporated in your application. Regards, Martin (*) DLL hell, to be precise. From pinard at iro.umontreal.ca Mon Dec 22 22:20:05 2003 From: pinard at iro.umontreal.ca (=?iso-8859-1?Q?Fran=E7ois?= Pinard) Date: Mon Dec 22 22:22:34 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <3FE75678.6090508@v.loewis.de> References: <1xqwodnr.fsf@python.net> <200312221933.hBMJXTI18559@c-24-5-183-134.client.comcast.net> <200312222003.hBMK3eH18732@c-24-5-183-134.client.comcast.net> <3FE75678.6090508@v.loewis.de> Message-ID: <20031223032005.GA16709@titan.progiciels-bpi.ca> [Martin von L?wis] > So to allow reproducable results, you need a guarantee that nothing > on the target system is incorporated in your application. I did not closely follow this thread, and apologise if I repeat arguments which have already been said. While I quite understand the argument and needs which Martin expresses, there are situations where one is happy to tune the behaviour of the embedded Python through the usual environment variables. It might depend if the embedding application intends to either expose or hide the existence of Python. If the application offers Python as an extension language for the system implemented by the application, Python is then exposed, and the user should be able to use environment variable to his/her advantage. If the application uses Python internally, all to itself, then the application likely wants to hide even the fact that it uses Python, exerts full control, or at least decide exactly how much or little the user is allowed to interfere, as Martin explained. If it is given that only one of these avenues is favoured, I would for one prefer that the "exposed" direction be the easier to implement, which may be the statu quo over the current situation. The "hiding" direction might be evaluated so it requires a bit more work for the implementor, about deleting or forcing the values of a few sensitive environment variables. Especially given that a Python hiding application is likely to control other environment variables as well (like maybe those related to loader path search, or localisation). -- Fran?ois Pinard http://www.iro.umontreal.ca/~pinard From lists at hlabs.spb.ru Tue Dec 23 06:05:37 2003 From: lists at hlabs.spb.ru (Dmitry Vasiliev) Date: Tue Dec 23 03:05:34 2003 Subject: [Python-Dev] Importing packages from command line In-Reply-To: <3FE6CC59.10100@iinet.net.au> References: <1071839460.3fe2f8e4dbbad@mcherm.com> <3FE6D94A.2060904@hlabs.spb.ru> <3FE6CC59.10100@iinet.net.au> Message-ID: <3FE82181.50309@hlabs.spb.ru> Nick Coghlan wrote: > Dmitry Vasiliev wrote: > >> The main idea is to treating package as a program and run package >> initialization code from command line. The advantage is zipping all >> program modules in one zip archive and running the program from >> command line without unzipping it, like Java's jar's. But this idea >> need more thoughts however... > > python runzip.py mypkg.zip > > where runzip.py is a pure Python script that works as you suggest. > > I'm fairly sure zipimport would allow you to write the program you > describe in pure Python. I've obviously never written such a program, > though. Code is fairly simple. import sys sys.path.insert(0, "module.zip") import module > It's certainly an interesting packaging idea for small Python programs > with several files, but for which a full-blown installer would be overkill. -- Dmitry Vasiliev (dima at hlabs.spb.ru) From lists at hlabs.spb.ru Tue Dec 23 06:09:27 2003 From: lists at hlabs.spb.ru (Dmitry Vasiliev) Date: Tue Dec 23 03:09:19 2003 Subject: [Python-Dev] Importing packages from command line In-Reply-To: <200312221807.14352.aleaxit@yahoo.com> References: <1071839460.3fe2f8e4dbbad@mcherm.com> <3FE6D94A.2060904@hlabs.spb.ru> <200312221807.14352.aleaxit@yahoo.com> Message-ID: <3FE82267.3030607@hlabs.spb.ru> Alex Martelli wrote: >>The main idea is to treating package as a program and run package >>initialization code from command line. The advantage is zipping all >>program modules in one zip archive and running the program from command >>line without unzipping it, like Java's jar's. But this idea need more >>thoughts however... > > Couldn't you use: > python -c "import package_name" > for this purpose even today? python -c "import sys; sys.path.insert(0, 'module.zip'); import module" Seems ugly... :) -- Dmitry Vasiliev (dima at hlabs.spb.ru) From aleaxit at yahoo.com Tue Dec 23 04:05:41 2003 From: aleaxit at yahoo.com (Alex Martelli) Date: Tue Dec 23 04:05:51 2003 Subject: [Python-Dev] Importing packages from command line In-Reply-To: <3FE82267.3030607@hlabs.spb.ru> References: <1071839460.3fe2f8e4dbbad@mcherm.com> <200312221807.14352.aleaxit@yahoo.com> <3FE82267.3030607@hlabs.spb.ru> Message-ID: <200312231005.41130.aleaxit@yahoo.com> On Tuesday 23 December 2003 12:09 pm, Dmitry Vasiliev wrote: > Alex Martelli wrote: > >>The main idea is to treating package as a program and run package > >>initialization code from command line. The advantage is zipping all > >>program modules in one zip archive and running the program from command > >>line without unzipping it, like Java's jar's. But this idea need more > >>thoughts however... > > > > Couldn't you use: > > python -c "import package_name" > > for this purpose even today? > > python -c "import sys; sys.path.insert(0, 'module.zip'); import module" > > Seems ugly... :) Well, if the zipfile isn't on your sys.path already, then you'd have to insert it explicitly anyway -- surely, even if a switch "-p" to mean import existed, you wouldn't want it to force python to snoop into EVERY zipfile around?! PYTHONPATH=module.zip python -c "import module" is one way you might express "insert into path and import" using a decent shell (cygwin's bash on Windows, for example). The proposed: PYTHONPATH=module.zip python -p module doesn't appear to offer a _major_ enhancement, just a very minor one, in my personal opinion. One potential advantage of -p might be to let it be present _together_ with one -c (even better might be to allow multiple -c, actually). If you want to run a "while foo.bep(): foo.zlup()" you can do it today only with a shell that allows easy input of one multiline argument. But again it seems a rather marginal issue to me, personally. Alex From ncoghlan at iinet.net.au Tue Dec 23 05:03:55 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Tue Dec 23 05:04:00 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: <200312222213.hBMMD2m19449@c-24-5-183-134.client.comcast.net> References: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.net> <1072131005.13676.9.camel@anthem> <200312222213.hBMMD2m19449@c-24-5-183-134.client.comcast.net> Message-ID: <3FE8130B.1080601@iinet.net.au> [Barry] >>Would it be better if object.__hash__() raised a NotImplementedError? [Guido] > It can't -- it's type.__hash__(object). If I understand Barry's suggestion correctly, he means to keep object.__hash__, but have it raise a specific, meaningful error instead of making a potentially incorrect assumption as it does now. E.g. (not real code, since this would behave differently in Py 2.3.3) >>> class Works(object): pass >>> class Breaks(object): ... def __cmp__(): return 0 # All instances are created equal! >>> work = Works() >>> break = Breaks() >>> hash(work) 45632543 >>> hash(break) Traceback (most recent call last): NotImplementedError: Must explicitly define __hash__ for non-default comparisons hash(work) is fine, since there is no __cmp__ or __eq__ override in Works, and hence object.__hash__ never gets called. hash(break) raises the exception because of the existence of the (rather useless) __cmp__ function in the Breaks class. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From martin at v.loewis.de Tue Dec 23 04:33:53 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 23 05:13:45 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312222156.hBMLueA19333@c-24-5-183-134.client.comcast.net> References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <200312221907.hBMJ7hg18485@c-24-5-183-134.client.comcast.net> <200312221959.hBMJxhY18697@c-24-5-183-134.client.comcast.net> <200312222156.hBMLueA19333@c-24-5-183-134.client.comcast.net> Message-ID: <3FE80C01.9080203@v.loewis.de> Guido van Rossum wrote: > I don't have the OpenBSD strchr.c source code online here > so I'll stop speculating here... It's this: http://www.openbsd.org/cgi-bin/cvsweb/src/lib/libc/string/index.c I can't see anything wrong in it, and it hasn't significantly changed in ages, either. I'd be curious what p_name points to when the process crashes. (gdb) p *p_name Regards, Martin From martin at v.loewis.de Tue Dec 23 03:51:01 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 23 05:48:17 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <20031222221058.GD24252@unpythonic.net> References: <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> <20031222221058.GD24252@unpythonic.net> Message-ID: <3FE801F5.9010304@v.loewis.de> Jeff Epler wrote: > You could > submit an interpretation request, but I think the committee would > concur with my reading. Hmm. This is what I now did: I submitted an interpretation request to PASC: Edition of Specification (Year): 2003 Defect code : 3. Clarification required It is unclear whether calling system(3) invokes atfork handlers in a conforming implementation. system(3) specifies "The environment of the executed command shall be as if a child process were created using fork(), and the child process invoked the sh utility using execl() as follows:" In particular, usage of the word "environment" is confusing here. It may refer just to environment variables, however, the "Application usage" sections indicates that also signal handlers should be arranged as if the process was created through fork() and execl(). This still makes not clear whether handlers installed through pthread_atfork() are invoked. Action: The description of system() should change to "system() behaves as if a new process was created using fork(), and the child process invoked the sh utility using execl() ..." In addition, the application usage section should make it clear that atfork handlers are invoked. Regards, Martin From martin at v.loewis.de Tue Dec 23 04:01:58 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 23 08:43:12 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> References: <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> <20031222221058.GD24252@unpythonic.net> <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> Message-ID: <3FE80486.9060807@v.loewis.de> Guido van Rossum wrote: > How hard would it be to reimplement our own system() and popen() using > only POSIX calls, for POSIX systems? I've always thought of these to > be pretty simple combinations of fork() and exec(), with an assumption > of a working /bin/sh. I would be concerned that we bypass magic that the system vendor put into system(), which is essential for proper operation. For example, on Linux, system() blocks SIGINT in the parent process while the child is running. Also, the shell executable that system() uses may not be /bin/sh. OTOH, using the same underlying implementation on all systems makes Python behave more predictable. In the specific case, we would not even need pthread_atfork anymore, as we now could invoke PyOS_AfterFork in the child ourselves. Regards, Martin From jepler at unpythonic.net Tue Dec 23 09:05:48 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Tue Dec 23 09:06:02 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <3FE80486.9060807@v.loewis.de> References: <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> <20031222221058.GD24252@unpythonic.net> <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> <3FE80486.9060807@v.loewis.de> Message-ID: <20031223140547.GA7907@unpythonic.net> On Tue, Dec 23, 2003 at 10:01:58AM +0100, Martin v. Loewis wrote: > I would be concerned that we bypass magic that the system vendor put > into system(), which is essential for proper operation. For example, > on Linux, system() blocks SIGINT in the parent process while the > child is running. Also, the shell executable that system() uses > may not be /bin/sh. This behavior (blocking SIGINT and SIGQUIT in the parent) is part of the specification of system(): The system() function shall ignore the SIGINT and SIGQUIT signals, and shall block the SIGCHLD signal, while waiting for the command to terminate. If this might cause the application to miss a signal that would have killed it, then the application should examine the return value from system() and take whatever action is appropriate to the application if the command terminated due to receipt of a signal. Jeff From tim.one at comcast.net Tue Dec 23 09:20:49 2003 From: tim.one at comcast.net (Tim Peters) Date: Tue Dec 23 09:20:53 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <3FE80C01.9080203@v.loewis.de> Message-ID: [Guido] >> I don't have the OpenBSD strchr.c source code online here >> so I'll stop speculating here... [Martin v. Loewis] > It's this: > > http://www.openbsd.org/cgi-bin/cvsweb/src/lib/libc/string/index.c > > I can't see anything wrong in it, and it hasn't significantly > changed in ages, either. Looks fine to me too -- and it's a very simple function. Google didn't turn up any suggestion of strchr problems under OpenBSD either. Hate to say it, but the pointer passed *to* strchr must be insane, and that makes it more likely a Python, or platform compiler, bug. > I'd be curious what p_name points to when the process crashes. > > (gdb) p *p_name From kbk at shore.net Tue Dec 23 10:10:54 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Tue Dec 23 10:11:00 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: (Tim Peters's message of "Tue, 23 Dec 2003 09:20:49 -0500") References: Message-ID: "Tim Peters" writes: > Hate to say it, but the pointer passed *to* strchr must be insane, > and that makes it more likely a Python, or platform compiler, bug. There are two calls to load_next() in import_module_ex(). The segfault is occuring during the second call. The code is somewhat pathological in that the callee, load_next(), is modifying the caller's /parameters/ by changing the contents of name. For some reason, the compiler emits code which makes a copy of import_module_ex()'s parameters in the stack frame. When load_next() is called, the reference &name is the location in the parameter area of the frame, but when name is tested in the while loop, the copy in the local area of the frame is used. Since this has not been modified by load_next(), the fact that name has been set to 0x00 is missed. load_next() gets called erroneously and passes a null pointer to strchr. I tried a volatile declaration, but no joy. Adding a proper local, mod_name, resolved the problem. -- KBK Index: Python/import.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Python/import.c,v retrieving revision 2.225 diff -c -r2.225 import.c *** Python/import.c 20 Nov 2003 01:44:58 -0000 2.225 --- Python/import.c 23 Dec 2003 14:56:40 -0000 *************** *** 1871,1876 **** --- 1871,1877 ---- PyObject *fromlist) { char buf[MAXPATHLEN+1]; + char *mod_name; int buflen = 0; PyObject *parent, *head, *next, *tail; *************** *** 1878,1891 **** if (parent == NULL) return NULL; ! head = load_next(parent, Py_None, &name, buf, &buflen); if (head == NULL) return NULL; tail = head; Py_INCREF(tail); ! while (name) { ! next = load_next(tail, tail, &name, buf, &buflen); Py_DECREF(tail); if (next == NULL) { Py_DECREF(head); --- 1879,1893 ---- if (parent == NULL) return NULL; ! mod_name = name; ! head = load_next(parent, Py_None, &mod_name, buf, &buflen); if (head == NULL) return NULL; tail = head; Py_INCREF(tail); ! while (mod_name) { ! next = load_next(tail, tail, &mod_name, buf, &buflen); Py_DECREF(tail); if (next == NULL) { Py_DECREF(head); From charleshixsn at earthlink.net Tue Dec 23 10:54:46 2003 From: charleshixsn at earthlink.net (Charles Hixson) Date: Tue Dec 23 11:00:33 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> Message-ID: <3FE86546.50003@earthlink.net> Guido van Rossum wrote: >> I like syntax that reads most important left-to-right, so what about >> from MODULE import NAMES as RENAME searching HOW >> or >> import NAMES as RENAME from MODULE searching HOW >> >> searching could be 'absolute' 'relative' or any of the other >> suggested words. If you wanted to get fancy it could be a list, if >> it isn't a list people who truly care could cascade try/except on >> ImportError. >> >> -jackdied >> >> ps, I like the second version better but it is less like the current >> version, >> which is a drawback. >> > > > This got several encouraging responses, so I have to consider it. > > Unfortunately it looks like 'searching' would have to become a > reserved word; if we use the same trick we used to avoid making 'as' a > reserved word, that would be too confusing to decode in the code > generator. The HOW could be just an identifier then. > > I'm not sure how well it'll read in actual examples. Let's try. > Adapting an example from Opal that was used here before... > > from COGS import generate searching relative > > To me, that looks a lot like someone left the commas out of > > from COGS import generate, searching, relative > > In general I'm not a big fan of "wordy" clauses like this -- they work > better in languages like SQL or COBOL because those are case-sensitive > so you can use a difference in capitalization there; there the former > would be > > FROM COGS IMPORT generate SEARCHING RELATIVE > > (using imaginary syntax) while the latter would be > > FROM COGS IMPORT generate, searching, relative > > while in Python the long sequence of lowercase words becomes a blur. > > So if the two alternatives are simply 'searching absolute' and > 'searching relative', with the default being 'searching absolute' in > Python 3.0 (but the current ambiguous search in Python 2.x) I'd still > prefer making the distinction with "from ...MODULE" vs. "from MODULE" > rather than with a searching clause. And I still prefer the > single-dot syntax over three dots, because it can be used to spell > importing from the parent or grandparent package explicitly. > > A separate suggestion is to switch from "from X import Y" to "import Y > from X". This seems a gratuitous change even if it reads or edits > marginally better. Now's not the time to tinker with marginal stuff > -- that should've been done 10 years ago. > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/charleshixsn%40earthlink.net > > > > Not necessarily. import could be a special function that doesn't need parentheses around it's arguments or comma separators. That could be syntax sugar for: import (names, namedAs, fromModule, searchmethod) or import (names =[theNames], namedAd=[localNames], fromModule=fromModule, searchMethod=upFromCurrent) OTOH, to me it looks more like Smalltalk than like Python. An explicit import function without special sugar looks more Pythonic. (But that would break backward compatibility.) Perhaps the current method could be deprecated, and an import function be the replacement? From guido at python.org Tue Dec 23 11:11:40 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 23 11:11:49 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: Your message of "Tue, 23 Dec 2003 10:01:58 +0100." <3FE80486.9060807@v.loewis.de> References: <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> <20031222221058.GD24252@unpythonic.net> <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> <3FE80486.9060807@v.loewis.de> Message-ID: <200312231611.hBNGBek20630@c-24-5-183-134.client.comcast.net> > > How hard would it be to reimplement our own system() and popen() using > > only POSIX calls, for POSIX systems? I've always thought of these to > > be pretty simple combinations of fork() and exec(), with an assumption > > of a working /bin/sh. > > I would be concerned that we bypass magic that the system vendor put > into system(), which is essential for proper operation. For example, > on Linux, system() blocks SIGINT in the parent process while the > child is running. Also, the shell executable that system() uses > may not be /bin/sh. In practice, I think we can do as well as vendors -- there really isn't that much to it. Systems where /bin/sh doesn't exist will have other problems... And we get to do it right when called from a thread. > OTOH, using the same underlying implementation on all systems makes > Python behave more predictable. > > In the specific case, we would not even need pthread_atfork anymore, > as we now could invoke PyOS_AfterFork in the child ourselves. Right! --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Dec 23 11:20:08 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 23 11:20:14 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Your message of "Tue, 23 Dec 2003 10:10:54 EST." References: Message-ID: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> > There are two calls to load_next() in import_module_ex(). The > segfault is occuring during the second call. > > The code is somewhat pathological in that the callee, load_next(), is > modifying the caller's /parameters/ by changing the contents of name. > > For some reason, the compiler emits code which makes a copy of > import_module_ex()'s parameters in the stack frame. When load_next() > is called, the reference &name is the location in the > parameter area of the frame, but when name is tested in the while > loop, the copy in the local area of the frame is used. Since this has > not been modified by load_next(), the fact that name has been set to > 0x00 is missed. load_next() gets called erroneously and passes a null > pointer to strchr. > > I tried a volatile declaration, but no joy. Adding a proper local, > mod_name, resolved the problem. Wow. Thanks for the analysis. But this is clearly a compiler bug. Where do we report that? And why would it be unique to OpenBSD? In the mean time, Kurt, please check in your fix -- it can't hurt, and we might as well avoid the pain for the next person who wants to build a debugging Python. The fix could use a comment referring to a compiler bug, to keep the next maintainer from unfixing it. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Tue Dec 23 11:47:02 2003 From: barry at python.org (Barry Warsaw) Date: Tue Dec 23 11:47:09 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: <3FE8130B.1080601@iinet.net.au> References: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.net> <1072131005.13676.9.camel@anthem> <200312222213.hBMMD2m19449@c-24-5-183-134.client.comcast.net> <3FE8130B.1080601@iinet.net.au> Message-ID: <1072198021.31242.8.camel@geddy> On Tue, 2003-12-23 at 05:03, Nick Coghlan wrote: > If I understand Barry's suggestion correctly, he means to keep > object.__hash__, but have it raise a specific, meaningful error instead > of making a potentially incorrect assumption as it does now. Right! Thanks Nick. -Barry From klm at zope.com Tue Dec 23 12:03:00 2003 From: klm at zope.com (Ken Manheimer) Date: Tue Dec 23 12:11:50 2003 Subject: [Python-Dev] Relative import In-Reply-To: Message-ID: On Mon, 22 Dec 2003, Devin wrote: > The first part stems from Guido's idea about using a '.' at the beginning > of a relative import. It seemed a bit weird at first, but I eventually > became very comfortable with the idea after thinking about it in respect > to absolute imports. I'll explain what I mean. > > If you currently want to import a module 'a.b.c.d', you write: > > import a.b.c.d > > Now, if you're in the module 'a.b', you could still write: > > import a.b.c.d > > ... but the 'a.b' part is redundant because you're already in the module > 'a.b'. If you take away 'a.b', then you end up with: > > import .c.d I also look at it this way, and find it both intuitive and mnemonic. My model is slighly different than yours, though, in a way that simplifies the expression for going up the relative chain. I'll explain... > This flowed well in my head, and looks natural. However, the suggestion > about '.' or '..' referring to the parent looks _ugly and unnatural_. > > Let's say '.' was used to spell "parent module", and let's assume that > the module 'a.b.c.d' wants to import 'a.b.e'. The absolute import would > be spelled: > > import a.b.e > > ... and the relative import would be spelled: > > import .....e # .[parent].[parent].e > > Yuck! (no offense :) Truly, yuck. But in my model, the leading '.' dot, itself, stands for the containing package, and '..' stands for the containing package's package, and so forth: import ..e # [parent-of-parent].e I don't think it's yucky at all, this way. (I also don't think it's un-pythonic - python uses punctuation all over to denote type literals ('{'/'}'), cues which delineate statement suites (':'), and so forth.) In fact, i think it's a lot more mnemonic and encompassing than the alternatives that use words (though i haven't looked at all the alternatives very carefully yet). Even using the leading '.' syntax, it would be good to have a specific identifier for the containing package, realized in each module and respected as a special identifier by the package machinery. I like '__pkg__': import __pkg__.sibling # == 'import .sibling' and import __pkg__.__pkg__.uncle # == 'import ..uncle' import __pkg__.__pkg__.uncle.cousin # == 'import ..uncle.cousin' I think the leading '.' dot forms are a lot nicer, but the presence of a module named '__pkg__' would be useful for programmatically mucking navigating the package structure. Ken Manheimer klm@zope.com From mwh at python.net Tue Dec 23 12:24:27 2003 From: mwh at python.net (Michael Hudson) Date: Tue Dec 23 12:24:31 2003 Subject: [Python-Dev] Relative import In-Reply-To: <3FE86546.50003@earthlink.net> (Charles Hixson's message of "Tue, 23 Dec 2003 07:54:46 -0800") References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <3FE86546.50003@earthlink.net> Message-ID: <2mad5jcur8.fsf@starship.python.net> Charles Hixson writes: > Not necessarily. import could be a special function that doesn't need > parentheses around it's arguments or comma separators. That could be > syntax sugar for: > import (names, namedAs, fromModule, searchmethod) or > import (names =[theNames], namedAd=[localNames], > fromModule=fromModule, searchMethod=upFromCurrent) > > OTOH, to me it looks more like Smalltalk than like Python. An > explicit import function without special sugar looks more Pythonic. > (But that would break backward compatibility.) > Perhaps the current method could be deprecated, and an import function > be the replacement? I really don't think you could make that work (unless you want to write math = import("math") everywhere, which you can almost do today...) Cheers, mwh -- Ya, ya, ya, except ... if I were built out of KSR chips, I'd be running at 25 or 50 MHz, and would be wrong about ALMOST EVERYTHING almost ALL THE TIME just due to being a computer! -- Tim Peters, 30 Apr 97 From devin at whitebread.org Tue Dec 23 13:45:18 2003 From: devin at whitebread.org (Devin) Date: Tue Dec 23 13:50:12 2003 Subject: [Python-Dev] Relative import In-Reply-To: Message-ID: On Tue, 23 Dec 2003, Ken Manheimer wrote: > > This flowed well in my head, and looks natural. However, the suggestion > > about '.' or '..' referring to the parent looks _ugly and unnatural_. > > > > Let's say '.' was used to spell "parent module", and let's assume that > > the module 'a.b.c.d' wants to import 'a.b.e'. The absolute import would > > be spelled: > > > > import a.b.e > > > > ... and the relative import would be spelled: > > > > import .....e # .[parent].[parent].e > > > > Yuck! (no offense :) > > Truly, yuck. But in my model, the leading '.' dot, itself, stands for > the containing package, and '..' stands for the containing package's > package, and so forth: > > import ..e # [parent-of-parent].e I can't quite get used to the idea of '.' having two different meanings depending on how many '.'s are found next to each other in an import statement. I believe that the notation above _is_ concise, but _is not_ very intuitive ... > Even using the leading '.' syntax, it would be good to have a specific > identifier for the containing package, realized in each module and > respected as a special identifier by the package machinery. I like > '__pkg__': > > import __pkg__.sibling # == 'import .sibling' > > and > > import __pkg__.__pkg__.uncle # == 'import ..uncle' > import __pkg__.__pkg__.uncle.cousin # == 'import ..uncle.cousin' ... while the notation here _is_ intuitive (to a python programmer), but _is not_ concise. I still favor this syntax to the former syntax. I like '__pkg__' better than my initial suggestion ('__parent__'). It's more aesthetically pleasing. :) -- Devin devin@whitebread.org http://www.whitebread.org/ From chrisandannreedy at comcast.net Tue Dec 23 15:03:37 2003 From: chrisandannreedy at comcast.net (Chris and Ann Reedy) Date: Tue Dec 23 15:03:43 2003 Subject: [Python-Dev] Another Strategy for Relative Import Message-ID: <3FE89F99.8BC0D067@comcast.net> Ken Manheimer wrote: > > Even using the leading '.' syntax, it would be good to have a specific > identifier for the containing package, realized in each module and > respected as a special identifier by the package machinery. I like > '__pkg__': > > import __pkg__.sibling # == 'import .sibling' > > and > > import __pkg__.__pkg__.uncle # == 'import ..uncle' > import __pkg__.__pkg__.uncle.cousin # == 'import ..uncle.cousin' > > I think the leading '.' dot forms are a lot nicer, but the presence of > a module named '__pkg__' would be useful for programmatically mucking > navigating the package structure. Delurking: Knowing that the statement: import foo.bar has the effect of setting the variable foo to reference package foo, even if the variable foo already references something else, I was wondering if it would make sense to changes the semantics of import so that if the variable foo is already assigned and is a package (or maybe a module in some cases), then importing foo.bar would have the effect of importing the module or package bar within foo, using foo.__path__ to search for the module bar and foo.__name__ + '.bar' as the name of the module. It seems to me that such an approach might be used in a number of cases: 1. If there is a standard name for the parent of a package, e.g. __pkg__, then import__pkg__.bar just works. 2. This approach allows the following kind of thing to work: if : import xyz as util else: import xyz_old as util import util.bar 3. This could be used to clean up the problem with os.path. The idea here is that since initialization of os creates the attribute os.path which references a module, then: import os.path would have the effect of importing os and then noting that os.path exists and is a module, so no further work is required. I haven't (yet?) attempted to work out whether this could be used to handle all the use cases for relative import. Chris P.S. I've been lurking on this list for a about six months. I just moved between coasts (US) and things are still a mess. I would like to get more involved with Python. I'll send a message in the near future introducing myself. From barry at barrys-emacs.org Wed Dec 24 12:37:06 2003 From: barry at barrys-emacs.org (Barry Scott) Date: Wed Dec 24 12:37:15 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.ne t> References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> Message-ID: <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> At 23-12-2003 16:20, Guido van Rossum wrote: >Wow. Thanks for the analysis. But this is clearly a compiler bug. >Where do we report that? And why would it be unique to OpenBSD? Because the OpenBSD folks have their own code generator to improve the security of OpenBSD. They code gen to make it impossible/improbable to use a stack overflow attack. I found they fix the compiler quickly once you tell then on the OpenBSD dev list whats wrong with a recipe to reproduce. You may find that they will not fix for 3.3 as 3.4 is out and the tool chain moved from a.out to ELF in 3.4. Barry From guido at python.org Wed Dec 24 12:47:22 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 24 12:47:27 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Your message of "Wed, 24 Dec 2003 17:37:06 GMT." <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> Message-ID: <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> > >Wow. Thanks for the analysis. But this is clearly a compiler bug. > >Where do we report that? And why would it be unique to OpenBSD? > > Because the OpenBSD folks have their own code generator to improve the > security of OpenBSD. They code gen to make it impossible/improbable > to use a stack overflow attack. I found they fix the compiler quickly > once you tell then on the OpenBSD dev list whats wrong with a recipe to > reproduce. Sigh. So much for security. :-( > You may find that they will not fix for 3.3 as 3.4 is out and the > tool chain moved from a.out to ELF in 3.4. Double sigh. Strange approach to security. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Dec 24 13:12:39 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 24 13:12:45 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Your message of "Wed, 24 Dec 2003 09:47:22 PST." <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> Message-ID: <200312241812.hBOICdK22045@c-24-5-183-134.client.comcast.net> I've tried to look for a way to report OpenBSD bugs, but it appears the only way to report one is by using their sendbug command, which assumes you've got OpenBSD installed. Would someone here be so kind as to report the bug we've found? A pointer to Kurt's last message in this thread should be a good piece of evidence. --Guido van Rossum (home page: http://www.python.org/~guido/) From kbk at shore.net Wed Dec 24 15:22:07 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Wed Dec 24 15:22:13 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312241812.hBOICdK22045@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Wed, 24 Dec 2003 10:12:39 -0800") References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> <200312241812.hBOICdK22045@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: > I've tried to look for a way to report OpenBSD bugs, but it appears > the only way to report one is by using their sendbug command, which > assumes you've got OpenBSD installed. Would someone here be so kind > as to report the bug we've found? A pointer to Kurt's last message in > this thread should be a good piece of evidence. I'm not ignoring you, shore.net is delaying inbound messages over 24h and that's slowing me down. The problem is related to the ProPolice "stack smash" protector. It does exactly what I saw: It copies any pointer arguments to an area in the frame below the locally declared buffers. The error appears to be in referencing the original location in the arguments when making the call to load_next(). One of the ProPolice ideas is that a pointer (especially a pointer to a function) can be hacked by a buffer overrun to point to rogue code. www.trl.ibm.com/projects/security/ssp/ The implementation appears to be copying all argument pointers for good measure. I'm looking into this to see if there is a better way to fix the import code. Making a local copy of the pointer fixes the segfault but it's not clear to me yet whether that's a solid fix for all uses of this code in Python. As far as reporting the bug, I can do that once I localize the problem in gcc and see whether it's been fixed. A quick search of the OpenBSD bug database came up dry, but maybe I don't know how to phrase the question yet. -- KBK From kbk at shore.net Wed Dec 24 15:45:16 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Wed Dec 24 15:45:29 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Wed, 24 Dec 2003 09:47:22 -0800") References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: > Double sigh. Strange approach to security. Their overall approach is outstanding. They understand that the right thing to do is fix insecure code and they audit their base distro extensively. I think they have around a dozen people, including some security professionals, on the audit team. However, it's not possible to audit all the code in the ports, so they have implemented several defensive measures which make it more difficult to mount an attack: 1. ProPolice - anti-stacksmash methods. Sets a canary after the return address, moves local and argument buffers just below that, moves other locals below the buffers, and copies argument pointers into the latter area. If a buffer gets overrun, it nukes the canary. 2. W^X - memory which can be written by an app can't be executed, and vice-versa, assuming MMU support. 3. Non-executable stack @ 3.2 (note that this can cause problems with some functional languages) 4. Greatly reduced use of SUID/GUID binaries. Use of chroot and priviledge separation. It's not surprising there are some residual bugs in the implementation, at least at 3.3. I haven't moved my box to 3.4 yet. -- KBK From andymac at bullseye.apana.org.au Thu Dec 25 08:40:35 2003 From: andymac at bullseye.apana.org.au (Andrew MacIntyre) Date: Thu Dec 25 08:48:31 2003 Subject: [Python-Dev] status of 2.3 branch for maintenance checkins Message-ID: <20031226003609.O1176@bullseye.apana.org.au> I don't recall any advice as to whether the 2.3 branch is again open for checkins post 2.3 - Is Jack still sorting out the Mac release? Regards, Andrew. -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From anthony at interlink.com.au Thu Dec 25 12:47:43 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Dec 25 12:48:04 2003 Subject: [Python-Dev] status of 2.3 branch for maintenance checkins In-Reply-To: <20031226003609.O1176@bullseye.apana.org.au> Message-ID: <200312251747.hBPHlham032180@localhost.localdomain> >>> Andrew MacIntyre wrote > I don't recall any advice as to whether the 2.3 branch is again open for > checkins post 2.3 - Is Jack still sorting out the Mac release? It's still closed, until Jack says otherwise. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From jeremy at alum.mit.edu Fri Dec 26 14:06:31 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Fri Dec 26 14:10:04 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects listobject.c,2.172,2.173 In-Reply-To: References: Message-ID: <1072465591.31901.71.camel@localhost.localdomain> On Fri, 2003-12-26 at 14:05, jhylton@users.sourceforge.net wrote: > Update of /cvsroot/python/python/dist/src/Objects > In directory sc8-pr-cvs1:/tmp/cvs-serv32521 > > Modified Files: > listobject.c > Log Message: > Revert previous two checkins to repair test failure. > > The special-case code that was removed could return a value indicating > success but leave an exception set. test_fileinput failed in a debug > build as a result. If there's some value to the optimization I just removed, let's discuss it on python-dev. I could have fixed the new code, but it seems a lot of hairy new code with magic constants that only covered a few corner cases. The checkin message pointed to a python-list discussion that didn't have much real evidence in it. Jeremy From jeremy at alum.mit.edu Fri Dec 26 15:22:52 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Fri Dec 26 15:26:33 2003 Subject: [Python-Dev] test_unicode_file fails on Linux Message-ID: <1072470171.31901.74.camel@localhost.localdomain> The utime() call is failing for one of the Unicode file names. build> ./python ../Lib/test/test_unicode_file.py test_directories (__main__.TestUnicodeFiles) ... ok test_equivalent_files (__main__.TestUnicodeFiles) ... ok test_single_files (__main__.TestUnicodeFiles) ... '@test-\xc3\xa0\xc3\xb2' '@test-\xc3\xa0\xc3\xb2' u'@test-\xe0\xf2' ERROR ====================================================================== ERROR: test_single_files (__main__.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "../Lib/test/test_unicode_file.py", line 143, in test_single_files self._test_single(TESTFN_UNICODE) File "../Lib/test/test_unicode_file.py", line 117, in _test_single self._do_single(filename) File "../Lib/test/test_unicode_file.py", line 33, in _do_single os.utime(filename, None) UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-7: ordinal not in range(128) I'm running on a RH9 system. I see that the code hasn't changed since the beginning of the month, so I'm surprised to only discover it now. I can't recall the last time I did a full test run, so I'm not sure how long it has been failing. Jeremy From skip at pobox.com Fri Dec 26 16:17:22 2003 From: skip at pobox.com (Skip Montanaro) Date: Fri Dec 26 16:17:31 2003 Subject: [Python-Dev] test_unicode_file fails on Linux In-Reply-To: <1072470171.31901.74.camel@localhost.localdomain> References: <1072470171.31901.74.camel@localhost.localdomain> Message-ID: <16364.42338.127559.952496@montanaro.dyndns.org> Jeremy> The utime() call is failing for one of the Unicode file names. Been failing for me for awhile (Mac OS X), but in a bit different fashion: % ./python.exe ../Lib/test/regrtest.py -v test_unicode_file test_unicode_file test_directories (test.test_unicode_file.TestUnicodeFiles) ... FAIL test_equivalent_files (test.test_unicode_file.TestUnicodeFiles) ... ok test_single_files (test.test_unicode_file.TestUnicodeFiles) ... FAIL ====================================================================== FAIL: test_directories (test.test_unicode_file.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/skip/src/python/head/dist/src/Lib/test/test_unicode_file.py", line 155, in test_directories self._do_directory(TESTFN_ENCODED+ext, TESTFN_ENCODED+ext, os.getcwd) File "/Users/skip/src/python/head/dist/src/Lib/test/test_unicode_file.py", line 103, in _do_directory make_name) AssertionError: '@test-a\xcc\x80o\xcc\x80.dir' != '@test-\xc3\xa0\xc3\xb2.dir' ====================================================================== FAIL: test_single_files (test.test_unicode_file.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/skip/src/python/head/dist/src/Lib/test/test_unicode_file.py", line 141, in test_single_files self._test_single(TESTFN_ENCODED) File "/Users/skip/src/python/head/dist/src/Lib/test/test_unicode_file.py", line 116, in _test_single self._do_single(filename) File "/Users/skip/src/python/head/dist/src/Lib/test/test_unicode_file.py", line 41, in _do_single self.failUnless(base in os.listdir(path)) AssertionError ---------------------------------------------------------------------- Ran 3 tests in 0.031s FAILED (failures=2) test test_unicode_file failed -- errors occurred; run in verbose mode for details 1 test failed: test_unicode_file I thought this was a known issue (Unicode normalization differences?), so didn't bother to report it. Skip From Jack.Jansen at cwi.nl Fri Dec 26 16:58:33 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Fri Dec 26 16:58:37 2003 Subject: [Python-Dev] status of 2.3 branch for maintenance checkins In-Reply-To: <20031226003609.O1176@bullseye.apana.org.au> References: <20031226003609.O1176@bullseye.apana.org.au> Message-ID: On 25-dec-03, at 14:40, Andrew MacIntyre wrote: > I don't recall any advice as to whether the 2.3 branch is again open > for > checkins post 2.3 - Is Jack still sorting out the Mac release? Still busy. I expect to be done this wweekend, but I'll send a note here when I'm done. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From tim.one at comcast.net Fri Dec 26 17:01:32 2003 From: tim.one at comcast.net (Tim Peters) Date: Fri Dec 26 17:01:42 2003 Subject: [Python-Dev] test_unicode_file fails on Linux In-Reply-To: <1072470171.31901.74.camel@localhost.localdomain> Message-ID: [Jeremy Hylton] > The utime() call is failing for one of the Unicode file names. > > build> ./python ../Lib/test/test_unicode_file.py > test_directories (__main__.TestUnicodeFiles) ... ok > test_equivalent_files (__main__.TestUnicodeFiles) ... ok > test_single_files (__main__.TestUnicodeFiles) ... > '@test-\xc3\xa0\xc3\xb2' > '@test-\xc3\xa0\xc3\xb2' > u'@test-\xe0\xf2' > ERROR > > ====================================================================== > ERROR: test_single_files (__main__.TestUnicodeFiles) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "../Lib/test/test_unicode_file.py", line 143, in > test_single_files > self._test_single(TESTFN_UNICODE) > File "../Lib/test/test_unicode_file.py", line 117, in _test_single > self._do_single(filename) > File "../Lib/test/test_unicode_file.py", line 33, in _do_single > os.utime(filename, None) > UnicodeEncodeError: 'ascii' codec can't encode characters in position > 6-7: ordinal not in range(128) > > I'm running on a RH9 system. > > I see that the code hasn't changed since the beginning of the month, > so I'm surprised to only discover it now. I can't recall the last > time I did a full test run, so I'm not sure how long it has been > failing. It's been failing "quite a while" on Win98SE too, in the same way. I believe MarkH knows (or should know) about it already, but he's on vacation now: C:\Code\python\PCbuild>python ../lib/test/test_unicode_file.py test_directories (__main__.TestUnicodeFiles) ... ok test_equivalent_files (__main__.TestUnicodeFiles) ... ok test_single_files (__main__.TestUnicodeFiles) ... ERROR ====================================================================== ERROR: test_single_files (__main__.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_unicode_file.py", line 142, in test_single_files self._test_single(TESTFN_UNICODE) File "../lib/test/test_unicode_file.py", line 116, in _test_single self._do_single(filename) File "../lib/test/test_unicode_file.py", line 32, in _do_single os.utime(filename, None) UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-7: ordinal not in range(128) ---------------------------------------------------------------------- Ran 3 tests in 0.160s FAILED (errors=1) Traceback (most recent call last): File "../lib/test/test_unicode_file.py", line 171, in ? test_main() File "../lib/test/test_unicode_file.py", line 168, in test_main run_suite(suite) File "C:\CODE\PYTHON\lib\test\test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "../lib/test/test_unicode_file.py", line 142, in test_single_files self._test_single(TESTFN_UNICODE) File "../lib/test/test_unicode_file.py", line 116, in _test_single self._do_single(filename) File "../lib/test/test_unicode_file.py", line 32, in _do_single os.utime(filename, None) UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-7: ordinal not in range(128) test_urllib2 is also failing on Win98SE, and that's more recent: C:\Code\python\PCbuild>python ../lib/test/test_urllib2.py test_trivial (__main__.TrivialTests) ... ok test_handled (__main__.OpenerDirectorTests) ... ok test_handler_order (__main__.OpenerDirectorTests) ... ok test_http_error (__main__.OpenerDirectorTests) ... ok test_processors (__main__.OpenerDirectorTests) ... ok test_raise (__main__.OpenerDirectorTests) ... ok test_errors (__main__.HandlerTests) ... ok test_file (__main__.HandlerTests) ... ERROR test_ftp (__main__.HandlerTests) ... ok test_http (__main__.HandlerTests) ... ok test_redirect (__main__.HandlerTests) ... ok test_build_opener (__main__.MiscTests) ... ok ====================================================================== ERROR: test_file (__main__.HandlerTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_urllib2.py", line 345, in test_file r = h.file_open(Request(url)) File "C:\CODE\PYTHON\lib\urllib2.py", line 1058, in file_open return self.open_local_file(req) File "C:\CODE\PYTHON\lib\urllib2.py", line 1073, in open_local_file stats = os.stat(localfile) OSError: [Errno 2] No such file or directory: '\\test.txt' ---------------------------------------------------------------------- Ran 12 tests in 0.170s FAILED (errors=1) Traceback (most recent call last): File "../lib/test/test_urllib2.py", line 628, in ? test_main(verbose=True) File "../lib/test/test_urllib2.py", line 624, in test_main MiscTests, File "C:\CODE\PYTHON\lib\test\test_support.py", line 290, in run_unittest run_suite(suite, testclass) File "C:\CODE\PYTHON\lib\test\test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "../lib/test/test_urllib2.py", line 345, in test_file r = h.file_open(Request(url)) File "C:\CODE\PYTHON\lib\urllib2.py", line 1058, in file_open return self.open_local_file(req) File "C:\CODE\PYTHON\lib\urllib2.py", line 1073, in open_local_file stats = os.stat(localfile) OSError: [Errno 2] No such file or directory: '\\test.txt' test_bsddb3 is in such bad shape on Win98SE now I don't know where to begin. The compiler warnings about mixing signed and unsigned in comparisons in _sre.c also persist. Other than that, everything's great. From sjoerd at acm.org Fri Dec 26 15:39:00 2003 From: sjoerd at acm.org (Sjoerd Mullender) Date: Fri Dec 26 17:39:55 2003 Subject: [Python-Dev] Fixes for imageop module Message-ID: <3FEC9C64.5080805@acm.org> I have some fixes for the imageop module to make it work on systems with a different byte-order than IRIX (e.g. Linux). Shall I check them in or is it not worth it for such an old module? Anything else that needs to be changed when I check this in? The fixes of course will result in different results on Linux than before, so that's the main reason for asking. -- Sjoerd Mullender From guido at python.org Fri Dec 26 17:49:43 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 26 17:49:49 2003 Subject: [Python-Dev] Fixes for imageop module In-Reply-To: Your message of "Fri, 26 Dec 2003 21:39:00 +0100." <3FEC9C64.5080805@acm.org> References: <3FEC9C64.5080805@acm.org> Message-ID: <200312262249.hBQMnhf24306@c-24-5-183-134.client.comcast.net> > I have some fixes for the imageop module to make it work on systems with > a different byte-order than IRIX (e.g. Linux). Shall I check them in or > is it not worth it for such an old module? Anything else that needs to > be changed when I check this in? > The fixes of course will result in different results on Linux than > before, so that's the main reason for asking. I don't see imageop in the list of deprecated modules in PEP 4, and apparently *you* are still using it, so as long as you don't mind being potentially the sole maintainer and user, I don't mind these being in the distribution. What do you mean by different results on Linux? Was this module previously doing something bogus there? --Guido van Rossum (home page: http://www.python.org/~guido/) From sjoerd at acm.org Fri Dec 26 18:09:39 2003 From: sjoerd at acm.org (Sjoerd Mullender) Date: Fri Dec 26 18:09:44 2003 Subject: [Python-Dev] Fixes for imageop module In-Reply-To: <200312262249.hBQMnhf24306@c-24-5-183-134.client.comcast.net> References: <3FEC9C64.5080805@acm.org> <200312262249.hBQMnhf24306@c-24-5-183-134.client.comcast.net> Message-ID: <3FECBFB3.7090906@acm.org> Guido van Rossum wrote: >>I have some fixes for the imageop module to make it work on systems with >>a different byte-order than IRIX (e.g. Linux). Shall I check them in or >>is it not worth it for such an old module? Anything else that needs to >>be changed when I check this in? >>The fixes of course will result in different results on Linux than >>before, so that's the main reason for asking. > > > I don't see imageop in the list of deprecated modules in PEP 4, and > apparently *you* are still using it, so as long as you don't mind > being potentially the sole maintainer and user, I don't mind these > being in the distribution. > > What do you mean by different results on Linux? Was this module > previously doing something bogus there? > > --Guido van Rossum (home page: http://www.python.org/~guido/) The main difference is: *************** *** 570,577 **** r = (r<<5) | (r<<3) | (r>>1); g = (g<<5) | (g<<3) | (g>>1); b = (b<<6) | (b<<4) | (b<<2) | b; ! nvalue = r | (g<<8) | (b<<16); ! *ncp++ = nvalue; } return rv; } --- 560,569 ---- r = (r<<5) | (r<<3) | (r>>1); g = (g<<5) | (g<<3) | (g>>1); b = (b<<6) | (b<<4) | (b<<2) | b; ! *ncp++ = 0; ! *ncp++ = b; ! *ncp++ = g; ! *ncp++ = r; } return rv; } where the old ncp is an "Py_UInt32 *" and the new ncp is an "unsigned char *". This results in different byte ordering on a little endian machine such as Intel. -- Sjoerd Mullender From jackjansen at mac.com Sat Dec 27 10:37:08 2003 From: jackjansen at mac.com (Jack Jansen) Date: Sat Dec 27 10:37:15 2003 Subject: [Python-Dev] 2.3.3: test_import wreaks havoc when it runs out of memory Message-ID: <87A87885-3882-11D8-87A5-000A27B19B96@mac.com> During testing 2.3.3 for MacOS9 I think I came across a bug somewhere in import, but I don't have the time to track it down (and there's a workaround), and it's probably only ever going to happen on OS9. Should I put it in the bugs database anyway? If test_import runs out of memory (probably in the test_module_with_large_stack test) it fails with a MemoryError (as it should). However, from that point on there is something fishy in the state of the interpreter: any later test that should simply fail with an ImportError gets a SystemError: NULL return without exception in stead. After a couple of these the interpreter either hangs completely or crashes (I think around the threading tests, but I'm not sure). Note that it's not a question of simply all memory being tied up: it seems test_import does release most of the memory used, and many tests also pass after test_import. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From guido at python.org Sat Dec 27 12:49:46 2003 From: guido at python.org (Guido van Rossum) Date: Sat Dec 27 12:49:52 2003 Subject: [Python-Dev] Fixes for imageop module In-Reply-To: Your message of "Sat, 27 Dec 2003 00:09:39 +0100." <3FECBFB3.7090906@acm.org> References: <3FEC9C64.5080805@acm.org> <200312262249.hBQMnhf24306@c-24-5-183-134.client.comcast.net> <3FECBFB3.7090906@acm.org> Message-ID: <200312271749.hBRHnkN25210@c-24-5-183-134.client.comcast.net> [Sjoerd] > >>I have some fixes for the imageop module to make it work on systems with > >>a different byte-order than IRIX (e.g. Linux). Shall I check them in or > >>is it not worth it for such an old module? Anything else that needs to > >>be changed when I check this in? > >>The fixes of course will result in different results on Linux than > >>before, so that's the main reason for asking. > Guido van Rossum wrote: > > I don't see imageop in the list of deprecated modules in PEP 4, and > > apparently *you* are still using it, so as long as you don't mind > > being potentially the sole maintainer and user, I don't mind these > > being in the distribution. > > > > What do you mean by different results on Linux? Was this module > > previously doing something bogus there? [Sjoerd] > The main difference is: > *************** > *** 570,577 **** > r = (r<<5) | (r<<3) | (r>>1); > g = (g<<5) | (g<<3) | (g>>1); > b = (b<<6) | (b<<4) | (b<<2) | b; > ! nvalue = r | (g<<8) | (b<<16); > ! *ncp++ = nvalue; > } > return rv; > } > --- 560,569 ---- > r = (r<<5) | (r<<3) | (r>>1); > g = (g<<5) | (g<<3) | (g>>1); > b = (b<<6) | (b<<4) | (b<<2) | b; > ! *ncp++ = 0; > ! *ncp++ = b; > ! *ncp++ = g; > ! *ncp++ = r; > } > return rv; > } > > where the old ncp is an "Py_UInt32 *" and the new ncp is an "unsigned > char *". This results in different byte ordering on a little endian > machine such as Intel. Hm. Anybody who uses the imageop module currently on Linux will find their programs broken. The strings manipulated by imageop all end up either being written to a file or handed to some drawing code, and changing the byte order would definitely break things! So I don't think this is an acceptable change. I take it that for IRIX, the byte order implied by the old code is simply wrong? Maybe the module can be given a global (shrudder?) byte order setting that you can change but that defaults to the old setting? --Guido van Rossum (home page: http://www.python.org/~guido/) From sjoerd at acm.org Sat Dec 27 13:12:50 2003 From: sjoerd at acm.org (Sjoerd Mullender) Date: Sat Dec 27 13:13:02 2003 Subject: [Python-Dev] Fixes for imageop module In-Reply-To: <200312271749.hBRHnkN25210@c-24-5-183-134.client.comcast.net> References: <3FEC9C64.5080805@acm.org> <200312262249.hBQMnhf24306@c-24-5-183-134.client.comcast.net> <3FECBFB3.7090906@acm.org> <200312271749.hBRHnkN25210@c-24-5-183-134.client.comcast.net> Message-ID: <3FEDCBA2.3030607@acm.org> Guido van Rossum wrote: > [Sjoerd] > >>>>I have some fixes for the imageop module to make it work on systems with >>>>a different byte-order than IRIX (e.g. Linux). Shall I check them in or >>>>is it not worth it for such an old module? Anything else that needs to >>>>be changed when I check this in? >>>>The fixes of course will result in different results on Linux than >>>>before, so that's the main reason for asking. > > >>Guido van Rossum wrote: >> >>>I don't see imageop in the list of deprecated modules in PEP 4, and >>>apparently *you* are still using it, so as long as you don't mind >>>being potentially the sole maintainer and user, I don't mind these >>>being in the distribution. >>> >>>What do you mean by different results on Linux? Was this module >>>previously doing something bogus there? > > > [Sjoerd] > >>The main difference is: >>*************** >>*** 570,577 **** >> r = (r<<5) | (r<<3) | (r>>1); >> g = (g<<5) | (g<<3) | (g>>1); >> b = (b<<6) | (b<<4) | (b<<2) | b; >>! nvalue = r | (g<<8) | (b<<16); >>! *ncp++ = nvalue; >> } >> return rv; >> } >>--- 560,569 ---- >> r = (r<<5) | (r<<3) | (r>>1); >> g = (g<<5) | (g<<3) | (g>>1); >> b = (b<<6) | (b<<4) | (b<<2) | b; >>! *ncp++ = 0; >>! *ncp++ = b; >>! *ncp++ = g; >>! *ncp++ = r; >> } >> return rv; >> } >> >>where the old ncp is an "Py_UInt32 *" and the new ncp is an "unsigned >>char *". This results in different byte ordering on a little endian >>machine such as Intel. > > > Hm. Anybody who uses the imageop module currently on Linux will find > their programs broken. The strings manipulated by imageop all end up > either being written to a file or handed to some drawing code, and > changing the byte order would definitely break things! That's why I asked. > So I don't think this is an acceptable change. I take it that for > IRIX, the byte order implied by the old code is simply wrong? Maybe > the module can be given a global (shrudder?) byte order setting that > you can change but that defaults to the old setting? The problem is, the documentation says: "This is the same format as used by gl.lrectwrite() and the imgfile module." This implies the byte order that you get on the SGI which is opposite of what you get on Intel. The code produced the correct results on the SGI, but not on Intel. (By the way, I'm away from computers for a week starting tomorrow extremely early.) -- Sjoerd Mullender From troels at 2-10.org Sat Dec 27 15:10:05 2003 From: troels at 2-10.org (Troels Therkelsen) Date: Sat Dec 27 15:10:11 2003 Subject: [Python-Dev] 2.3.3 Compilation warnings... Message-ID: Hey all, I don't know what the policy on compilation warnings are with the development of Python, but I'd imagine that it's "no warnings". I get the following warnings: gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes - I. -I./Include -DPy_BUILD_CORE -o Python/errors.o Python/errors.c Python/errors.c:602: warning: function declaration isn't a prototype gcc -pthread -Xlinker -export-dynamic -o python \ Modules/python.o \ libpython2.3.a -lpthread -ldl -lutil -lm libpython2.3.a(posixmodule.o): In function `posix_tmpnam': /usr/local/src/Python-2.3.3/./Modules/posixmodule.c:5833: the use of `tmpnam_r' is dangerous, better use `mkstemp' libpython2.3.a(posixmodule.o): In function `posix_tempnam': /usr/local/src/Python-2.3.3/./Modules/posixmodule.c:5788: the use of `tempnam' is dangerous, better use `mkstemp' building 'readline' extension gcc -pthread -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -fno-strict-aliasin g -I. -I/usr/local/src/Python-2.3.3/./Include -I/usr/local/include -I/usr/local/ src/Python-2.3.3/Include -I/usr/local/src/Python-2.3.3 -c /usr/local/src/Python-2.3.3/Modules/readline.c -o build/temp.linux-i686-2.3/readline.o /usr/local/src/Python-2.3.3/Modules/readline.c: In function `flex_complete': /usr/local/src/Python-2.3.3/Modules/readline.c:583: warning: implicit declaration of function `completion_matches' /usr/local/src/Python-2.3.3/Modules/readline.c:583: warning: return makes pointer from integer without a cast I guess the warnings in posixmodule has always been there, I just haven't been paying much attention to it and I thought I might just as well report all warnings. System specs: $ uname -a Linux gateway 2.2.19 #22 Wed Jun 20 18:12:16 PDT 2001 i686 unknown $ gcc -v Reading specs from /usr/lib/gcc-lib/i386-slackware-linux/2.95.3/specs gcc version 2.95.3 20010315 (release) Regards, Troels Therkelsen From guido at python.org Sat Dec 27 17:39:47 2003 From: guido at python.org (Guido van Rossum) Date: Sat Dec 27 17:39:53 2003 Subject: [Python-Dev] 2.3.3 Compilation warnings... In-Reply-To: Your message of "Sat, 27 Dec 2003 21:10:05 +0100." References: Message-ID: <200312272239.hBRMdlo01094@c-24-5-183-134.client.comcast.net> > I don't know what the policy on compilation warnings are with the > development of Python, but I'd imagine that it's "no warnings". Right, if possible. > I get the following warnings: > > gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes - > I. -I./Include -DPy_BUILD_CORE -o Python/errors.o Python/errors.c > Python/errors.c:602: warning: function declaration isn't a prototype > > gcc -pthread -Xlinker -export-dynamic -o python \ > Modules/python.o \ > libpython2.3.a -lpthread -ldl -lutil -lm > libpython2.3.a(posixmodule.o): In function `posix_tmpnam': > /usr/local/src/Python-2.3.3/./Modules/posixmodule.c:5833: the use of `tmpnam_r' > is dangerous, better use `mkstemp' > libpython2.3.a(posixmodule.o): In function `posix_tempnam': > /usr/local/src/Python-2.3.3/./Modules/posixmodule.c:5788: the use of `tempnam' > is dangerous, better use `mkstemp' Note that this is the linker speaking, not the compiler. The only way to get rid of this warning is not to use those two entry points, which means removing the functions os.tempnam() and os.tmpnam(). That will eventually happy. These functions have been deprecated already, but for backwards compatibility, we need to provide them for a few more releases. In the mean time, I don't know how to tell the linker to suppress the warnings. > building 'readline' extension > gcc -pthread -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -fno-strict-aliasin > g -I. -I/usr/local/src/Python-2.3.3/./Include -I/usr/local/include -I/usr/local/ > src/Python-2.3.3/Include -I/usr/local/src/Python-2.3.3 -c > /usr/local/src/Python-2.3.3/Modules/readline.c -o > build/temp.linux-i686-2.3/readline.o > /usr/local/src/Python-2.3.3/Modules/readline.c: In function `flex_complete': > /usr/local/src/Python-2.3.3/Modules/readline.c:583: warning: implicit > declaration of function `completion_matches' > /usr/local/src/Python-2.3.3/Modules/readline.c:583: warning: return makes > pointer from integer without a cast This seems a case of a missing function declaration in readline.h. Unfortunately being compatible with all versions of GNU readline that are around is very tricky. If you have a patch, I'd consider it, but it would have to be tested on a number of different platforms before being accepted. I don't have the time to do all the verification myself. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Dec 27 17:41:16 2003 From: guido at python.org (Guido van Rossum) Date: Sat Dec 27 17:41:23 2003 Subject: [Python-Dev] Fixes for imageop module In-Reply-To: Your message of "Sat, 27 Dec 2003 19:12:50 +0100." <3FEDCBA2.3030607@acm.org> References: <3FEC9C64.5080805@acm.org> <200312262249.hBQMnhf24306@c-24-5-183-134.client.comcast.net> <3FECBFB3.7090906@acm.org> <200312271749.hBRHnkN25210@c-24-5-183-134.client.comcast.net> <3FEDCBA2.3030607@acm.org> Message-ID: <200312272241.hBRMfHb01333@c-24-5-183-134.client.comcast.net> > > Hm. Anybody who uses the imageop module currently on Linux will find > > their programs broken. The strings manipulated by imageop all end up > > either being written to a file or handed to some drawing code, and > > changing the byte order would definitely break things! > > That's why I asked. > > > So I don't think this is an acceptable change. I take it that for > > IRIX, the byte order implied by the old code is simply wrong? Maybe > > the module can be given a global (shrudder?) byte order setting that > > you can change but that defaults to the old setting? > > The problem is, the documentation says: "This is the same format as used > by gl.lrectwrite() and the imgfile module." This implies the byte order > that you get on the SGI which is opposite of what you get on Intel. The > code produced the correct results on the SGI, but not on Intel. Your fix would be okay if until now it was *only* used on IRIX with gl.lrectwrite() and/or the imgfile module. But how can we prove that? > (By the way, I'm away from computers for a week starting tomorrow > extremely early.) It's okay to way a week before making a decision. --Guido van Rossum (home page: http://www.python.org/~guido/) From edcjones at erols.com Sat Dec 27 18:13:49 2003 From: edcjones at erols.com (Edward C. Jones) Date: Sat Dec 27 18:17:43 2003 Subject: [Python-Dev] Valgrind on Python? Message-ID: <3FEE122D.6040702@erols.com> Has anyone run valgrind on Python? I get a lot of messages, probably mostly incorrect. Most are from "PyObject_Free (Objects/obmalloc.c:711)". I am using Python 2.3.2 and gcc 3.2.2 with Linux. From skanigsvv at terland.ternopil.ua Sat Dec 27 20:19:54 2003 From: skanigsvv at terland.ternopil.ua (Skanigsvv) Date: Sat Dec 27 20:20:18 2003 Subject: [Python-Dev] Python-dev In-Reply-To: References: Message-ID: <9GLL219D5I8FF0CF@terland.ternopil.ua> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031228/e5f01921/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: v1.gif Type: image/gif Size: 7859 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20031228/e5f01921/v1-0001.gif From andymac at bullseye.apana.org.au Sat Dec 27 18:26:59 2003 From: andymac at bullseye.apana.org.au (Andrew MacIntyre) Date: Sat Dec 27 20:49:33 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects listobject.c, 2.172, 2.173 In-Reply-To: <1072465591.31901.71.camel@localhost.localdomain> References: <1072465591.31901.71.camel@localhost.localdomain> Message-ID: <20031228092926.D12899@bullseye.apana.org.au> On Fri, 26 Dec 2003, Jeremy Hylton wrote: > > Modified Files: > > listobject.c > > Log Message: > > Revert previous two checkins to repair test failure. > > > > The special-case code that was removed could return a value indicating > > success but leave an exception set. test_fileinput failed in a debug > > build as a result. > > If there's some value to the optimization I just removed, let's discuss > it on python-dev. I could have fixed the new code, but it seems a lot > of hairy new code with magic constants that only covered a few corner > cases. The checkin message pointed to a python-list discussion that > didn't have much real evidence in it. While Duncan Booth's tests didn't properly account for setup overhead and his coverage of the range of source element counts insufficient, the performance difference between the two approaches is real and can be quite significant - list_ass_slice() is a big win for large source lists, and a non-trivial loss for small lists (>35% for 10 or fewer elements, >14% up to 64 elements). The change was an attempt to maximise the performance for all cases. If performance of list() is not seen as particularly crucial, then I would suggest that the special case calling of list_ass_slice() should probably be removed completely. The special casing of empty source sequences is not really important IMO, but the timings looked good... Some of the interest in this area comes from that class of Python user who wants to make a shallow copy of a list, and looks for a copy() method on list objects, and failing that finds copy.copy(). Some of them eventually twig that list() also does this, and doesn't require importing the copy module. These users don't feel comfortable with the slice approach, despite the fact that it is considerably faster. In fact I just noticed a posting to python-list/c.l.p from Alex Martelli in which he suggests list() is not unreasonable to use for this purpose. I don't have any problem with the slice, myself. While I tested the change extensively, I'll confess to not trying a debug build :-(. BTW, my ISP's dialin is playing silly buggers at the moment, so I'm not seeing email in a timely fashion. :-( That's not an excuse for the PyDict_CheckExact goof, because I saw that checkin msg and still managed not to see the problem, and hadn't gotten around to building from a fresh CVS checkout :-( Thanks for fixing that. Regards, Andrew. -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From tree at basistech.com Sat Dec 27 20:55:52 2003 From: tree at basistech.com (Tom Emerson) Date: Sat Dec 27 21:01:34 2003 Subject: [Python-Dev] Valgrind on Python? In-Reply-To: <3FEE122D.6040702@erols.com> References: <3FEE122D.6040702@erols.com> Message-ID: <16366.14376.408291.541535@magrathea.basistech.com> Yes: valgrind and pymalloc do not get along very well. If you either setup Valgrind exceptions, or better, build without pymaloc, you (and valgrind) will be much happier. -tree -- Tom Emerson Basis Technology Corp. Software Architect http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever" From skip at manatee.mojam.com Sun Dec 28 08:01:17 2003 From: skip at manatee.mojam.com (Skip Montanaro) Date: Sun Dec 28 08:01:26 2003 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200312281301.hBSD1HoM014593@manatee.mojam.com> Bug/Patch Summary ----------------- 618 open / 4455 total bugs (+70) 220 open / 2504 total patches (+22) New Bugs -------- socketmodule.c does not pickup all symbols with MS visC++ (2003-12-14) http://python.org/sf/860134 PythonIDE does not save for PythonLauncher (2003-12-15) http://python.org/sf/860242 fileinput does not use universal input (2003-12-15) http://python.org/sf/860515 Core dump problems (2003-12-15) http://python.org/sf/860547 UnboundLocalError in cgitb.py (2003-12-16) http://python.org/sf/861340 PythonIDE on osx can't run script with commandline python (2003-12-19) http://python.org/sf/862941 ConfigParser.getint failure on 2.3.3 (2003-12-21) http://python.org/sf/864063 2.3.3 tests on cygwin (2003-12-22) http://python.org/sf/864374 doctest chokes on recursive members (2003-12-23) http://python.org/sf/864944 Python 2.3.3 won't build on MacOSX 10.2 (2003-12-23) http://python.org/sf/864985 Hangs when opening 2nd shell with subprocess under windows (2003-12-23) http://python.org/sf/865014 bsddb test_all.py - incorrect (2003-12-23) http://python.org/sf/865120 SimpleHTTPServer docs out of date (2003-12-26) http://python.org/sf/866222 New Patches ----------- tests and processors patch for urllib2 (2003-12-02) http://python.org/sf/852995 PyErrr_Display and traceback.format_exception_only behaviour (2003-12-15) http://python.org/sf/860326 installing modules are explained in old filenames (2003-12-18) http://python.org/sf/862531 heapq: A way to change the compare function (2003-12-28) http://python.org/sf/866594 Closed Bugs ----------- Over restricted type checking on eval() function (2000-09-22) http://python.org/sf/215126 smtpd.py needs documentation (2001-08-14) http://python.org/sf/450803 Confusions in formatfloat (2002-03-20) http://python.org/sf/532631 email package does not work with mailbox (2002-07-26) http://python.org/sf/586899 Another dealloc stack killer (2002-08-25) http://python.org/sf/600007 doctest fails with TypeError (2003-07-02) http://python.org/sf/764504 doctest.DocTestSuite does not support __test__ (2003-07-15) http://python.org/sf/772091 bsddb3 hash craps out with threads (2003-07-21) http://python.org/sf/775414 Windows _bsddb link warnings (2003-08-01) http://python.org/sf/781614 Minor FP bug in object.c (2003-08-15) http://python.org/sf/789290 Parser wart (2003-08-27) http://python.org/sf/796116 shutil.copyfile fails when dst exists read-only (2003-09-22) http://python.org/sf/810879 Proto 2 pickle vs dict subclass (2003-10-20) http://python.org/sf/826897 os.makedirs() cannot handle "." (2003-10-24) http://python.org/sf/829532 reading shelves is really slow (2003-11-26) http://python.org/sf/849662 Typo in Popen3 description (2003-11-28) http://python.org/sf/850818 Doc/README has broken link (2003-11-28) http://python.org/sf/850823 Missing BuildRequires in src rpm specfile (2003-11-28) http://python.org/sf/851020 socket.recv() raises MemoryError exception (WindowsXP ONLY) (2003-12-03) http://python.org/sf/853507 UserDict.DictMixin's comments should be a docstring (2003-12-07) http://python.org/sf/856072 Remove notion of deprecated string.{atol,atof} functions (2003-12-10) http://python.org/sf/857821 Pathological case segmentation fault in issubclass (2003-12-10) http://python.org/sf/858016 typo in doc (2003-12-14) http://python.org/sf/859810 typo in docs (2003-12-14) http://python.org/sf/859811 Closed Patches -------------- 658254: accept None for time.ctime() and friends (2003-01-06) http://python.org/sf/663482 doctest handles comments incorrectly (2003-01-15) http://python.org/sf/668500 iconv_codec 3rd generation (2003-04-13) http://python.org/sf/720585 test_timeout updates (2003-04-28) http://python.org/sf/728815 Port tests to unittest (Part 2) (2003-05-13) http://python.org/sf/736962 traceback module caches sources invalid (2003-05-13) http://python.org/sf/737473 ast-branch: multiple function fixes (2003-06-11) http://python.org/sf/752911 itertools roundrobin() (2003-06-17) http://python.org/sf/756253 add time elapsed to gc debug output (2003-06-25) http://python.org/sf/760990 A ForwardingHandler for logging (2003-09-10) http://python.org/sf/804180 call com_set_lineno more often (2003-11-28) http://python.org/sf/850789 zipimport.c is broken on 64-bit SusE AMD, here's a fix (2003-12-11) http://python.org/sf/858317 documentation bool change fix (2003-12-12) http://python.org/sf/859286 Get rid of compiler warnings on unicodeobject.c (2003-12-13) http://python.org/sf/859573 From barry at barrys-emacs.org Sun Dec 28 14:48:23 2003 From: barry at barrys-emacs.org (Barry Scott) Date: Sun Dec 28 14:48:31 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312241812.hBOICdK22045@c-24-5-183-134.client.comcast.ne t> References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> <200312241812.hBOICdK22045@c-24-5-183-134.client.comcast.net> Message-ID: <6.0.1.1.2.20031228194701.023490c8@torment.chelsea.private> At 24-12-2003 18:12, Guido van Rossum wrote: >I've tried to look for a way to report OpenBSD bugs, but it appears >the only way to report one is by using their sendbug command, which >assumes you've got OpenBSD installed. Would someone here be so kind >as to report the bug we've found? A pointer to Kurt's last message in >this thread should be a good piece of evidence. > >--Guido van Rossum (home page: http://www.python.org/~guido/) From the "How to report a problem" page on www.openbsd.org they say mail to bugs@openbsd.org. Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031228/13980c19/attachment.html From barry at barrys-emacs.org Sun Dec 28 14:53:33 2003 From: barry at barrys-emacs.org (Barry Scott) Date: Sun Dec 28 14:53:39 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> Message-ID: <6.0.1.1.2.20031228195231.022a77c8@torment.chelsea.private> You can use -fno-stack-protector to turn off the propolice code. Why not do that until there is a working combination? Barry From perky at i18n.org Mon Dec 29 03:17:24 2003 From: perky at i18n.org (Hye-Shik Chang) Date: Mon Dec 29 03:17:31 2003 Subject: [Python-Dev] Optimization versus code bloat Message-ID: <20031229081724.GA42076@i18n.org> Hello! I posted a patch[1] optimizing str.split with specialized splitter as unicode.split do. The patch accellates str.split about 10~20% and adds about 50 lines. I'm curious whether it's acceptable ratio for python-dev moods. [1] http://www.python.org/sf/866875 Hye-Shik From python-dev at zesty.ca Mon Dec 29 04:43:16 2003 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Mon Dec 29 04:43:31 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220151629.GA24651@panix.com> Message-ID: I've been distracted by holiday events, but this requires a reply. On Sat, 20 Dec 2003, Aahz wrote: > Exactly. From my observations of these discussions, there are > essentially only two reasons for restricted execution: > > * To simplify things by reducing the potential solution space > > * To protect a system against a hostile attacker There is a huge blind spot in your claim. You forgot: * To limit the damage caused by a bug in your program * To make your programs have more predictable behaviour Capabilities are about making clear what parts of your program can and can't do. > Supposedly there's a middle ground of untrusted but non-hostile code, > but what's the point of providing support for that? Have you ever used a library written by someone else? Have you ever found a bug in something you wrote yourself? "Untrusted but non-hostile code" is what all of us write every day. -- ?!ng From guido at python.org Mon Dec 29 09:52:46 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 29 09:53:02 2003 Subject: [Python-Dev] Optimization versus code bloat In-Reply-To: Your message of "Mon, 29 Dec 2003 17:17:24 +0900." <20031229081724.GA42076@i18n.org> References: <20031229081724.GA42076@i18n.org> Message-ID: <200312291452.hBTEql011394@c-24-5-183-134.client.comcast.net> > I posted a patch[1] optimizing str.split with specialized splitter > as unicode.split do. The patch accellates str.split about 10~20% > and adds about 50 lines. I'm curious whether it's acceptable ratio > for python-dev moods. > > [1] http://www.python.org/sf/866875 Works for me -- split is a pretty common operation! I might even use this speedup in the Parrot benchmark. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Mon Dec 29 09:56:06 2003 From: aahz at pythoncraft.com (Aahz) Date: Mon Dec 29 09:56:10 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: References: <20031220151629.GA24651@panix.com> Message-ID: <20031229145606.GA24858@panix.com> On Mon, Dec 29, 2003, Ka-Ping Yee wrote: > On Sat, 20 Dec 2003, Aahz wrote: >> >> Exactly. From my observations of these discussions, there are >> essentially only two reasons for restricted execution: >> >> * To simplify things by reducing the potential solution space >> >> * To protect a system against a hostile attacker > > There is a huge blind spot in your claim. You forgot: > > * To limit the damage caused by a bug in your program > > * To make your programs have more predictable behaviour > > Capabilities are about making clear what parts of your program > can and can't do. My understanding is that capabilities are not the same thing as restricted execution; it's certainly the case that we could provide some kind of capability model that doesn't meet the demands of restricted execution. While your points are valid, they are not points that I've seen brought up before in the context of restricted execution. There are many other programming models that address those issues; why do you think that restricted execution makes a particularly good choice? (Say, as compared to design-by-contract.) >> Supposedly there's a middle ground of untrusted but non-hostile code, >> but what's the point of providing support for that? > > Have you ever used a library written by someone else? Have you ever > found a bug in something you wrote yourself? > > "Untrusted but non-hostile code" is what all of us write every day. Well, sure, but if that's the only issue in play, I don't think that the stringent demands of restricted execution are needed. The question is whether the benefits of restricted execution are worth the effort required (and possibly the performance penalty). Not-quite tangent: I was talking to someone a few days ago, an experienced programmer who hasn't yet tried Python (and who I'm trying to persuade -- but he's an old Lisp and Smalltalk hacker who hates B&D syntax). He's also running a web server at home; I commented that if I did that, I'd use a server written in Python because it's more secure. He said that he'd seen more security alerts for Zope than for Apache, which forced me to explain that Zope isn't a web server, it's an application framework that allows people to run arbitrary Python code. >From my POV, that's the context in which we either need to provide truly robust restricted execution or just decide that we can't do it. And as I've pointed out before, restricted execution in the context of something like a web server is almost certainly going to require something external to Python in order to prevent excessive CPU and memory usage -- which is an issue that has grown in difficulty given the new prevalence of event-driven programming (instead of forking and threading). -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From martin at v.loewis.de Mon Dec 29 10:25:40 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 10:25:55 2003 Subject: [Python-Dev] First version of Python MSI available Message-ID: <3FF04774.9020404@v.loewis.de> I have committed python/nondist/sandbox/msi to the CVS. This is a MSI generator for Python on Windows, which I'd like to propose as a replacement for Wise. A sample .msi file is available at http://www.dcl.hpi.uni-potsdam.de/home/loewis/python2.4.0.msi This installer includes binaries compiled with MSVC 7.1 (aka VC.NET 2003). The MSI file tries to copy most features and user interface from the Wise installer, but adjusts to the feature set of MSI. I'm aware of a few issues, but I'd like to request feedback in any case. Known issues: - the C runtime DLL is not yet included. It turns out that the VC.NET merge module for msvcr71.dll is useless, so I have to extract the DLL from it and incorporate that; I'd also like to include some auto-detection of DLLs used. - MSI "administrative installs" (network installs) are not tested yet; neither is advertisement (this should not work well, in any case, as the MSI file won't advertise the shortcuts) - the package doesn't ask for confirmation when installing into an existing directory. Not sure how to do this with MSI. Please let me know what you think, Martin From martin at v.loewis.de Mon Dec 29 10:27:28 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 10:27:46 2003 Subject: [Python-Dev] Switching to VC.NET 2003 Message-ID: <3FF047E0.8070905@v.loewis.de> In my own sandbox, I have prepared project files for VC.NET. Unless there are any strong objections, I'd like to switch over to VC.NET shortly, disabling VC6 builds. In the process, I will also update the build dependencies to more recent versions of several packages. Regards, Martin From guido at python.org Mon Dec 29 10:32:30 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 29 10:32:35 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: Your message of "Mon, 29 Dec 2003 16:27:28 +0100." <3FF047E0.8070905@v.loewis.de> References: <3FF047E0.8070905@v.loewis.de> Message-ID: <200312291532.hBTFWU011539@c-24-5-183-134.client.comcast.net> > In my own sandbox, I have prepared project files for VC.NET. > > Unless there are any strong objections, I'd like to switch > over to VC.NET shortly, disabling VC6 builds. In the process, > I will also update the build dependencies to more recent > versions of several packages. Will the resulting installer still install a working version of Python on old versions of Windows, like win95 and win98? I presume this will require that the MSI support is present on the target system; does this exist for Win95 etc? How about Windows NT 4.0? If not, what's the oldest Windows version still supported? --Guido van Rossum (home page: http://www.python.org/~guido/) From logistix at cathoderaymission.net Mon Dec 29 11:09:10 2003 From: logistix at cathoderaymission.net (logistix) Date: Mon Dec 29 10:54:29 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <200312291532.hBTFWU011539@c-24-5-183-134.client.comcast.net> Message-ID: > In my own sandbox, I have prepared project files for VC.NET. > > Unless there are any strong objections, I'd like to switch > over to VC.NET shortly, disabling VC6 builds. In the process, > I will also update the build dependencies to more recent > versions of several packages. > Are you using a Visual Studio "project" or a makefile based build? I believe you need a makefile based build if you want to let people develop with the free SDK from microsoft. I have some notes here that I posted a while ago: http://tinyurl.com/34ljh From martin at v.loewis.de Mon Dec 29 11:00:47 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 11:01:01 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <200312291532.hBTFWU011539@c-24-5-183-134.client.comcast.net> References: <3FF047E0.8070905@v.loewis.de> <200312291532.hBTFWU011539@c-24-5-183-134.client.comcast.net> Message-ID: <3FF04FAF.40209@v.loewis.de> Guido van Rossum wrote: > Will the resulting installer still install a working version of Python > on old versions of Windows, like win95 and win98? I don't know; in principle, it should (or atleast, we should be able to make it work if desired). > I presume this will > require that the MSI support is present on the target system; does > this exist for Win95 etc? How about Windows NT 4.0? Yes; there is a four-file bootstrap mechanism that we could distribute. It consists of 1. a tiny setup.exe 2. an installer for installer for W9x 3. an installer for installer for WNT 4. the actual MSI file. I believe the setup.exe driver needs to be parametrized as part of the build process to hard-code the name of the MSI file; I have no procedure yet to do this. I would propose to make parts 1..3 optional downloads; we can also (alternatively) point users to the redistributable download at http://www.microsoft.com/msdownload/platformsdk/instmsi.htm With that procedure, they would explicitly need to install installer, then reboot (I think), then install Python. On some installations, installer might be present already, but in a too-old version (I use the schema of Installer 2.0). In principle, the already-present installer should find out that it is too old, and suggest some sort of upgrade. > If not, what's the oldest Windows version still supported? A different question is the version of shared libraries that the Python binaries depend on. On the one hand, this is msvcr71.dll, which is not present by default on any Windows release, so we need to ship that. Another issue is Winsock 2 (which, I believe, is in wsock32.dll). A quick research indicates that this was present in Win95 already. I don't know whether all of the entry points that we use was present already; confirmation appreciated. In principle, usage of IPv6 could be a problem. In another principle, the platform SDK arranges to transparently emulate the IPv6 API on systems where it is not natively available, by means of GetProcAddress. This all needs testing. Regards, Martin From martin at v.loewis.de Mon Dec 29 11:06:39 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 11:06:52 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: References: Message-ID: <3FF0510F.3070504@v.loewis.de> logistix wrote: > Are you using a Visual Studio "project" or a makefile based build? I > believe you need a makefile based build if you want to let people > develop with the free SDK from microsoft. I'm using project files, so users of the free compiler would have to write their makefiles themselves. It might be possible to write a generator that produces a makefile automatically from the VC.NET project files; contributions are welcome. Maintaining both project files and makefiles in parallel is not feasible; dropping the project files is also unacceptable. Regards, Martin From martin at v.loewis.de Mon Dec 29 11:12:44 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 11:13:04 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF04FAF.40209@v.loewis.de> References: <3FF047E0.8070905@v.loewis.de> <200312291532.hBTFWU011539@c-24-5-183-134.client.comcast.net> <3FF04FAF.40209@v.loewis.de> Message-ID: <3FF0527C.6010909@v.loewis.de> Martin v. Loewis wrote: > On some installations, installer might be present already, > but in a too-old version (I use the schema of Installer 2.0). > In principle, the already-present installer should find out > that it is too old, and suggest some sort of upgrade. Following up to myself: If this turns out to be a frequent problem, we could probably go back to an older installer scheme - in particular, if we don't want to fight Windows File Protection (which is only transparently supported in most recent installer versions). In the sample installer, I put python24.dll into TARGETDIR, not into System32Folder. I'd like to keep it that way, as Microsoft discourages deployment into system32. Of course, if there is a good reason to install python24.dll into system32, changing the installer generator to do so would be only a small effort. Regards, Martin From pje at telecommunity.com Mon Dec 29 11:36:06 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Dec 29 11:37:31 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF047E0.8070905@v.loewis.de> Message-ID: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> At 04:27 PM 12/29/03 +0100, Martin v. Loewis wrote: >In my own sandbox, I have prepared project files for VC.NET. > >Unless there are any strong objections, I'd like to switch >over to VC.NET shortly, disabling VC6 builds. In the process, >I will also update the build dependencies to more recent >versions of several packages. Was the question of mingw32-built extensions ever resolved? That is, will we be able to build extensions for the standard Windows Python using the distutils' "mingw32" compiler, as is possible with Python 2.2? If this hasn't been resolved, and somebody can send me a binary for a .NET-built Python, I'll be happy to test. I have several Pyrex and C extensions of varying vintage that I can use for such a test. From tim.one at comcast.net Mon Dec 29 11:38:33 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 29 11:38:40 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF0527C.6010909@v.loewis.de> Message-ID: [Martin v. Loewis] > ... > > In the sample installer, I put python24.dll into TARGETDIR, > not into System32Folder. I'd like to keep it that way, > as Microsoft discourages deployment into system32. Of course, > if there is a good reason to install python24.dll into > system32, changing the installer generator to do so would > be only a small effort. We need to hear from Mark Hammond about this, but he's on vacation now. The current installer's "non-admin install" option leaves system folders alone, and works fine for everything I routinely do on Windows. The point to stuffing the Python DLL into a system directory had (IIRC) something to do with a Windows Service embedding a Python component via COM, in which case "the virtual command line" that started the service has no idea where Python might live on the box. Or something like that -- or something not like that at all. That's why we need Mark . From martin at v.loewis.de Mon Dec 29 11:43:13 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 11:43:32 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> Message-ID: <3FF059A1.8090202@v.loewis.de> Phillip J. Eby wrote: > Was the question of mingw32-built extensions ever resolved? That is, > will we be able to build extensions for the standard Windows Python > using the distutils' "mingw32" compiler, as is possible with Python 2.2? I don't know; I don't use mingw32. OTOH, I cannot see what the problem might be - except that you will have to link with msvcr71.dll at the end, not with msvcrt40.dll. > If this hasn't been resolved, and somebody can send me a binary for a > .NET-built Python, I'll be happy to test. Sure: Please try my installer at http://www.dcl.hpi.uni-potsdam.de/home/loewis/python2.4.0.msi Notice that this doesn't include msvcr71.dll, which I could provide in private. It also does not include msvcr71.lib - I have no clue how you would get a copy of that. But then, somehow, msvcrt40.lib (or is it msvcrt40.a) must have gotten into mingw32 also. Regards, Martin From martin at v.loewis.de Mon Dec 29 11:45:52 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 11:46:17 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: References: Message-ID: <3FF05A40.4070709@v.loewis.de> Tim Peters wrote: > The point to > stuffing the Python DLL into a system directory had (IIRC) something to do > with a Windows Service embedding a Python component via COM, in which case > "the virtual command line" that started the service has no idea where Python > might live on the box. Ah, ok. I'll figure out in the mean time how to incorporate msvcr71.dll into the package in the first place; putting it then into a "shared component" (along with pythonxy.dll) should be trivial. Regards, Martin From martin at v.loewis.de Mon Dec 29 11:50:18 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 11:50:34 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF059A1.8090202@v.loewis.de> References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> Message-ID: <3FF05B4A.2070706@v.loewis.de> Martin v. Loewis wrote: > It also does not include msvcr71.lib - I have no clue how you would > get a copy of that. But then, somehow, msvcrt40.lib (or is it > msvcrt40.a) must have gotten into mingw32 also. Some googling shows there is hope: http://sources.redhat.com/ml/cygwin-announce/2003-09/msg00089.html 2003-06-18 Earnie Boyd * Makefile.in (LIBS): Add new MSVCRT libraries libmsvcr70 and libmsvcr71, including debug versions. (msvcr70.def, msvcr70d.def, msvcr71.def, msvcr71.def): New targets. So atleast the part of linking with msvcr71.dll should be possible. As I said, I cannot imagine what else might be a problem. Regards, Martin From aleaxit at yahoo.com Mon Dec 29 11:53:09 2003 From: aleaxit at yahoo.com (Alex Martelli) Date: Mon Dec 29 11:53:23 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF0510F.3070504@v.loewis.de> References: <3FF0510F.3070504@v.loewis.de> Message-ID: <200312291753.09793.aleaxit@yahoo.com> On Monday 29 December 2003 05:06 pm, Martin v. Loewis wrote: ... > It might be possible to write a generator that produces a makefile > automatically from the VC.NET project files; contributions At the time of Visual Studio 6, such a "generator" had been written by Microsoft, and was part of the VS6 package -- i.e., you could produce and save a makefile for the project you had loaded. Is this no more the case for the current release of Visual Studio...? Alex From logistix at cathoderaymission.net Mon Dec 29 12:14:53 2003 From: logistix at cathoderaymission.net (logistix) Date: Mon Dec 29 12:00:38 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <200312291753.09793.aleaxit@yahoo.com> Message-ID: On Mon, 29 Dec 2003, Alex Martelli wrote: > On Monday 29 December 2003 05:06 pm, Martin v. Loewis wrote: > ... > > It might be possible to write a generator that produces a makefile > > automatically from the VC.NET project files; contributions > > At the time of Visual Studio 6, such a "generator" had been written > by Microsoft, and was part of the VS6 package -- i.e., you could produce > and save a makefile for the project you had loaded. Is this no more the case > for the current release of Visual Studio...? > > > Alex > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/logistix%40cathoderaymission.net > Yes, they removed that functionality (or at least I wasn't able to find it). I used Visual Studio 6.0 to generate a good makefile in my tests. In VC 7.1, you can feed the project file into the compiler via a command line flag. I think you needed to use the GUI to build in 6.0, so MS provided the makefile export feature for people who wanted to do automated builds and other stuff where you couldn't rely on the gui. So the feature became obsolete from MS's perspective. Unfortunately, I don't think the command-line compilers in the SDK will accept a Visual Studio project file. From martin at v.loewis.de Mon Dec 29 12:03:20 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 12:04:23 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <200312291753.09793.aleaxit@yahoo.com> References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> Message-ID: <3FF05E58.1070108@v.loewis.de> Alex Martelli wrote: > At the time of Visual Studio 6, such a "generator" had been written > by Microsoft, and was part of the VS6 package -- i.e., you could produce > and save a makefile for the project you had loaded. Is this no more the case > for the current release of Visual Studio...? No. Microsoft has dropped that feature (perhaps they will restore it in VS.NET 2005 or something :-( However, it is now simpler to process project files; they are XML files with a fairly obvious vocabulary (*). A friend of mine once wrote a generator that produces GNU makefiles out of vcproj files; such generators tend to be quite application-specific, as they need to take custom build steps into account, and translate them properly. Most likely, the generator we would use would not be useful outside Python. I would like to see a single makefile generated, instead of one per project, as the VC6 generator would do; having so many makefiles is painful. That makefile should have an "all" target in addition to the targets for the individual projects. Regards, Martin (*) OTOH, the "solution" files (formerly workspaces) don't use XML syntax, and have changed between VS.NET 2002 and VS.NET 2003. They would need to be considered as well, as they contain the inter-project dependencies (including the list of all projects in the solution). From martin at v.loewis.de Mon Dec 29 12:24:57 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 12:25:42 2003 Subject: [Python-Dev] str.ascii_lower Message-ID: <3FF06369.9020508@v.loewis.de> Looking at python.org/sf/866982, I find it troubling that there are languages where "I".lower() != "i" (for those of you not familiar with Turkish: the lower-case letter of "I" is U+0131, LATIN SMALL LETTER DOTLESS I, which is \xfd in iso-8859-9). As a solution, I'd like to propose a new method ascii_lower, which is locale-unaware and only works for bytes 65..90 (returning the byte itself for all other characters). Similarly, ascii_upper might be needed "for symmetry"; I don't know whether the symmetry should extend beyond those two. This, in turn, should be used inside the codecs library where encoding names are normalized to lower case. What do you think? Regards, Martin From guido at python.org Mon Dec 29 12:37:37 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 29 12:37:42 2003 Subject: [Python-Dev] str.ascii_lower In-Reply-To: Your message of "Mon, 29 Dec 2003 18:24:57 +0100." <3FF06369.9020508@v.loewis.de> References: <3FF06369.9020508@v.loewis.de> Message-ID: <200312291737.hBTHbbM11741@c-24-5-183-134.client.comcast.net> > Looking at python.org/sf/866982, I find it troubling that > there are languages where "I".lower() != "i" > (for those of you not familiar with Turkish: the lower-case > letter of "I" is U+0131, LATIN SMALL LETTER DOTLESS I, > which is \xfd in iso-8859-9). > > As a solution, I'd like to propose a new method ascii_lower, > which is locale-unaware and only works for bytes 65..90 > (returning the byte itself for all other characters). > > Similarly, ascii_upper might be needed "for symmetry"; > I don't know whether the symmetry should extend beyond > those two. > > This, in turn, should be used inside the codecs library > where encoding names are normalized to lower case. > > What do you think? I never though there were locales possible that affected the mappings inside ASCII either. But shouldnt' this work just as well if it's only for encoding names (which I'd hope would be ASCII themselves): def ascii_lower(s): return str(unicode(s).lower()) The unicode() call converts ASCII to Unicode, which should always work for encoding names, and the Unicode lower() is locale-independent. This seems more elegant than adding yet more methods to the str type. --Guido van Rossum (home page: http://www.python.org/~guido/) From jepler at unpythonic.net Mon Dec 29 12:40:08 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Mon Dec 29 12:40:11 2003 Subject: [Python-Dev] str.ascii_lower In-Reply-To: <3FF06369.9020508@v.loewis.de> References: <3FF06369.9020508@v.loewis.de> Message-ID: <20031229174008.GH6171@unpythonic.net> On Mon, Dec 29, 2003 at 06:24:57PM +0100, Martin v. Loewis wrote: > Looking at python.org/sf/866982, I find it troubling that > there are languages where "I".lower() != "i" > (for those of you not familiar with Turkish: the lower-case > letter of "I" is U+0131, LATIN SMALL LETTER DOTLESS I, > which is \xfd in iso-8859-9). This post caused me to notice the following behavior. Is it "right"? >>> import locale >>> locale.setlocale(locale.LC_CTYPE, "tr_TR") 'tr_TR' >>> locale.getlocale()[1] # Expected charset 'ISO8859-9' >>> "I".lower() # Expected behavior '\xfd' >>> u"I".lower() # Python bug? (should be u'\u0131') u'i' >>> locale.setlocale(locale.LC_CTYPE, "tr_TR.UTF-8") 'tr_TR.UTF-8' >>> "I".lower() # C library bug? (should be "\xc4\xb1")* 'I' >>> locale.setlocale(locale.LC_CTYPE, "en_US.UTF-8") 'en_US.UTF-8' >>> "I".lower() # (UTF-8 locale works properly in english) 'i' Jeff * RedHat 9, glibc-2.3.2-11.9 From skip at pobox.com Mon Dec 29 09:01:28 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 29 12:42:06 2003 Subject: [Python-Dev] Optimization versus code bloat In-Reply-To: <20031229081724.GA42076@i18n.org> References: <20031229081724.GA42076@i18n.org> Message-ID: <16368.13240.502549.132898@montanaro.dyndns.org> Hye-Shik> I posted a patch[1] optimizing str.split with specialized Hye-Shik> splitter as unicode.split do. The patch accellates str.split Hye-Shik> about 10~20% and adds about 50 lines. I'm curious whether it's Hye-Shik> acceptable ratio for python-dev moods. Hye-Shik> [1] http://www.python.org/sf/866875 Seems like a reasonable addition to me, especially if the same optimization is performed for unicode objects. I'm a bit confused about all the changes to test_string.py though. Since your change is only for performance it seems to me that no changes to the test suite would have been necessary unless it wasn't covering some critical tests. Was that just a rearrangement of that file or did you add some new test cases? Whether or not your optimization is accepted, if your test cases beef up the test suite, I'd vote to add them. Skip From pje at telecommunity.com Mon Dec 29 12:44:46 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Dec 29 12:46:08 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF059A1.8090202@v.loewis.de> References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> Message-ID: <5.1.1.6.0.20031229123908.023122e0@telecommunity.com> At 05:43 PM 12/29/03 +0100, Martin v. Loewis wrote: OTOH, I cannot see what the problem might be - except that you will >have to link with msvcr71.dll at the end, not with msvcrt40.dll. > >>If this hasn't been resolved, and somebody can send me a binary for a >>.NET-built Python, I'll be happy to test. > >Sure: Please try my installer at > >http://www.dcl.hpi.uni-potsdam.de/home/loewis/python2.4.0.msi > >Notice that this doesn't include msvcr71.dll, which I could provide >in private. Please do. >It also does not include msvcr71.lib - I have no clue how you would >get a copy of that. But then, somehow, msvcrt40.lib (or is it >msvcrt40.a) must have gotten into mingw32 also. Actually, one generates that by using dlltool and various other programs to extract a .def file from python2x.dll. I haven't generated one in ages, so I'm going to have to dig back up how to do that. The rest is handled by the distutils, pretty much. I don't think there's actually any direct linking to the MS VC runtime, anyway, if one is only using Python API calls. But I guess I'll find out. Hopefully I'll have enough clock cycles to spare on Thursday to get everything set up and tested. From guido at python.org Mon Dec 29 12:47:39 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 29 12:47:44 2003 Subject: [Python-Dev] str.ascii_lower In-Reply-To: Your message of "Mon, 29 Dec 2003 11:40:08 CST." <20031229174008.GH6171@unpythonic.net> References: <3FF06369.9020508@v.loewis.de> <20031229174008.GH6171@unpythonic.net> Message-ID: <200312291747.hBTHldO11794@c-24-5-183-134.client.comcast.net> > This post caused me to notice the following behavior. Is it "right"? > > >>> import locale > >>> locale.setlocale(locale.LC_CTYPE, "tr_TR") > 'tr_TR' > >>> locale.getlocale()[1] # Expected charset > 'ISO8859-9' > >>> "I".lower() # Expected behavior > '\xfd' > >>> u"I".lower() # Python bug? (should be u'\u0131') > u'i' Why? Unicode strings are not affected by the locale. > >>> locale.setlocale(locale.LC_CTYPE, "tr_TR.UTF-8") > 'tr_TR.UTF-8' > >>> "I".lower() # C library bug? (should be "\xc4\xb1")* > 'I' > >>> locale.setlocale(locale.LC_CTYPE, "en_US.UTF-8") > 'en_US.UTF-8' > >>> "I".lower() # (UTF-8 locale works properly in english) > 'i' I have no idea what adding UTF8 to the local means. Is this something that Python's locale-awareness does or is it simply recognized by the C library? --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Mon Dec 29 12:59:18 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 13:00:16 2003 Subject: [Python-Dev] str.ascii_lower In-Reply-To: <200312291737.hBTHbbM11741@c-24-5-183-134.client.comcast.net> References: <3FF06369.9020508@v.loewis.de> <200312291737.hBTHbbM11741@c-24-5-183-134.client.comcast.net> Message-ID: <3FF06B76.3010004@v.loewis.de> Guido van Rossum wrote: > But shouldnt' this work just as well if it's only for encoding names > (which I'd hope would be ASCII themselves): > > def ascii_lower(s): > return str(unicode(s).lower()) Certainly. Another implementation would be lower_map = string.maketrans(string.ascii_upper, string.ascii_lower) def ascii_lower(str): return str.translate(lower_map) > This seems more elegant than adding yet more methods to the str type. Ok. I'll fix it in this direction, then. There would be two copies of this function that I can see: One for the codecs (which uses tolower() in C code), and one in the email module. And yes, codec names must be pure ASCII. Regards, Martin From martin at v.loewis.de Mon Dec 29 13:04:56 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 13:05:06 2003 Subject: [Python-Dev] str.ascii_lower In-Reply-To: <20031229174008.GH6171@unpythonic.net> References: <3FF06369.9020508@v.loewis.de> <20031229174008.GH6171@unpythonic.net> Message-ID: <3FF06CC8.5070705@v.loewis.de> Jeff Epler wrote: >>>>u"I".lower() # Python bug? (should be u'\u0131') > > u'i' As Guido says: unicode.tolower is locale-inaware; it uses the Unicode Consortium character properties instead to determine the lower-case character. >>>>"I".lower() # C library bug? (should be "\xc4\xb1")* > > 'I' This is really a limitation of the C language, not of the C library. The interface is char tolower(char input); so it can only accept and return a single char. Multi-byte characters are not supported in that interface. Traditionally, for characters that cannot be converted, tolower returns its argument. >>>>"I".lower() # (UTF-8 locale works properly in english) > > 'i' This is because "i" is a single byte in UTF-8. Regards, Martin From jepler at unpythonic.net Mon Dec 29 13:09:27 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Mon Dec 29 13:09:30 2003 Subject: [Python-Dev] str.ascii_lower In-Reply-To: <200312291747.hBTHldO11794@c-24-5-183-134.client.comcast.net> References: <3FF06369.9020508@v.loewis.de> <20031229174008.GH6171@unpythonic.net> <200312291747.hBTHldO11794@c-24-5-183-134.client.comcast.net> Message-ID: <20031229180927.GI6171@unpythonic.net> I stand corrected about the behavior of Unicode in the presence of locales. On Mon, Dec 29, 2003 at 09:47:39AM -0800, Guido van Rossum wrote: > > >>> locale.setlocale(locale.LC_CTYPE, "tr_TR.UTF-8") > > 'tr_TR.UTF-8' > > >>> "I".lower() # C library bug? (should be "\xc4\xb1")* > > 'I' > > >>> locale.setlocale(locale.LC_CTYPE, "en_US.UTF-8") > > 'en_US.UTF-8' > > >>> "I".lower() # (UTF-8 locale works properly in english) > > 'i' > > I have no idea what adding UTF8 to the local means. Is this something > that Python's locale-awareness does or is it simply recognized by the > C library? "A locale name is typically of the form language[_territory] [.code-set][@modifier]" -- man setlocale() on my system RedHat 9 made a halfhearted attempt to use UTF-8 as the encoding for all locales. So it sets LANG=en_US.UTF-8 by default. In theory, tr_TR.UTF_8 should be the Turkish locale with UTF-8 characters, but it behaves incorrectly by having "I".lower() == "I". Well, since my earlier post combined a misunderstanding of how Python works with a possible C library bug, I guess I raised two non-issues. Sorry for wasting everyone's time. Jeff From martin at v.loewis.de Mon Dec 29 13:09:40 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 13:09:59 2003 Subject: [Python-Dev] str.ascii_lower In-Reply-To: <200312291747.hBTHldO11794@c-24-5-183-134.client.comcast.net> References: <3FF06369.9020508@v.loewis.de> <20031229174008.GH6171@unpythonic.net> <200312291747.hBTHldO11794@c-24-5-183-134.client.comcast.net> Message-ID: <3FF06DE4.1010406@v.loewis.de> Guido van Rossum wrote: > I have no idea what adding UTF8 to the local means. Is this something > that Python's locale-awareness does or is it simply recognized by the > C library? It's recognized by the C library, both on Windows and Unix. The full format is _@. By default, each implies a default , which can be overridden. On Linux, is a IANA name (e.g. de_DE.ISO-8859-15); on Windows, it is a code page (e.g. german_Germany.1252). Regards, Martin From paoloinvernizzi at dmsware.com Mon Dec 29 15:30:07 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Mon Dec 29 15:30:28 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3FF05E58.1070108@v.loewis.de> References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> <3FF05E58.1070108@v.loewis.de> Message-ID: <3FF08ECF.4010302@dmsware.com> Martin v. Loewis wrote: > No. Microsoft has dropped that feature (perhaps they will restore it > in VS.NET 2005 or something :-( > > However, it is now simpler to process project files; they are XML > files with a fairly obvious vocabulary (*). A friend of mine once > wrote a generator that produces GNU makefiles out of vcproj files; > such generators tend to be quite application-specific, as they > need to take custom build steps into account, and translate them > properly. Most likely, the generator we would use would not be > useful outside Python. I can suggest the fact that Scons can handle multiple builders (cygwin, mingw, VC6, VC7, VC7.1) easily and for VC it can generate the proper Project files? Can be a solution to support a single scons (makefile like) file and from that generate the proper project files? Users that wants to keep working with the IDE can do that! A free solution for a user that want to build python in a win32 environment without Visual Studio (whatever edition) could be: - Download python (from 1.5.2 above) and scons (as a make replacement) - Build python with the free Microsoft SDK compiler via scons. Scons has also an integrated autoconf-like environment, so it's possible to have a path to unify the classic configure/make *nix way with the win32 one. BTW, scons makefiles are just python files! Think only about the *update the python build number* problem that arose some times ago! It's trivial if handled by a builder script with-python-steroids! --- Paolo Invernizzi From martin at v.loewis.de Mon Dec 29 16:55:38 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 16:56:00 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3FF08ECF.4010302@dmsware.com> References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> <3FF05E58.1070108@v.loewis.de> <3FF08ECF.4010302@dmsware.com> Message-ID: <3FF0A2DA.4010205@v.loewis.de> Paolo Invernizzi wrote: > Can be a solution to support a single scons (makefile like) file and > from that generate the proper project files? That might be an option, but not the one I'm working on; it isn't even one that I'm particularly interested in. If I were maintaining the Windows port (which I'm not), I would dislike the solution because it would mean I have to learn scons... > Users that wants to keep working with the IDE can do that! ... and it would also mean that I *cannot* work with the IDE: I cannot make modifications to the projects in the IDE, because I would have to make the modifications in the scons files, not in the VC projects. > A free solution for a user that want to build python in a win32 > environment without Visual Studio (whatever edition) could be: > > - Download python (from 1.5.2 above) and scons (as a make replacement) > - Build python with the free Microsoft SDK compiler via scons. Such a free solution is already available, through cygwin/mingw. However, this is free software, so if you are interested, feel free to contribute. If you want your contribution be accepted, be prepared to answer "yes" to the question "Are you going to maintain the build process for the next three years"? If somebody volunteers to maintain such a build process, I would not object to having this process in parallel to the VC build process (to which I'm willing to contribute, for the next three years). Regards, Martin From paoloinvernizzi at dmsware.com Mon Dec 29 17:23:01 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Mon Dec 29 17:23:01 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3FF0A2DA.4010205@v.loewis.de> References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> <3FF05E58.1070108@v.loewis.de> <3FF08ECF.4010302@dmsware.com> <3FF0A2DA.4010205@v.loewis.de> Message-ID: <3FF0A945.2020107@dmsware.com> Martin v. Loewis wrote: > If I were maintaining the > Windows port (which I'm not), I would dislike the solution because it > would mean I have to learn scons... It is only an alternative to learning a new tool for converting VC project files to nmake specific makefile. > I cannot > make modifications to the projects in the IDE, because I would have > to make the modifications in the scons files, not in the VC projects. That's true. You have to make the modifications in the scons file, but that modifications (usually adding or removing some c module, or I'm missing something?) can be used by VC/cygwin/mgwin/*nix user... potentially. > Such a free solution is already available, through cygwin/mingw. But that leave you with the original problem... makefiles for the free microsoft compiler... > However, this is free software, so if you are interested, feel free > to contribute. If you want your contribution be accepted, be prepared > to answer "yes" to the question "Are you going to maintain the build > process for the next three years"? > If somebody volunteers to maintain such a build process, I would not > object to having this process in parallel to the VC build process > (to which I'm willing to contribute, for the next three years). I'm not actually in the position to reply "yes" to that question, I have too few spare time... but you are tempting me... :-) It was just an alternative suggestion that come in my mind after having read previous threads. I've done some month ago a scratch scons script for VC6 and VC7 compilation of the core python dll, pythonw.exe and python.exe... I must take a look to my messy sandbox at works! --- Paolo Invernizzi From perky at i18n.org Mon Dec 29 19:34:08 2003 From: perky at i18n.org (Hye-Shik Chang) Date: Mon Dec 29 19:34:14 2003 Subject: [Python-Dev] Optimization versus code bloat In-Reply-To: <16368.13240.502549.132898@montanaro.dyndns.org> References: <20031229081724.GA42076@i18n.org> <16368.13240.502549.132898@montanaro.dyndns.org> Message-ID: <20031230003408.GA17100@i18n.org> On Mon, Dec 29, 2003 at 08:01:28AM -0600, Skip Montanaro wrote: > > Hye-Shik> I posted a patch[1] optimizing str.split with specialized > Hye-Shik> splitter as unicode.split do. The patch accellates str.split > Hye-Shik> about 10~20% and adds about 50 lines. I'm curious whether it's > Hye-Shik> acceptable ratio for python-dev moods. > > Hye-Shik> [1] http://www.python.org/sf/866875 > > Seems like a reasonable addition to me, especially if the same optimization > is performed for unicode objects. I'm a bit confused about all the changes > to test_string.py though. Since your change is only for performance it > seems to me that no changes to the test suite would have been necessary > unless it wasn't covering some critical tests. Was that just a > rearrangement of that file or did you add some new test cases? Nah. Because I added one more specialized splitter, I should add unittests for that subsequently. So we need to test each test cases for every specialized splitters (whitespace, string and a character). Thanks for your review! :-) Hye-Shik From martin at v.loewis.de Mon Dec 29 20:00:36 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 20:00:53 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3FF0A945.2020107@dmsware.com> References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> <3FF05E58.1070108@v.loewis.de> <3FF08ECF.4010302@dmsware.com> <3FF0A2DA.4010205@v.loewis.de> <3FF0A945.2020107@dmsware.com> Message-ID: <3FF0CE34.4060409@v.loewis.de> Paolo Invernizzi wrote: >> If I were maintaining the >> Windows port (which I'm not), I would dislike the solution because it >> would mean I have to learn scons... > > > It is only an alternative to learning a new tool for converting VC > project files to nmake specific makefile. It's different. That tool would have to be written from scratch, which would be a large effort for a single person. Learning scons would be a relatively small effort for a relatively large group of people (not only immediate maintainers of the Windows port, but anybody contributing to it, also). > That's true. You have to make the modifications in the scons file, but > that modifications (usually adding or removing some c module, or I'm > missing something?) can be used by VC/cygwin/mgwin/*nix user... > potentially. The more typical modification is the addition of an extension module, which means adding a new target. That becomes more and more tricky these days, as these new targets often have prerequisites that are difficult to build themselves (e.g. finding an available copy of OpenSSL, invoking Perl to build OpenSSL, invoking nmake to build zlib, etc). I somewhat doubt that the scons-to-vcproj generator gets such complex build dependencies correct - at which point the maintainers would need to look into the inner workings of scons to find out if it really can represent the build step in all supported "output build procedures", and how to make scons do it correctly. > But that leave you with the original problem... makefiles for the free > microsoft compiler... That, of course, is not a problem for the majority of the Python-Windows contributors, who already have a copy of VC 7.1 (thanks to a generous offer earlier this year). So for whoever this is a problem, they would need to find a solution. They need to find a way today already (as Python does not provide nmake files out of the box at the moment, either). The solution does not need to materialize itself next week, as the release of Python 2.4 is still months away. > I've done some month ago a scratch scons script for VC6 and VC7 > compilation of the core python dll, pythonw.exe and python.exe... > > I must take a look to my messy sandbox at works! You could also maintain this "outside" of the standard Python distribution, as some kind of "drop-in" zip file. That would put no obligation onto you - your users could either use it, or leave it (and we could advertise it as an alternative somewhere). Regards, Martin From pf_moore at yahoo.co.uk Tue Dec 30 06:45:00 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Tue Dec 30 06:44:54 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 References: <3FF047E0.8070905@v.loewis.de> <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> Message-ID: "Phillip J. Eby" writes: > At 04:27 PM 12/29/03 +0100, Martin v. Loewis wrote: >>In my own sandbox, I have prepared project files for VC.NET. >> >>Unless there are any strong objections, I'd like to switch >>over to VC.NET shortly, disabling VC6 builds. In the process, >>I will also update the build dependencies to more recent >>versions of several packages. > > Was the question of mingw32-built extensions ever resolved? That is, > will we be able to build extensions for the standard Windows Python > using the distutils' "mingw32" compiler, as is possible with Python > 2.2? It is (pretty close) to being solved - recent versions of mingw32 allow building with msvcr71 via a -lmsvcr71 flag. However, last time I tried this, the generated linker commands didn't quite work, and needed a bit of fiddling. I have the latest mingw, though, so I can easily try this. Paul -- This signature intentionally left blank From pf_moore at yahoo.co.uk Tue Dec 30 06:49:36 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Tue Dec 30 06:50:22 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> Message-ID: "Martin v. Loewis" writes: > OTOH, I cannot see what the problem might be - except that you will > have to link with msvcr71.dll at the end, not with msvcrt40.dll. There's an issue with mingw using malloc/free from msvcrt in its startup code - by the time msvcr71 gets linked in, the startup code has already resolved these two to msvcrt. I believe this is (nearly) resolved. Also, I've never seen a real problem caused by this, just dire hearsay about problems using incompatible runtimes... >> If this hasn't been resolved, and somebody can send me a binary for >> a .NET-built Python, I'll be happy to test. > > Sure: Please try my installer at > > http://www.dcl.hpi.uni-potsdam.de/home/loewis/python2.4.0.msi Does this install cleanly alongside a "production" Python 2.3? Ie, will it leave the meaning of the "python" command, and command line associations for .py files, etc, unchanged? As a test install, I'd like it to have no effect on my main Python (I have no test machine to install on separately). > Notice that this doesn't include msvcr71.dll, which I could provide > in private. Not a problem for me - I have this. > It also does not include msvcr71.lib - I have no clue how you would > get a copy of that. But then, somehow, msvcrt40.lib (or is it > msvcrt40.a) must have gotten into mingw32 also. The appropriate library is indeed supplied with mingw. Paul. -- This signature intentionally left blank From pf_moore at yahoo.co.uk Tue Dec 30 07:02:00 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Tue Dec 30 07:01:54 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> <5.1.1.6.0.20031229123908.023122e0@telecommunity.com> Message-ID: "Phillip J. Eby" writes: > The rest is handled by the distutils, pretty much. I don't think > there's actually any direct linking to the MS VC runtime, anyway, if > one is only using Python API calls. But I guess I'll find out. One thing that I imagine does need looking at is modifying distutils to check whether Python was built with VC7.1, and if so add a -lmsvcr71 flag to the gcc command line when compiling with mingw. This can be hacked by hand, but only at the expense of making setup.py version-specific (or doing your own test in setup.py). I don't know if Martin has already done this, but it needs doing. I'm not a distutils expert, but I am willing to look at it in the longer term. BTW, on another note, which 3rd party package developers have access to MSVC7.1, or can build with Mingw? Has anyone surveyed this? I, for one, have installed the following packages which have extension modules involved: cx_Oracle mxBase PIL pygame (not used much) win32all twisted wxPython pyXML Metakit If there wasn't a Windows binary version for 2.4 produced, this would cause me problems. At the very least, I'd suggest a warning post on c.l.p and c.l.p.announce, something to the effect of "Python 2.4 will be built with MSVC 7.1. Extension developers supplying binary distributions for Windows will need some way of building MSVC 7.1 compatible modules (MSVC 7.1 itself, or a recent version of the free Mingw compiler package) to continue providing binries." Paul. -- This signature intentionally left blank From skip at pobox.com Tue Dec 30 10:03:55 2003 From: skip at pobox.com (Skip Montanaro) Date: Tue Dec 30 10:04:05 2003 Subject: [Python-Dev] urllib2 doesn't grok URLs w/ user/passwd Message-ID: <16369.37851.229594.332869@montanaro.dyndns.org> SF seems to be down for some unscheduled reason. Posting here just so I don't completely forget about it should I exit my web browser before SF is back up... urllib2.urlopen("http://foo@www.python.org/") fails (at least in part) because it fails to separate the username and password from the hostname. Trying to open http://foo:bar@www.python.org/ reveals other shortcomings in its url parsing. It seems to me the syntactic bits shouldn't be difficult to resolve using urllib.spluituser(). I'm much less clear what to do with the username and password once they've been separated from the hostname. Skip From theller at python.net Tue Dec 30 11:10:09 2003 From: theller at python.net (Thomas Heller) Date: Tue Dec 30 10:10:55 2003 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/msi msi.py, 1.1.1.1, 1.2 msilib.py, 1.1.1.1, 1.2 In-Reply-To: (loewis@users.sourceforge.net's message of "Tue, 30 Dec 2003 07:02:45 -0800") References: Message-ID: loewis@users.sourceforge.net writes: > Update of /cvsroot/python/python/nondist/sandbox/msi > In directory sc8-pr-cvs1:/tmp/cvs-serv15532 > > Modified Files: > msi.py msilib.py > root.start_component("TARGETDIR", default_feature) > root.add_file("PCBuild/w9xpopen.exe") # XXX: separate component to only install on W9x Currently in the Wise script, w9xpopen.exe is also installed on NT/2k/XP, and I would like to leave it this way. py2exe needs it to create programs using popen that also run on w9x. Thomas From theller at python.net Tue Dec 30 11:25:17 2003 From: theller at python.net (Thomas Heller) Date: Tue Dec 30 10:26:01 2003 Subject: [Python-Dev] First version of Python MSI available In-Reply-To: <3FF04774.9020404@v.loewis.de> (Martin v. Loewis's message of "Mon, 29 Dec 2003 16:25:40 +0100") References: <3FF04774.9020404@v.loewis.de> Message-ID: "Martin v. Loewis" writes: > I have committed python/nondist/sandbox/msi to the CVS. > This is a MSI generator for Python on Windows, which I'd > like to propose as a replacement for Wise. A sample .msi > file is available at > > http://www.dcl.hpi.uni-potsdam.de/home/loewis/python2.4.0.msi > > This installer includes binaries compiled with MSVC 7.1 > (aka VC.NET 2003). > > The MSI file tries to copy most features and user interface > from the Wise installer, but adjusts to the feature set of MSI. > I'm aware of a few issues, but I'd like to request feedback > in any case. I still have MSVC 7.0 (not 7.1) installed, and trying to build an extension with distutils raises this error: running build_ext Traceback (most recent call last): File "setup.py", line 332, in ? options = options, File "c:\python24\lib\distutils\core.py", line 149, in setup dist.run_commands() File "c:\python24\lib\distutils\dist.py", line 907, in run_commands self.run_command(cmd) File "c:\python24\lib\distutils\dist.py", line 927, in run_command cmd_obj.run() File "c:\python24\lib\distutils\command\build.py", line 107, in run self.run_command(cmd_name) File "c:\python24\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "c:\python24\lib\distutils\dist.py", line 927, in run_command cmd_obj.run() File "c:\python24\lib\distutils\command\build_ext.py", line 243, in run force=self.force) File "c:\python24\lib\distutils\ccompiler.py", line 1177, in new_compiler return klass (None, dry_run, force) File "c:\python24\lib\distutils\msvccompiler.py", line 206, in __init__ self.__macros = MacroExpander(self.__version) File "c:\python24\lib\distutils\msvccompiler.py", line 112, in __init__ self.load_macros(version) File "c:\python24\lib\distutils\msvccompiler.py", line 128, in load_macros self.set_macro("FrameworkSDKDir", net, "sdkinstallrootv1.1") File "c:\python24\lib\distutils\msvccompiler.py", line 118, in set_macro self.macros["$(%s)" % macro] = d[key] KeyError: 'sdkinstallrootv1.1' > c:\python24\lib\distutils\msvccompiler.py(118)set_macro() -> self.macros["$(%s)" % macro] = d[key] Here is a dump of some variable: (Pdb) print d {'dbgmanageddebugger': 'C:\\Programme\\Gemeinsame Dateien\\Microsoft Shared\\VS7Debug\\vs7jit.exe PID %d APPDOM %d EXTEXT "%s" EVTHDL %d', 'dbgjitdebuglaunchsetting': 2, 'installroot': 'C:\\WINDOWS\\Microsoft.NET\\Framework\\', 'sdkinstallroot': 'C:\\Programme\\Microsoft Visual Studio .NET\\FrameworkSDK\\'} (Pdb) print key sdkinstallrootv1.1 (Pdb) print macro FrameworkSDKDir (Pdb) Thomas From theller at python.net Tue Dec 30 11:27:40 2003 From: theller at python.net (Thomas Heller) Date: Tue Dec 30 10:28:22 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: (Paul Moore's message of "Tue, 30 Dec 2003 11:49:36 +0000") References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> Message-ID: Paul Moore writes: > "Martin v. Loewis" writes: > >> Sure: Please try my installer at >> >> http://www.dcl.hpi.uni-potsdam.de/home/loewis/python2.4.0.msi > > Does this install cleanly alongside a "production" Python 2.3? Ie, > will it leave the meaning of the "python" command, and command line > associations for .py files, etc, unchanged? No, it changes the associations for .py files. It would be useful to have the option to create no associations. As a workaround, installing the MSI file, and afterwards simply reinstalling python-2.3.3.exe seems to work fine. Thomas From barry at python.org Tue Dec 30 10:41:25 2003 From: barry at python.org (Barry Warsaw) Date: Tue Dec 30 10:41:33 2003 Subject: [Python-Dev] Head broken when --enable-unicode=ucs4 is used Message-ID: <1072798884.9216.77.camel@anthem> I've been building Python with --enable-unicode=ucs4 ever since I upgraded to RedHat 9. IIUC, this option is required to get Python and RH9's default Tk to play nicely together. The Python cvs trunk (i.e. 2.4) is broken when using this option. The pain starts immediately: gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o Modules/python.o Modules/python.c In file included from Include/Python.h:82, from Modules/python.c:3: Include/unicodeobject.h:73:2: #error Must define Py_UNICODE_SIZE In file included from Include/Python.h:82, from Modules/python.c:3: Include/unicodeobject.h:131: parse error before "Py_UNICODE" Include/unicodeobject.h:131: warning: type defaults to `int' in declaration of `Py_UNICODE' Include/unicodeobject.h:131: warning: data definition has no type or storage class Include/unicodeobject.h:370: parse error before "Py_UNICODE" Include/unicodeobject.h:370: warning: no semicolon at end of struct or union Include/unicodeobject.h:375: parse error before '}' token Include/unicodeobject.h:375: warning: type defaults to `int' in declaration of `PyUnicodeObject' Include/unicodeobject.h:375: warning: data definition has no type or storage class [...continuing on for seemingly ever...] Without this switch, the compilation appears to work. I don't have time right now to investigate further, but if no one else can reproduce (or fix) this, I'll try to help out later. -Barry From aleaxit at yahoo.com Tue Dec 30 11:11:44 2003 From: aleaxit at yahoo.com (Alex Martelli) Date: Tue Dec 30 11:11:51 2003 Subject: [Python-Dev] urllib2 doesn't grok URLs w/ user/passwd In-Reply-To: <16369.37851.229594.332869@montanaro.dyndns.org> References: <16369.37851.229594.332869@montanaro.dyndns.org> Message-ID: <200312301711.44687.aleaxit@yahoo.com> On Tuesday 30 December 2003 04:03 pm, Skip Montanaro wrote: > SF seems to be down for some unscheduled reason. Posting here just so I > don't completely forget about it should I exit my web browser before SF is > back up... > > urllib2.urlopen("http://foo@www.python.org/") fails (at least in part) > because it fails to separate the username and password from the hostname. > Trying to open http://foo:bar@www.python.org/ reveals other shortcomings in > its url parsing. It seems to me the syntactic bits shouldn't be difficult > to resolve using urllib.spluituser(). I'm much less clear what to do with > the username and password once they've been separated from the hostname. Presumably they need to be kept somewhere and sent in the Authorization header in case the server returns a 401 error and challenge (or a proxy returns a 407 error and challenge) -- or maybe the Authorization header (with the base 64 encoding of user:pass) can be sent even as part of the first request to speed things up (assuming an authorization scheme of Basic). RFC 2617, I believe. urllib2's architecture delegates authorization to separate components, of course, so I guess the userid and password should just be handed over to such components if they're present, but I haven't looked into that in detail. Alex From kdart at kdart.com Tue Dec 30 13:01:42 2003 From: kdart at kdart.com (Keith Dart) Date: Tue Dec 30 13:01:46 2003 Subject: [Python-Dev] feedback on Enum class Message-ID: <1072807302.11336.37.camel@leviathan.kdart.com> Greetings everyone. I hope you all had a great Christmas holiday. 8-) I am posting this Enum (named number) class to this list for possible feedback. It can be used as a regular integer but when printed (stringified) it yeilds its name. The hash also makes named numbers with the same integer value into unique dictionary keys. That is the part I am not sure about... Comments appreciated. class Enum(int): __slots__ = ("_name") def __new__(cls, val, name): v = int.__new__(cls, val) v._name = str(name) return v def __str__(self): return self._name def __repr__(self): return "%s(%d, %r)" % (self.__class__.__name__, self,self._name) def __hash__(self): return int.__hash__(self) + hash(self._name) -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031230/df76d679/attachment.bin From tismer at stackless.com Tue Dec 30 13:38:29 2003 From: tismer at stackless.com (Christian Tismer) Date: Tue Dec 30 13:38:35 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF047E0.8070905@v.loewis.de> References: <3FF047E0.8070905@v.loewis.de> Message-ID: <3FF1C625.7070103@stackless.com> Martin v. Loewis wrote: > In my own sandbox, I have prepared project files for VC.NET. > > Unless there are any strong objections, I'd like to switch > over to VC.NET shortly, disabling VC6 builds. In the process, > I will also update the build dependencies to more recent > versions of several packages. Ach du Sch***e. I don't want to buy another M$ compiler. Is there a freeware alternative? ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From paoloinvernizzi at dmsware.com Tue Dec 30 13:41:34 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Tue Dec 30 13:41:36 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3FF0CE34.4060409@v.loewis.de> References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> <3FF05E58.1070108@v.loewis.de> <3FF08ECF.4010302@dmsware.com> <3FF0A2DA.4010205@v.loewis.de> <3FF0A945.2020107@dmsware.com> <3FF0CE34.4060409@v.loewis.de> Message-ID: Martin v. Loewis wrote: > The more typical modification is the addition of an extension module, > which means adding a new target. That becomes more and more tricky > these days, as these new targets often have prerequisites that are > difficult to build themselves (e.g. finding an available copy of > OpenSSL, invoking Perl to build OpenSSL, invoking nmake to build > zlib, etc). Ok. For what I have understood reading the PCbuild/readme.txt (I admit I never tried to build on win32 an *hard* module, like tk or bdb, or ssl), the building process of that modules is still mostly a manual job... "follow the Windows instructions for building the Sleepycat software" > I somewhat doubt that the scons-to-vcproj generator gets such > complex build dependencies correct - at which point the maintainers > would need to look into the inner workings of scons to find out if > it really can represent the build step in all supported "output > build procedures", and how to make scons do it correctly. The VC project generated by scons simply binds the IDE build command to scons itself, so when the IDE need to build something, it actually launch scons, which can handle a lot of mutual dependend target easily. Well, summarizing all, I can try some *non trivial* build process, like the python exes plus core dll, plus out-of-the-box modules, plus OpenSSL or bdb, and see what can be done with it... I'll post something as soon as I've done some tries. --- Paolo Invernizzi From pf_moore at yahoo.co.uk Tue Dec 30 13:56:24 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Tue Dec 30 13:56:13 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> Message-ID: <3cb2xhgn.fsf@yahoo.co.uk> Thomas Heller writes: > No, it changes the associations for .py files. It would be useful to > have the option to create no associations. Agreed, that would be useful. (More generally, if there's anything else needed to install a second Python version which is never used by default, but only used when explicitly requested, that would be useful too.) Hmm, for example, does the installer set the registry key HKLM\Software\Microsoft\Windows\CurrentVersion\App Paths\python.exe That's what makes the command "python" work at the console prompt. Maybe just a single option in the installer, "Make this the default version of Python", which is true by default, would be best. This could control file associations, App Paths, and anything else related. > As a workaround, installing the MSI file, and afterwards simply > reinstalling python-2.3.3.exe seems to work fine. Yup, that's a good workaround. Paul. -- This signature intentionally left blank From guido at python.org Tue Dec 30 14:28:52 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 30 14:26:16 2003 Subject: [Python-Dev] urllib2 doesn't grok URLs w/ user/passwd In-Reply-To: Your message of "Tue, 30 Dec 2003 17:11:44 +0100." <200312301711.44687.aleaxit@yahoo.com> References: <16369.37851.229594.332869@montanaro.dyndns.org> <200312301711.44687.aleaxit@yahoo.com> Message-ID: <200312301928.hBUJSqu01449@c-24-5-183-134.client.comcast.net> > On Tuesday 30 December 2003 04:03 pm, Skip Montanaro wrote: > > SF seems to be down for some unscheduled reason. Posting here just so I > > don't completely forget about it should I exit my web browser before SF is > > back up... > > > > urllib2.urlopen("http://foo@www.python.org/") fails (at least in part) > > because it fails to separate the username and password from the hostname. > > Trying to open http://foo:bar@www.python.org/ reveals other shortcomings in > > its url parsing. It seems to me the syntactic bits shouldn't be difficult > > to resolve using urllib.spluituser(). I'm much less clear what to do with > > the username and password once they've been separated from the hostname. > > Presumably they need to be kept somewhere and sent in the Authorization > header in case the server returns a 401 error and challenge (or a proxy > returns a 407 error and challenge) -- or maybe the Authorization header > (with the base 64 encoding of user:pass) can be sent even as part of the > first request to speed things up (assuming an authorization scheme of > Basic). This is what the ever-popular old urllib does. > RFC 2617, I believe. urllib2's architecture delegates authorization > to separate components, of course, so I guess the userid and password > should just be handed over to such components if they're present, but I > haven't looked into that in detail. Me neither. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Dec 30 14:34:25 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 30 14:31:50 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: Your message of "Tue, 30 Dec 2003 19:38:29 +0100." <3FF1C625.7070103@stackless.com> References: <3FF047E0.8070905@v.loewis.de> <3FF1C625.7070103@stackless.com> Message-ID: <200312301934.hBUJYPL01492@c-24-5-183-134.client.comcast.net> > I don't want to buy another M$ compiler. > Is there a freeware alternative? Ten core Python developers received a free VC7.1 copy from Microsoft. Maybe someone who hasn't opened theirs yet is willing to step down? MS also alluded to a free downloadable compiler, but I haven't tracked it down yet. And of course theer's always mingw (or whatever it's called). --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Tue Dec 30 14:59:55 2003 From: aahz at pythoncraft.com (Aahz) Date: Tue Dec 30 14:59:59 2003 Subject: [Python-Dev] feedback on Enum class In-Reply-To: <1072807302.11336.37.camel@leviathan.kdart.com> References: <1072807302.11336.37.camel@leviathan.kdart.com> Message-ID: <20031230195954.GA26455@panix.com> On Tue, Dec 30, 2003, Keith Dart wrote: > > I am posting this Enum (named number) class to this list for possible > feedback. It can be used as a regular integer but when printed > (stringified) it yeilds its name. The hash also makes named numbers with > the same integer value into unique dictionary keys. That is the part I > am not sure about... Comments appreciated. This should probably go to comp.lang.python. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From theller at python.net Tue Dec 30 16:17:19 2003 From: theller at python.net (Thomas Heller) Date: Tue Dec 30 15:17:17 2003 Subject: [Python-Dev] First version of Python MSI available In-Reply-To: <3FF04774.9020404@v.loewis.de> (Martin v. Loewis's message of "Mon, 29 Dec 2003 16:25:40 +0100") References: <3FF04774.9020404@v.loewis.de> Message-ID: "Martin v. Loewis" writes: > I have committed python/nondist/sandbox/msi to the CVS. > This is a MSI generator for Python on Windows, which I'd > like to propose as a replacement for Wise. A sample .msi > file is available at > > http://www.dcl.hpi.uni-potsdam.de/home/loewis/python2.4.0.msi > > This installer includes binaries compiled with MSVC 7.1 > (aka VC.NET 2003). > > The MSI file tries to copy most features and user interface > from the Wise installer, but adjusts to the feature set of MSI. > I'm aware of a few issues, but I'd like to request feedback > in any case. The installer does not install pyconfig.h into Python24\include. After copying this file manually (and installing VC7.1 - puh) I'm now able to build extensions with distutils. Also distutils\command\wininst.exe is not installed. Thomas From kbk at shore.net Tue Dec 30 15:17:29 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Tue Dec 30 15:17:36 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <200312301934.hBUJYPL01492@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Tue, 30 Dec 2003 11:34:25 -0800") References: <3FF047E0.8070905@v.loewis.de> <3FF1C625.7070103@stackless.com> <200312301934.hBUJYPL01492@c-24-5-183-134.client.comcast.net> Message-ID: <87znda12na.fsf@hydra.localdomain> Guido van Rossum writes: > Ten core Python developers received a free VC7.1 copy from Microsoft. > Maybe someone who hasn't opened theirs yet is willing to step down? Maybe there are a few more available? -- KBK From guido at python.org Tue Dec 30 15:28:23 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 30 15:25:49 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: Your message of "Tue, 30 Dec 2003 15:17:29 EST." <87znda12na.fsf@hydra.localdomain> References: <3FF047E0.8070905@v.loewis.de> <3FF1C625.7070103@stackless.com> <200312301934.hBUJYPL01492@c-24-5-183-134.client.comcast.net> <87znda12na.fsf@hydra.localdomain> Message-ID: <200312302028.hBUKSNN01582@c-24-5-183-134.client.comcast.net> > > Ten core Python developers received a free VC7.1 copy from Microsoft. > > Maybe someone who hasn't opened theirs yet is willing to step down? > > Maybe there are a few more available? I'm not going to ask. --Guido van Rossum (home page: http://www.python.org/~guido/) From jjl at pobox.com Tue Dec 30 16:55:45 2003 From: jjl at pobox.com (John J Lee) Date: Tue Dec 30 16:56:05 2003 Subject: [Python-Dev] urllib2 doesn't grok URLs w/ user/passwd In-Reply-To: <200312301711.44687.aleaxit@yahoo.com> References: <16369.37851.229594.332869@montanaro.dyndns.org> <200312301711.44687.aleaxit@yahoo.com> Message-ID: On Tue, 30 Dec 2003, Alex Martelli wrote: > On Tuesday 30 December 2003 04:03 pm, Skip Montanaro wrote: > > SF seems to be down for some unscheduled reason. Posting here just so I > > don't completely forget about it should I exit my web browser before SF is > > back up... > > > > urllib2.urlopen("http://foo@www.python.org/") fails (at least in part) > > because it fails to separate the username and password from the hostname. > > Trying to open http://foo:bar@www.python.org/ reveals other shortcomings in [...] > Presumably they need to be kept somewhere and sent in the Authorization > header in case the server returns a 401 error and challenge (or a proxy > returns a 407 error and challenge) -- or maybe the Authorization header [...] urllib2 already knows about this syntax for proxy auth (ProxyHandler.proxy_open). If somebody fixes this (assuming it needs fixing -- I haven't read what the standards say), note that the Basic authentication logic is duplicated, and perhaps broken in one place, which should probably be fixed at the same time: http://www.google.com/groups?threadm=87d6dq2jx0.fsf%40pobox.com If nobody does it first, I'll fix it *eventually* (I'm lazy about configuring a local proxy and web server...). BTW, is somebody planning a 2.3.4 (or 2.2.4)? When? That would motivate me to fix urllib2 bugs sooner. John From martin at v.loewis.de Tue Dec 30 17:48:08 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 30 17:48:52 2003 Subject: [Python-Dev] First version of Python MSI available In-Reply-To: References: <3FF04774.9020404@v.loewis.de> Message-ID: <3FF200A8.5090604@v.loewis.de> Thomas Heller wrote: > The installer does not install pyconfig.h into Python24\include. [...] > Also distutils\command\wininst.exe is not installed. Thanks, will fix. Regards, Martin From martin at v.loewis.de Tue Dec 30 17:52:07 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 30 17:52:26 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF1C625.7070103@stackless.com> References: <3FF047E0.8070905@v.loewis.de> <3FF1C625.7070103@stackless.com> Message-ID: <3FF20197.1090802@v.loewis.de> Christian Tismer wrote: > Ach du Sch***e. > I don't want to buy another M$ compiler. > Is there a freeware alternative? Depends on what you want to do. If all you want to do is build extensions, this should be possible with mingw. Actually, with some work, it might be possible to build all of Python with mingw. As for freeware alternatives: Linux is a good alternative to Windows, too :-) Regards, Martin From martin at v.loewis.de Tue Dec 30 18:05:57 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 30 18:06:11 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> Message-ID: <3FF204D5.4010501@v.loewis.de> Paul Moore wrote: > Does this install cleanly alongside a "production" Python 2.3? Ie, > will it leave the meaning of the "python" command, and command line > associations for .py files, etc, unchanged? No. It will overwrite the associations, and, upon uninstallation, it will entirely remove them. OTOH, it does install into a separate folder, and it creates shortcuts in a separate start menu folder. > As a test install, I'd like it to have no effect on my main Python (I > have no test machine to install on separately). You could generate such an installer yourself, setting a variable in msi.py. However, I can make installation of the shortcuts optional, in future builds of the installer (to be enabled through a public MSI property). Regards, Martin From martin at v.loewis.de Tue Dec 30 18:08:57 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 30 18:10:00 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3cb2xhgn.fsf@yahoo.co.uk> References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> <3cb2xhgn.fsf@yahoo.co.uk> Message-ID: <3FF20589.2030702@v.loewis.de> Paul Moore wrote: > Hmm, for example, does the installer set the registry key > HKLM\Software\Microsoft\Windows\CurrentVersion\App Paths\python.exe Yes, it does. > Maybe just a single option in the installer, "Make this the default > version of Python", which is true by default, would be best. This > could control file associations, App Paths, and anything else related. Authoring new user interface interface is a tedious task. It is much easier to just have a public property, so you would have to install it as msiexec.exe python2.4.0.msi DEFAULTPYTHON=1 Regards, Martin From martin at v.loewis.de Tue Dec 30 18:13:36 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 30 18:13:54 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> <5.1.1.6.0.20031229123908.023122e0@telecommunity.com> Message-ID: <3FF206A0.3070002@v.loewis.de> Paul Moore wrote: > I don't know if Martin has already done this, but it needs doing. I'm > not a distutils expert, but I am willing to look at it in the longer > term. I haven't changed any Python file at all in the process of compiling with VC 7.1. However, Python (since 2.3) indicates the compiler used to build it in sys.version; I believe this could be used as an indication to find out whether it was build with VC6 or VC7.1 (dunno whether it could also tell apart 7.0 and 7.1). > If there wasn't a Windows binary version for 2.4 produced, this would > cause me problems. Python 2.4 is still months ahead, I expect more copies of VC 7.1 being available by that time (unless Microsoft releases the next compiler version before that - but they are unlikely to do so until the very end of 2004). > At the very least, I'd suggest a warning post on c.l.p and > c.l.p.announce, something to the effect of "Python 2.4 will be built > with MSVC 7.1. Extension developers supplying binary distributions for > Windows will need some way of building MSVC 7.1 compatible modules > (MSVC 7.1 itself, or a recent version of the free Mingw compiler > package) to continue providing binries." I expect the word will spread quickly; again, there is plenty of time to prepare for that. Regards, Martin From martin at v.loewis.de Tue Dec 30 18:17:01 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 30 18:17:12 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> <3FF05E58.1070108@v.loewis.de> <3FF08ECF.4010302@dmsware.com> <3FF0A2DA.4010205@v.loewis.de> <3FF0A945.2020107@dmsware.com> <3FF0CE34.4060409@v.loewis.de> Message-ID: <3FF2076D.7060806@v.loewis.de> Paolo Invernizzi wrote: > Ok. For what I have understood reading the PCbuild/readme.txt (I admit I > never tried to build on win32 an *hard* module, like tk or bdb, or ssl), > the building process of that modules is still mostly a manual job... > "follow the Windows instructions for building the Sleepycat software" It depends: yes for tk and sleepycat, no for ssl and bzip2. > I'll post something as soon as I've done some tries. Please keep us posted. It would be good if it involved no more manual intervention than the current process; it would be a plus if the manual intervention would be reduced. It would also be a plus if you could figure out some of the pending issues while you are at it, like how to stop the linker warnings when linking _bsddb.pyd :-) Regards, Martin From paoloinvernizzi at dmsware.com Tue Dec 30 18:47:52 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Tue Dec 30 18:47:53 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3FF206A0.3070002@v.loewis.de> References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> <5.1.1.6.0.20031229123908.023122e0@telecommunity.com> <3FF206A0.3070002@v.loewis.de> Message-ID: Martin v. Loewis wrote: > I believe this could be used as an > indication to find out whether it was build with VC6 or VC7.1 (dunno > whether it could also tell apart 7.0 and 7.1). For a version built with VC7.1 the sys.version is: 2.4a0 (#0, Dec 15 2003, 17:22:56) [MSC v.1310 32 bit (Intel)] If I remember well, for VC7.0 it's [MSC v.1300 bit (Intel)] --- Paolo Invernizzi From paoloinvernizzi at dmsware.com Tue Dec 30 18:54:59 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Tue Dec 30 18:55:00 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3FF2076D.7060806@v.loewis.de> References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> <3FF05E58.1070108@v.loewis.de> <3FF08ECF.4010302@dmsware.com> <3FF0A2DA.4010205@v.loewis.de> <3FF0A945.2020107@dmsware.com> <3FF0CE34.4060409@v.loewis.de> <3FF2076D.7060806@v.loewis.de> Message-ID: Martin v. Loewis wrote: > Please keep us posted. It would be good if it involved no more manual > intervention than the current process; it would be a plus if the manual > intervention would be reduced. Ok. > It would also be a plus if you could figure out some of the pending > issues while you are at it, like how to stop the linker warnings when > linking _bsddb.pyd :-) Ok ;-) -- Paolo Invernizzi From trentm at ActiveState.com Tue Dec 30 19:27:32 2003 From: trentm at ActiveState.com (Trent Mick) Date: Tue Dec 30 19:33:07 2003 Subject: [Python-Dev] "What's New" in docs In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE04BF@au3010avexu1.global.avaya.com>; from tdelaney@avaya.com on Fri, Dec 12, 2003 at 10:30:39AM +1100 References: <338366A6D2E2CA4C9DAEAE652E12A1DEFE04BF@au3010avexu1.global.avaya.com> Message-ID: <20031230162732.B12400@ActiveState.com> [Delaney, Timothy C (Timothy) wrote] > What's New in... > Python 2.3 > 1 PEP 218: A Standard Set Datatype > Python 2.2 > 1 PEP 234: Iterators (Behind the times here I know but:) ActivePython's chm does this. The TOC has: What's New Release Notes What's New in Python 2.3? What's New in Python 2.2? What's New in Python 2.1? The NEWS file Trent -- Trent Mick TrentM@ActiveState.com From kbk at shore.net Tue Dec 30 20:31:22 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Tue Dec 30 20:31:27 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312241812.hBOICdK22045@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Wed, 24 Dec 2003 10:12:39 -0800") References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> <200312241812.hBOICdK22045@c-24-5-183-134.client.comcast.net> Message-ID: <87brppdb85.fsf@hydra.localdomain> Guido van Rossum writes: > Would someone here be so kind as to report the bug we've found? Bug submitted with cut-down: http://cvs.openbsd.org/cgi-bin/query-pr-wrapper?full=yes&numbers=3622 It has been verified that this is fixed at OpenBSD 3.4. Therefore it's only a problem with 3.3 because there was no stack protector prior to that. Workaround when using -g option is to add -fno-stack-protector or -O1. I don't think any change to Python is warranted. -- KBK From raymond.hettinger at verizon.net Tue Dec 30 20:40:12 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue Dec 30 20:40:41 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation Message-ID: <003901c3cf3f$08c19720$e841fea9@oemcomputer> Dmitry Vasiliev has submitted a C implementation for the bisect module. My thoughts are to accept it with the following changes: * leave the original code intact for teaching purposes * import the C version from _bisect * make sure the C code is not list specific and will work with any container supporting __len__(), __getitem__(), and insert(). Do you guys have any objections? Raymond Hettinger From guido at python.org Tue Dec 30 20:47:31 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 30 20:44:57 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation In-Reply-To: Your message of "Tue, 30 Dec 2003 20:40:12 EST." <003901c3cf3f$08c19720$e841fea9@oemcomputer> References: <003901c3cf3f$08c19720$e841fea9@oemcomputer> Message-ID: <200312310147.hBV1lV602218@c-24-5-183-134.client.comcast.net> > Dmitry Vasiliev has submitted a C implementation for the bisect module. > > My thoughts are to accept it with the following changes: > > * leave the original code intact for teaching purposes > * import the C version from _bisect > * make sure the C code is not list specific and will work with any > container supporting __len__(), __getitem__(), and insert(). > > Do you guys have any objections? Only that I wished you had done the same for heapq.py. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Tue Dec 30 22:23:34 2003 From: tim.one at comcast.net (Tim Peters) Date: Tue Dec 30 22:23:40 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation In-Reply-To: <200312310147.hBV1lV602218@c-24-5-183-134.client.comcast.net> Message-ID: [Raymond] >> Dmitry Vasiliev has submitted a C implementation for the bisect >> module. >> >> My thoughts are to accept it with the following changes: >> >> * leave the original code intact for teaching purposes +1 >> * import the C version from _bisect +1 >> * make sure the C code is not list specific and will work with any >> container supporting __len__(), __getitem__(), and insert(). -0.6. "Gratuitous hyper-generality", IMO -- I'm willing to bet nobody has ever written a production class, with that collection of methods, that would also be suitable for using with bisect. "list subtype" is good enough, and will leave the code clearer, quicker, and more maintainable. >> Do you guys have any objections? Six tenths of one, as above . [Guido] > Only that I wished you had done the same for heapq.py. It's not too late. IIRC, neither you nor I could make time to review the heapq plans before the C code got checked in. I was able to make a little time after, and mourned the loss of the educational value of the Python version (to Raymond, in pvt). That's a big enough point that it deserves public airing, though: everyone (except Guido, I guess) please remember what it was like when you first learned Python! Mounds and mounds of library code written in Python, doing interesting things with interesting techniques, and written at a high enough level that you weren't wholly buried under details. Python's C code is generally clean and educational too, but has a *much* smaller potential audience than our pure-Python modules. Nobody can sanely accuse me of not caring about speed (see a decade of speed-obsessed Python checkins for counterexamples ), but I cry a little each time a piece of the system gets recoded in C. Keeping the original Python code around is a nice compromise (pioneered, IIRC, by string.py, a loooong time ago). From python at rcn.com Tue Dec 30 23:27:03 2003 From: python at rcn.com (Raymond Hettinger) Date: Tue Dec 30 23:28:13 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation In-Reply-To: Message-ID: <005801c3cf56$703a1e60$e841fea9@oemcomputer> [Raymond] > Dmitry Vasiliev has submitted a C implementation for the bisect > module. > > My thoughts are to accept it with the following changes: > > * leave the original code intact for teaching purposes > * import the C version from _bisect > * make sure the C code is not list specific and will work with any > container supporting __len__(), __getitem__(), and insert(). > > Do you guys have any objections? > [Guido] > > Only that I wished you had done the same for heapq.py. [Tim] > It's not too late. IIRC, neither you nor I could make time to review the > heapq plans before the C code got checked in. I was able to make a little > time after, and mourned the loss of the educational value of the Python > version (to Raymond, in pvt). I would be happy to bring back the python version and make it coexist with the C version. IMO, it was a programming pearl unto itself and efforts were already made to preserve the extensive module docstring and make the code match the original line for line with the variable names and logic intact. The bisect module warrants even more consideration because it is more basic and more venerable. [Tim] > Nobody can sanely accuse me of not > caring about speed (see a decade of speed-obsessed Python checkins for > counterexamples ), but I cry a little each time a piece of the > system > gets recoded in C. Keeping the original Python code around is a nice > compromise (pioneered, IIRC, by string.py, a loooong time ago). Also, with the PyPy project, there is a chance that cleanly coded pure python modules will survive the test of time longer than their C counterparts. In the itertools module, I documented pure python equivalents to make the documentation more specific, to enchance understanding, and to make it easier to backport. For the most part, that effort was successful. The only downside is the double maintenance problem of keeping the two in sync. Given the stability of bisect, that is not much of an issue. Raymond Hettinger ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From skip at pobox.com Wed Dec 31 00:08:04 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Dec 31 00:08:13 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation In-Reply-To: References: <200312310147.hBV1lV602218@c-24-5-183-134.client.comcast.net> Message-ID: <16370.22964.429991.454763@montanaro.dyndns.org> Tim> Nobody can sanely accuse me of not caring about speed (see a decade Tim> of speed-obsessed Python checkins for counterexamples ), but Tim> I cry a little each time a piece of the system gets recoded in C. Tim> Keeping the original Python code around is a nice compromise Tim> (pioneered, IIRC, by string.py, a loooong time ago). It would be nice to keep the old Python implementations around for two reasons. First, there's the obvious educational value. Second, the PyPy folks will know where to find it when they need it. I think a Demo/Lib subdirectory (with appropriate README) would be a reasonable place to put such code out to pasture. It could be made available in source distributions but never installed. Skip From tjreedy at udel.edu Wed Dec 31 00:49:59 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Wed Dec 31 00:50:00 2003 Subject: [Python-Dev] Re: urllib2 doesn't grok URLs w/ user/passwd References: <16369.37851.229594.332869@montanaro.dyndns.org><200312301711.44687.aleaxit@yahoo.com> Message-ID: "John J Lee" wrote in message news:Pine.LNX.4.58.0312302129080.472@alice... > http://www.google.com/groups?threadm=87d6dq2jx0.fsf%40pobox.com > > If nobody does it first, I'll fix it *eventually* (I'm lazy about > configuring a local proxy and web server...). > > BTW, is somebody planning a 2.3.4 #yes >(or 2.2.4)? #no > When? #April-May > That would motivate me to fix urllib2 bugs sooner. I suspect more backported bug fixes will motivate a release. tjr From anthony at interlink.com.au Wed Dec 31 02:49:35 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Dec 31 02:50:08 2003 Subject: [Python-Dev] Re: urllib2 doesn't grok URLs w/ user/passwd In-Reply-To: Message-ID: <200312310749.hBV7naqk011830@localhost.localdomain> >>> "Terry Reedy" wrote > > BTW, is somebody planning a 2.3.4 #yes > >(or 2.2.4)? #no > > When? #April-May > > That would motivate me to fix urllib2 bugs sooner. > > I suspect more backported bug fixes will motivate a release. I plan a 2.3.4 in the April-May-June timeframe. This will _probably_ be the final release of 2.3 before 2.4, although we could have a 2.3.5 if there's a need for it at the end of the year (shortly after 2.4 is done would be my pick). I don't know that we'd have a need for many more 2.3 releases after that - particularly since, at the moment at least, 2.4 seems to be continuing down 2.3's conservative path as far as potentially dangerous changes. Note that, aside from major bug fixes, I'm not particularly driven by volume of bug fixes for selecting a release schedule - mostly it's driven by my own personal estimation of when I'm likely to have time available to do the release. Each release I do seems to take less time (yay for automation! boo that the major part of the automation seems to have gone missing somehow in the laptop theft/recovery of data from tape.... ah well). Anthony -- Anthony Baxter It's never too late to have a happy childhood. From 4q89v20mku at msn.com Wed Dec 31 16:33:44 2003 From: 4q89v20mku at msn.com (Cara Curtis) Date: Wed Dec 31 08:39:02 2003 Subject: [Python-Dev] Refill your medication online! yyjw l Message-ID: Rated Top Rated Online Store. HOT & NEW - Levitra/Lipitor/Nexium Weekly speciasls on all our drugs. -Zocor -Soma -Ambien -Phentermine -V1agra -Discount Generic's on ALL -MORE Next day discrete shipping on all products! http://www.rxstoreusa.biz/shopping Please, I wish to receive no more discounts on valuable items. http://www.rxstoreusa.biz/a.html jet djjdnjqj nldxi From barry at python.org Wed Dec 31 09:07:24 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 31 09:07:30 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation In-Reply-To: References: Message-ID: <1072879643.28895.241.camel@anthem> On Tue, 2003-12-30 at 22:23, Tim Peters wrote: > That's a big enough point that it deserves public airing, though: everyone > (except Guido, I guess) please remember what it was like when you first > learned Python! Mounds and mounds of library code written in Python, doing > interesting things with interesting techniques, and written at a high enough > level that you weren't wholly buried under details. Python's C code is > generally clean and educational too, but has a *much* smaller potential > audience than our pure-Python modules. Nobody can sanely accuse me of not > caring about speed (see a decade of speed-obsessed Python checkins for > counterexamples ), but I cry a little each time a piece of the system > gets recoded in C. Keeping the original Python code around is a nice > compromise (pioneered, IIRC, by string.py, a loooong time ago). Yes, yes, yes. I'd be bummed if modules like pickle.py got lost or out of sync with its C cousin. -Barry From barry at python.org Wed Dec 31 09:10:23 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 31 09:10:32 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation In-Reply-To: <16370.22964.429991.454763@montanaro.dyndns.org> References: <200312310147.hBV1lV602218@c-24-5-183-134.client.comcast.net> <16370.22964.429991.454763@montanaro.dyndns.org> Message-ID: <1072879822.28895.245.camel@anthem> On Wed, 2003-12-31 at 00:08, Skip Montanaro wrote: > It would be nice to keep the old Python implementations around for two > reasons. First, there's the obvious educational value. Second, the PyPy > folks will know where to find it when they need it. I think a Demo/Lib > subdirectory (with appropriate README) would be a reasonable place to put > such code out to pasture. It could be made available in source > distributions but never installed. The problem is that relegating stuff to Demo would very likely lead to bitrot, which is not a good thing. Another advantage of having current, working (and unit tested) pure-Python alternatives is that they are much easier to modify if you're experimenting with new features, or bug fixes, etc. OTOH, some of the really ancient Python modules will occasionally need attention to bring them up to modern coding standards, idioms, and practices. -Barry From tismer at stackless.com Wed Dec 31 09:44:10 2003 From: tismer at stackless.com (Christian Tismer) Date: Wed Dec 31 09:44:16 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF20197.1090802@v.loewis.de> References: <3FF047E0.8070905@v.loewis.de> <3FF1C625.7070103@stackless.com> <3FF20197.1090802@v.loewis.de> Message-ID: <3FF2E0BA.9090105@stackless.com> Martin v. Loewis wrote: [CT doesn't want to spend money on a new compiler] > Depends on what you want to do. If all you want > to do is build extensions, this should be possible > with mingw. Actually, with some work, it might be > possible to build all of Python with mingw. > > As for freeware alternatives: Linux is a good > alternative to Windows, too :-) The problem is: On Windows, and with VC++6.0, I really can fly, while on Linux I'm so-so. It would be a shame if I had to pass Stackless Windows support over to someone else, just because we changed compilers ;-) best wishes for the new year -- chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Wed Dec 31 09:45:28 2003 From: tismer at stackless.com (Christian Tismer) Date: Wed Dec 31 09:45:33 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <200312301934.hBUJYPL01492@c-24-5-183-134.client.comcast.net> References: <3FF047E0.8070905@v.loewis.de> <3FF1C625.7070103@stackless.com> <200312301934.hBUJYPL01492@c-24-5-183-134.client.comcast.net> Message-ID: <3FF2E108.9060101@stackless.com> Guido van Rossum wrote: >>I don't want to buy another M$ compiler. >>Is there a freeware alternative? > > > Ten core Python developers received a free VC7.1 copy from Microsoft. > Maybe someone who hasn't opened theirs yet is willing to step down? > > MS also alluded to a free downloadable compiler, but I haven't tracked > it down yet. And of course theer's always mingw (or whatever it's > called). If there is a way I can provide Windows binaries, and I can do some interactive debugging, I'm willing to switch tools, sure. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From skip at pobox.com Wed Dec 31 09:56:03 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Dec 31 09:56:14 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation In-Reply-To: <1072879822.28895.245.camel@anthem> References: <200312310147.hBV1lV602218@c-24-5-183-134.client.comcast.net> <16370.22964.429991.454763@montanaro.dyndns.org> <1072879822.28895.245.camel@anthem> Message-ID: <16370.58243.130247.574887@montanaro.dyndns.org> Barry> The problem is that relegating stuff to Demo would very likely Barry> lead to bitrot, which is not a good thing. True. Note, however, that since these files would be in a source distribution there's no reason they couldn't be factored into "make test" somehow. Barry> Another advantage of having current, working (and unit tested) Barry> pure-Python alternatives is that they are much easier to modify Barry> if you're experimenting with new features, or bug fixes, etc. Agreed. Skip From mcherm at mcherm.com Wed Dec 31 10:12:34 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Wed Dec 31 10:12:38 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 Message-ID: <1072883554.3ff2e762a975e@mcherm.com> [Paul Moore:] > Maybe just a single option in the installer, "Make this the default > version of Python", which is true by default, would be best. This > could control file associations, App Paths, and anything else related. [Martin v Loewis:] > Authoring new user interface interface is a tedious task. > It is much easier to just have a public property, so you would > have to install it as > > msiexec.exe python2.4.0.msi DEFAULTPYTHON=1 Yes, but it's a tedious task undertaken by a single individual who knows something about how msi files work. Whereas a checkbox in the installer is easily understood by any user, including the vast majority who wouldn't know what "msi" meant, much less how to run execute one by hand with public properties specified. Not that I'm going to complain about how you're handling this... after all, you're the one volunteering your time. But if you're interested in hearing random opinions from others, I am one who would find it awefully nice if the MS installer had an option to install a non-default version of Python. Then someday it would be much easier for me to keep around copies of Python 2.4, 2.4.1, 2.4.2, 2.5, 2.5.1, and 2.6 all on the same machine. If-the-PSF-were-paying-your-salary-THEN-I'd-complain -- Michael Chermside From barry at python.org Wed Dec 31 11:36:20 2003 From: barry at python.org (Barry A. Warsaw) Date: Wed Dec 31 11:36:25 2003 Subject: [Python-Dev] base64.py support for RFC 3548 Message-ID: <16370.64260.941889.163153@gargle.gargle.HOWL> Hi all, I found I needed more complete RFC 3548 support for a project I'm working on. This RFC contains the most up-to-date specification of Base64, Base32, and Base16, including a description of alternative alphabets, etc. I didn't want to create any new modules, so I modified the base64.py module to include a bunch of new functions, leaving the old API unchanged (i.e. encode(), decode(), encodestring(), decodestring()). I have new functions with docstrings, and an updated unit test. If you folks like the API I've chosen and are generally in favor of my changes, I'll update the documentation as well. The patch is available here: http://sourceforge.net/tracker/index.php?func=detail&aid=868496&group_id=5470&atid=305470 Currently assigned to Tim for review. -Barry From barry at python.org Wed Dec 31 11:41:11 2003 From: barry at python.org (Barry A. Warsaw) Date: Wed Dec 31 11:41:15 2003 Subject: [Python-Dev] regrtest.py mods for code coverage Message-ID: <16370.64551.432152.383782@gargle.gargle.HOWL> Hi all, I wanted to improve the code coverage for base64.py (see previous message, re: RFC 3548 support) in the test_base64.py unit test. So I did a fairly simpleminded adaptation of Zope3's test.py -T code coverage flag to regrtest.py. Very handy for upping coverage of a module's code! The patch is available here: http://sourceforge.net/tracker/index.php?func=detail&aid=868499&group_id=5470&atid=305470 Currently assigned to Jeremy for review. Please let me know what you think. -Barry From guido at python.org Wed Dec 31 12:54:15 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 12:54:25 2003 Subject: [Python-Dev] Pie-thon benchmark code ready Message-ID: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> While it's still 2003 in most of the US and Europe (and happy new year to the folks in Asia, Australia and New Zealand! :-), I present the official Pie-thon Parrot Benchmark: ftp://ftp.python.org/pub/python/parrotbench/parrotbench.tgz I'll quote from the README.txt file: """ This is a benchmark to be run in front of a live audience at OSCON 2004 between Python and Parrot. The bytecode must be Python 2.3 bytecode frozen in December 2003 (which is almost over as I write this :-). For some more background, see the python-dev thread around http://mail.python.org/pipermail/python-dev/2003-December/040977.html The benchmark here is intended to make Dan Sugalski's life difficult: there are some standard benchmark thingies (simple random algorithms using basic data types) but also a lot of play with odd corners of the language definition, and Python's extremely dynamic object model: funky descriptors, mutable classes, that sort of thing. The benchmark is supposed to run with either Python 2.3 or Python 2.4. [...] On timing: there's a requirement that the benchmark runs for at least 30 seconds. It currently runs for nearly a minute on my home box, which is a four-year-old 650 MHz Pentium box. If the contest hardware is so fast that it runs under a minute, there's a number in b.py that can be cranked up to increase the number of runs. (It takes 27 seconds on my recent work desktop, and on my screaming IBM T40 laptop it runs in 15 seconds, so I suspect that we should at least double the number of runs from 2 to 4.) """ I'd be happy to answer any questions. Let the competition begin! --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Wed Dec 31 13:12:13 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 13:12:48 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> Message-ID: At 9:54 AM -0800 12/31/03, Guido van Rossum wrote: >While it's still 2003 in most of the US and Europe (and happy new year >to the folks in Asia, Australia and New Zealand! :-), I present the >official Pie-thon Parrot Benchmark: > > ftp://ftp.python.org/pub/python/parrotbench/parrotbench.tgz Woohoo! The announcement's forwarded on to p6i for the denizens there to grovel over. I fully expect many hours of pain and misery ahead to make this work. :) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido at python.org Wed Dec 31 13:27:51 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 13:27:57 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: Your message of "Wed, 31 Dec 2003 13:12:13 EST." References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> Message-ID: <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> > > ftp://ftp.python.org/pub/python/parrotbench/parrotbench.tgz > > Woohoo! The announcement's forwarded on to p6i for the denizens there > to grovel over. I fully expect many hours of pain and misery ahead to > make this work. :) Enjoy. :-) Is there a online archive where I can watch the fun without signing up? BTW I forgot to mention that parts of the test are self-checking, but other parts require comparing of the output. The file 'out' is all you should care about. And if you want the bytecode, try this in a Unix shell: $ python2.3 -O [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import b >>> ^D $ You should now have optimized bytecode in the 8 files b*.pyo. --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Wed Dec 31 13:40:18 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 13:40:49 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> Message-ID: At 10:27 AM -0800 12/31/03, Guido van Rossum wrote: > > > ftp://ftp.python.org/pub/python/parrotbench/parrotbench.tgz >> >> Woohoo! The announcement's forwarded on to p6i for the denizens there >> to grovel over. I fully expect many hours of pain and misery ahead to >> make this work. :) > >Enjoy. :-) Is there a online archive where I can watch the fun >without signing up? Yep. There's a news<->mail gateway for all the perl.org lists with web archive, so you can read via a newsreader at news://nntp.perl.org/perl.perl6.internals or webbed at http://www.nntp.perl.org/group/perl.perl6.internals. It's a bit of a rough web archive, unfortunately, but the gateway does maintain threads. (Assuming the participants aren't using a broken mail program. Alas, I sometimes do) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From dan at sidhe.org Wed Dec 31 14:06:10 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 14:06:31 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> Message-ID: At 10:27 AM -0800 12/31/03, Guido van Rossum wrote: > > > ftp://ftp.python.org/pub/python/parrotbench/parrotbench.tgz >> >> Woohoo! The announcement's forwarded on to p6i for the denizens there >> to grovel over. I fully expect many hours of pain and misery ahead to >> make this work. :) > >Enjoy. :-) Is there a online archive where I can watch the fun >without signing up? > >BTW I forgot to mention that parts of the test are self-checking, but >other parts require comparing of the output. The file 'out' is all >you should care about. And if you want the bytecode, try this in a >Unix shell: > > $ python2.3 -O > [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import b > >>> ^D > $ Hrm. On my OS X laptop: lir:~/Desktop/parrotbench dan$ python -O Python 2.3 (#1, Sep 13 2003, 00:49:11) [GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import b --> iteration 0 --> b0 Traceback (most recent call last): File "", line 1, in ? File "b.py", line 12, in ? b0.main() File "b0.py", line 893, in main checkoutput(4201300315) File "b0.py", line 763, in checkoutput check(strhash(outputtext), n) File "b0.py", line 3, in check raise AssertionError("%.30r != %.30r" % (a, b)) AssertionError: 503203581L != 4201300315L >>> I'll try generating the bytecode on a linux box, but something seems amiss somewhere. (Perhaps just in my understanding of what should be happening) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido at python.org Wed Dec 31 14:31:02 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 14:31:08 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: Your message of "Wed, 31 Dec 2003 14:06:10 EST." References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> Message-ID: <200312311931.hBVJV2o05897@c-24-5-183-134.client.comcast.net> (Thanks for the p6i archives link BTW; Googling for "p6i" wasn't too helpful since its true name is perl6.internals. :-) > > $ python2.3 -O > > [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > > >>> import b > > >>> ^D > > $ > > Hrm. On my OS X laptop: > > lir:~/Desktop/parrotbench dan$ python -O > Python 2.3 (#1, Sep 13 2003, 00:49:11) > [GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import b > --> iteration 0 > --> b0 > Traceback (most recent call last): > File "", line 1, in ? > File "b.py", line 12, in ? > b0.main() > File "b0.py", line 893, in main > checkoutput(4201300315) > File "b0.py", line 763, in checkoutput > check(strhash(outputtext), n) > File "b0.py", line 3, in check > raise AssertionError("%.30r != %.30r" % (a, b)) > AssertionError: 503203581L != 4201300315L > >>> > > I'll try generating the bytecode on a linux box, but something seems > amiss somewhere. (Perhaps just in my understanding of what should be > happening) No, this means that the benchmark fails on that platform. But you got quite far (the first 5 sections of b0.py worked without a hitch). I notice you're using the original Python 2.3 release thjat came with Panther. It's a bit old. If upgrading to Python 2.3.3 doesn't help, can you send me the output of python b0.py ? Send it in private mail, it's rather large. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Wed Dec 31 16:16:34 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 16:18:25 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200312311931.hBVJV2o05897@c-24-5-183-134.client.comcast.net> References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> <200312311931.hBVJV2o05897@c-24-5-183-134.client.comcast.net> Message-ID: At 11:31 AM -0800 12/31/03, Guido van Rossum wrote: >(Thanks for the p6i archives link BTW; Googling for "p6i" wasn't too >helpful since its true name is perl6.internals. :-) > >> > $ python2.3 -O >> > [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 >> > Type "help", "copyright", "credits" or "license" for more information. >> > >>> import b >> > >>> ^D >> > $ >> > > Hrm. On my OS X laptop: [Failure snipped] > > >> I'll try generating the bytecode on a linux box, but something seems >> amiss somewhere. (Perhaps just in my understanding of what should be >> happening) > >No, this means that the benchmark fails on that platform. > >But you got quite far (the first 5 sections of b0.py worked without a >hitch). > >I notice you're using the original Python 2.3 release thjat came with >Panther. It's a bit old. If upgrading to Python 2.3.3 doesn't help, Turns out not to. I built and installed the 2.3.3 kit and it still fails, albeit differently. >can you send me the output of > > python b0.py > >? Send it in private mail, it's rather large. :-) Sent under a separate cover. Traceback (for those folks following along at home) is: lir:~/Desktop/parrotbench dan$ /usr/local/bin/python2.3 b0.py >b0_output_darwin.lst Traceback (most recent call last): File "b0.py", line 924, in ? main() File "b0.py", line 893, in main checkoutput(4201300315) File "b0.py", line 763, in checkoutput check(strhash(outputtext), n) File "b0.py", line 3, in check raise AssertionError("%.30r != %.30r" % (a, b)) AssertionError: 2772929724L != 4201300315L lir:~/Desktop/parrotbench dan$ I did get bytecode on a linux system, so I've something to work with, though it'll be tough to test times as we go on. (Though I figure if I can beat the 1.4GHz Celeron system with my 600MHz G3 laptop I'll be doing good :) I'll give this a shot on the 10.2 OS X server box I have at home, just for chuckles. (I can set it up as an automated build server if you want, which is all I use it for right now) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From skip at pobox.com Wed Dec 31 18:33:27 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Dec 31 18:33:37 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> <200312311931.hBVJV2o05897@c-24-5-183-134.client.comcast.net> Message-ID: <16371.23751.389244.816299@montanaro.dyndns.org> >> > Hrm. On my OS X laptop: Dan> [Failure snipped] >> I notice you're using the original Python 2.3 release thjat came with >> Panther. It's a bit old. If upgrading to Python 2.3.3 doesn't help, Dan> Turns out not to. I built and installed the 2.3.3 kit and it still Dan> fails, albeit differently. Works for me (Mac OS X 10.2.8, Python 2.3.2, GCC 3.3, parrotbench directory after cvs up around 5:30PM central time): montanaro:parrotbench% python2.3 -O Python 2.3.2 (#1, Dec 17 2003, 17:22:01) [GCC 3.3 20030304 (Apple Computer, Inc. build 1493)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import b --> iteration 0 --> b0 3141592653 3141592653 --> b1 False False 0 --> b2 ... --> b4 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] --> b5 OK. --> b6 --> All done. Did you make sure you didn't have any .py[co] files before starting? Skip From guido at python.org Wed Dec 31 18:34:07 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 18:34:14 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: Your message of "Wed, 31 Dec 2003 11:31:02 PST." Message-ID: <200312312334.hBVNY7506326@c-24-5-183-134.client.comcast.net> Dan reported a problem with the benchmark on Mac OS X: > > Hrm. On my OS X laptop: > > > > lir:~/Desktop/parrotbench dan$ python -O > > Python 2.3 (#1, Sep 13 2003, 00:49:11) > > [GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin > > Type "help", "copyright", "credits" or "license" for more information. > > >>> import b > > --> iteration 0 > > --> b0 > > Traceback (most recent call last): > > File "", line 1, in ? > > File "b.py", line 12, in ? > > b0.main() > > File "b0.py", line 893, in main > > checkoutput(4201300315) > > File "b0.py", line 763, in checkoutput > > check(strhash(outputtext), n) > > File "b0.py", line 3, in check > > raise AssertionError("%.30r != %.30r" % (a, b)) > > AssertionError: 503203581L != 4201300315L > > >>> After some off-line email exchanges I think I have a fix for this behavior, which must have to do with a different length of the addresses shown in the default repr(), e.g. "". Version 1.0.1 of the benchmark is at the same place as before: ftp://python.org/pub/python/parrotbench/parrotbench.tgz (You can tell whether you have the fixed version by looking at first line of README.txt; if it says "Parrot benchmark 1.0.1" you do.) I haven't heard back from Dan, but I assume that the fix works. Happy New Year everyone! --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Dec 31 18:45:00 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 18:45:07 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: Your message of "Wed, 31 Dec 2003 17:33:27 CST." <16371.23751.389244.816299@montanaro.dyndns.org> References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> <200312311931.hBVJV2o05897@c-24-5-183-134.client.comcast.net> <16371.23751.389244.816299@montanaro.dyndns.org> Message-ID: <200312312345.hBVNj0v06358@c-24-5-183-134.client.comcast.net> > Dan> Turns out not to. I built and installed the 2.3.3 kit and it still > Dan> fails, albeit differently. > > Works for me (Mac OS X 10.2.8, Python 2.3.2, GCC 3.3, parrotbench directory > after cvs up around 5:30PM central time): Check the README.txt. Does it say 1.0.1? Then you have my fixed version! I hope so, then I can go home. :-) > montanaro:parrotbench% python2.3 -O > Python 2.3.2 (#1, Dec 17 2003, 17:22:01) > [GCC 3.3 20030304 (Apple Computer, Inc. build 1493)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import b > --> iteration 0 > --> b0 > 3141592653 > 3141592653 > --> b1 > False > False ^^^^^ This actually means that you failed the stack overflow test. It should work when you run it like this: python -O b.py Or you could do sys.setrecursionlimit(1001) before importing b in the interactive session. But that's unrelated to Dan's problem (which happens before b0 completes). And even if that test fails, the .pyo files will be correct. --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Wed Dec 31 18:49:36 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 18:50:03 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200312312334.hBVNY7506326@c-24-5-183-134.client.comcast.net> References: <200312312334.hBVNY7506326@c-24-5-183-134.client.comcast.net> Message-ID: At 3:34 PM -0800 12/31/03, Guido van Rossum wrote: >After some off-line email exchanges I think I have a fix for this >behavior, which must have to do with a different length of the >addresses shown in the default repr(), e.g. "". > >Version 1.0.1 of the benchmark is at the same place as before: > > ftp://python.org/pub/python/parrotbench/parrotbench.tgz > >(You can tell whether you have the fixed version by looking at first >line of README.txt; if it says "Parrot benchmark 1.0.1" you do.) > >I haven't heard back from Dan, but I assume that the fix works. Yup, works just fine. (You caught me during the east coast commute home :) >Happy New Year everyone! Ho, ho, ho! -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From allison at sumeru.stanford.EDU Wed Dec 31 18:56:00 2003 From: allison at sumeru.stanford.EDU (Dennis Allison) Date: Wed Dec 31 18:56:49 2003 Subject: [Python-Dev] Are we collecting benchmark results across machines In-Reply-To: <200312312334.hBVNY7506326@c-24-5-183-134.client.comcast.net> Message-ID: I suspect there are other folks who have run the pie-con benchmarks on their machines. Perhaps we should construct a chart. Times below are in seconds. ~60 Guido's ancient 650Mhz Pentium ~27 Guido's desktop at work ~15 IBM T40 laptop 88.550 dual 450 MHz Pentium 2 22.340 Athlon XP2800 (1243.363 MHz clock) The latter two times are for Python 2.3.3 built out of the box using the Makefile (make time) shipped with the parrotbenchmarks ftp file on an unloaded machine. The times reported are the user time from the time triplet. From skip at pobox.com Wed Dec 31 18:56:25 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Dec 31 18:57:06 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200312312345.hBVNj0v06358@c-24-5-183-134.client.comcast.net> References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> <200312311931.hBVJV2o05897@c-24-5-183-134.client.comcast.net> <16371.23751.389244.816299@montanaro.dyndns.org> <200312312345.hBVNj0v06358@c-24-5-183-134.client.comcast.net> Message-ID: <16371.25129.466173.864597@montanaro.dyndns.org> >> Works for me (Mac OS X 10.2.8, Python 2.3.2, GCC 3.3, parrotbench >> directory after cvs up around 5:30PM central time): Guido> Check the README.txt. Does it say 1.0.1? Then you have my fixed Guido> version! Yes. Guido> I hope so, then I can go home. :-) I saw your checkin after my cvs up, so yes, you probably did fix the problem. Time to go find some champagne. Skip From guido at python.org Wed Dec 31 19:14:16 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 19:14:29 2003 Subject: [Python-Dev] Re: Are we collecting benchmark results across machines In-Reply-To: Your message of "Wed, 31 Dec 2003 15:56:00 PST." References: Message-ID: <200401010014.i010EGt06465@c-24-5-183-134.client.comcast.net> > I suspect there are other folks who have run the pie-con benchmarks on > their machines. Perhaps we should construct a chart. Times below are in > seconds. > > ~60 Guido's ancient 650Mhz Pentium > ~27 Guido's desktop at work > ~15 IBM T40 laptop > > 88.550 dual 450 MHz Pentium 2 > 22.340 Athlon XP2800 (1243.363 MHz clock) > > The latter two times are for Python 2.3.3 built out of the box using the > Makefile (make time) shipped with the parrotbenchmarks ftp file on an > unloaded machine. The times reported are the user time from the time > triplet. Perhaps we can turn this into a benchmark comparison chart. In particular, in my experience pystone is a pretty good indicator of system performance even if it's a lousy benchmark. I'll report the pystone numbers for those same three systems: home desktop: 10438.4 pystones/second work desktop: 17421.6 pystones/second laptop: 30198.1 pystones/second Multiplying the pystone numbers with the parrotbench times should give a constant if the benchmarks are equivalent. But this doesn't look good: I get home desktop: 626304 work desktop: 470383 laptop: 452972 The home desktop is a clear outlier; it's more than twice as slow on the parrot benchmark, but only 2/3rds slower on pystone... (This is begging for a spreadsheet. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Wed Dec 31 19:16:31 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 19:16:54 2003 Subject: [Python-Dev] Are we collecting benchmark results across machines In-Reply-To: References: Message-ID: At 3:56 PM -0800 12/31/03, Dennis Allison wrote: >I suspect there are other folks who have run the pie-con benchmarks on >their machines. Perhaps we should construct a chart. Times below are in >seconds. > >~60 Guido's ancient 650Mhz Pentium >~27 Guido's desktop at work >~15 IBM T40 laptop > >88.550 dual 450 MHz Pentium 2 >22.340 Athlon XP2800 (1243.363 MHz clock) Well, add in: real 1m20.412s user 1m1.580s sys 0m2.100s for this somewhat creaky 600MHz G3 iBook... -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido at python.org Wed Dec 31 19:26:03 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 19:26:09 2003 Subject: [Python-Dev] Are we collecting benchmark results across machines In-Reply-To: Your message of "Wed, 31 Dec 2003 19:16:31 EST." References: Message-ID: <200401010026.i010Q3B06511@c-24-5-183-134.client.comcast.net> > Well, add in: > > real 1m20.412s > user 1m1.580s > sys 0m2.100s > > for this somewhat creaky 600MHz G3 iBook... I've been looking at user times only, but on that box the discrepancy between user and real time is enormous! It also suggests that a 600 MHz G3 and a 650 P3 are pretty close, and contrary to what Apple seems to claim, the G3's MHz rating isn't worth much more than a P3's MHz rating. Could you run pystone too? python -c 'from test.pystone import main; main(); main(); main()' and then report the smallest pystones/second value. --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Wed Dec 31 19:34:43 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 19:34:56 2003 Subject: [Python-Dev] Are we collecting benchmark results across machines In-Reply-To: <200401010026.i010Q3B06511@c-24-5-183-134.client.comcast.net> References: <200401010026.i010Q3B06511@c-24-5-183-134.client.comcast.net> Message-ID: At 4:26 PM -0800 12/31/03, Guido van Rossum wrote: > > Well, add in: >> >> real 1m20.412s >> user 1m1.580s >> sys 0m2.100s >> >> for this somewhat creaky 600MHz G3 iBook... > >I've been looking at user times only, but on that box the discrepancy >between user and real time is enormous! Yeah, that's not uncommon. I'm not sure if there's a problem with the time command, or if it's something else. This is a laptop (currently on battery power, though with reduced performance turned off) and it's got a 100MHz main memory bus, which is definitely a limiting factor once code slips out of L1 cache. It's faster than my Gameboy, but some days I wonder how much... :) >It also suggests that a 600 MHz G3 and a 650 P3 are pretty close, and >contrary to what Apple seems to claim, the G3's MHz rating isn't worth >much more than a P3's MHz rating. Possibly. I'm not sure it's necessarily the best machine for that comparison, given the power/performance tradeoffs with laptops. I may give it a whirl on one of the G3 desktop machines around here to see how much bus speed matters. (I won't be surprised, given the likely working set for an interpreter, to find that bus speed makes more of a difference than CPU speed) >Could you run pystone too? > > python -c 'from test.pystone import main; main(); main(); main()' > >and then report the smallest pystones/second value. Smallest is: Pystone(1.1) time for 50000 passes = 6.61 This machine benchmarks at 7564.3 pystones/second -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido at python.org Wed Dec 31 19:39:05 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 19:39:11 2003 Subject: [Python-Dev] Are we collecting benchmark results across machines In-Reply-To: Your message of "Wed, 31 Dec 2003 19:34:43 EST." References: <200401010026.i010Q3B06511@c-24-5-183-134.client.comcast.net> Message-ID: <200401010039.i010d5U06553@c-24-5-183-134.client.comcast.net> > >Could you run pystone too? > > > > python -c 'from test.pystone import main; main(); main(); main()' > > > >and then report the smallest pystones/second value. > > Smallest is: > > Pystone(1.1) time for 50000 passes = 6.61 > This machine benchmarks at 7564.3 pystones/second Thanks -- though I made a mistake -- I should've asked for the largest pystone value -- larger is better for pystone, unlike for running times. Anyway, this value is reasonable given your parrotbench time. --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis at bluewin.ch Wed Dec 31 19:44:49 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Wed Dec 31 19:41:14 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.ne t> Message-ID: <5.2.1.1.0.20040101014159.03a31f50@pop.bluewin.ch> At 09:54 31.12.2003 -0800, Guido van Rossum wrote: >I'd be happy to answer any questions. Let the competition begin! wondering whether the correctness checks are too much dependent on details of the representation of objects, especially the order of keys in a dict repr etc... From guido at python.org Wed Dec 31 19:50:33 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 19:50:41 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: Your message of "Thu, 01 Jan 2004 01:44:49 +0100." <5.2.1.1.0.20040101014159.03a31f50@pop.bluewin.ch> References: <5.2.1.1.0.20040101014159.03a31f50@pop.bluewin.ch> Message-ID: <200401010050.i010oXA06606@c-24-5-183-134.client.comcast.net> > wondering whether the correctness checks are too much dependent on details > of the representation of objects, especially the order of keys in a dict > repr etc... Probably. :-( I could probably fix that by subclassing dict with something whose repr() sorts the keys -- it doesn't print the few large dicts that it uses, so this should be fine. The same approach could be used for fixing all repr()-related issues. I'll leave it to Dan to decide if he wants this to be fixed ASAP or if he's okay with putting it off until later -- I imagine it will take a while before he's got it running on Parrot well enough to hit the first assert... (There's also an issue with the distinction between repr() of int and long, which will disappear in Python 3.0, and repr() of str vs. unicode, which already doesn't exist in Jython. So I predict that the benchmark currently doesn't have a chance of passing on Jython. (But it would be interesting to time it anyway -- simply disable the raises in b0.check() and in b5.check().) --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Wed Dec 31 20:16:00 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 20:16:23 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200401010050.i010oXA06606@c-24-5-183-134.client.comcast.net> References: <5.2.1.1.0.20040101014159.03a31f50@pop.bluewin.ch> <200401010050.i010oXA06606@c-24-5-183-134.client.comcast.net> Message-ID: At 4:50 PM -0800 12/31/03, Guido van Rossum wrote: > > wondering whether the correctness checks are too much dependent on details >> of the representation of objects, especially the order of keys in a dict >> repr etc... > >Probably. :-( > >I could probably fix that by subclassing dict with something whose >repr() sorts the keys -- it doesn't print the few large dicts that it >uses, so this should be fine. The same approach could be used for >fixing all repr()-related issues. I'll leave it to Dan to decide if >he wants this to be fixed ASAP or if he's okay with putting it off >until later -- I imagine it will take a while before he's got it >running on Parrot well enough to hit the first assert... Heh. I'm OK with either leaving it as-is if the current ordering is considered correct, or altering the benchmark to guarantee ordering. (Or having the output compared by an external program to guarantee correctness, which is fine as well) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido at python.org Wed Dec 31 20:21:27 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 20:21:34 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: Your message of "Wed, 31 Dec 2003 20:16:00 EST." References: <5.2.1.1.0.20040101014159.03a31f50@pop.bluewin.ch> <200401010050.i010oXA06606@c-24-5-183-134.client.comcast.net> Message-ID: <200401010121.i011LRP06664@c-24-5-183-134.client.comcast.net> > >> wondering whether the correctness checks are too much dependent > >> on details of the representation of objects, especially the order > >> of keys in a dict repr etc... > > > >Probably. :-( > > > >I could probably fix that by subclassing dict with something whose > >repr() sorts the keys -- it doesn't print the few large dicts that it > >uses, so this should be fine. The same approach could be used for > >fixing all repr()-related issues. I'll leave it to Dan to decide if > >he wants this to be fixed ASAP or if he's okay with putting it off > >until later -- I imagine it will take a while before he's got it > >running on Parrot well enough to hit the first assert... > > Heh. I'm OK with either leaving it as-is if the current ordering is > considered correct, or altering the benchmark to guarantee ordering. > (Or having the output compared by an external program to guarantee > correctness, which is fine as well) The ordering is supposed to be an unimportant detail. The external program solution won't work due to the nature of the benchmark. I'll fix the benchmark some time next week. The fix should be fairly simple and localized. --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis at bluewin.ch Wed Dec 31 20:39:00 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Wed Dec 31 20:35:31 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200401010050.i010oXA06606@c-24-5-183-134.client.comcast.ne t> References: <5.2.1.1.0.20040101014159.03a31f50@pop.bluewin.ch> Message-ID: <5.2.1.1.0.20040101023157.02851b70@pop.bluewin.ch> At 16:50 31.12.2003 -0800, Guido van Rossum wrote: >(There's also an issue with the distinction between repr() of int and >long, which will disappear in Python 3.0, and repr() of str >vs. unicode, which already doesn't exist in Jython. So I predict >that the benchmark currently doesn't have a chance of passing on >Jython. (But it would be interesting to time it anyway -- simply >disable the raises in b0.check() and in b5.check().) Jython is still not up to run it. But yes it would be an interesting thing. Indeed in any case some other things wrt representation would be a problem still: Jython 2.2a0 on java1.4.0_02 (JIT: null) Type "copyright", "credits" or "license" for more information. >>> u'a' 'a' >>> def f(): pass ... >>> f >>> class X: ... pass ... >>> X >>> ... In general <...> reprs vary. From guido at python.org Wed Dec 31 23:44:01 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 23:44:07 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: Your message of "Thu, 01 Jan 2004 02:39:00 +0100." <5.2.1.1.0.20040101023157.02851b70@pop.bluewin.ch> References: <5.2.1.1.0.20040101014159.03a31f50@pop.bluewin.ch> <5.2.1.1.0.20040101023157.02851b70@pop.bluewin.ch> Message-ID: <200401010444.i014i1o06815@c-24-5-183-134.client.comcast.net> > Jython is still not up to run it. But yes it would be an interesting thing. Maybe you could find a sponsor for Jython development...? > Indeed in any case some other things wrt representation would be a problem > still: > > Jython 2.2a0 on java1.4.0_02 (JIT: null) > Type "copyright", "credits" or "license" for more information. > >>> u'a' > 'a' > >>> def f(): pass > ... > >>> f > > >>> class X: > ... pass > ... > >>> X > > >>> > ... > > In general <...> reprs vary. But I've seen enough people write code that parses <...> reprs in some way to make me think that maybe they should be standardized somewhat, at least to the point where different Python implementations should not differ gratuitously. E.g. Jython could be much closer to CPython by inserting 'at 0x'. It's not like standardizing this would close off an important implementation freedom for other Python implementation. (I won't go as far as requiring that the number should be the same as hex(id(x)). :-) --Guido van Rossum (home page: http://www.python.org/~guido/)