From tim.hochberg at ieee.org Wed Dec 1 06:35:30 2004 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed Dec 1 06:35:39 2004 Subject: [Python-Dev] Roster Deadline Message-ID: <41AD5822.2000501@ieee.org> Hi Larry, FYI: I asked EB about the roster deadline and she says that she doesn't know when it is either. Checking on the Lei Out web page didn't help much either. So, you are no wiser now than at the start of this message. -tim From fdrake at acm.org Wed Dec 1 07:34:51 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed Dec 1 07:35:12 2004 Subject: [Python-Dev] Python.org current docs In-Reply-To: <200411301612.37180.fdrake@acm.org> References: <8y8jrufo.fsf@python.net> <200411301612.37180.fdrake@acm.org> Message-ID: <200412010134.51160.fdrake@acm.org> On Tuesday 30 November 2004 02:46 pm, Thomas Heller wrote: > http://www.python.org/doc/current/ > and > http://docs.python.org/ > > > still point to 2.3.4 docs. I think everything is properly updated now. Please let me know if I've missed anything. -Fred -- Fred L. Drake, Jr. From mal at egenix.com Wed Dec 1 10:23:16 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Wed Dec 1 10:23:12 2004 Subject: [Python-Dev] File encodings In-Reply-To: <20041130130934.GC6610@burma.localdomain> References: <20041129190448.GA23399@burma.localdomain> <41AC34D2.1060909@egenix.com> <20041130130934.GC6610@burma.localdomain> Message-ID: <41AD8D84.7070903@egenix.com> Gustavo Niemeyer wrote: > [...] > >>The idiom presented by Bob is the right way to go: wrap >>sys.stdout with a StreamWriter. > > I don't see that as a good solution, since every Python software > that is internationalizaed will have do figure out this wrapping, > introducing extra overhead unnecessarily. I don't see any unnecessary overhead and using the wrappers is really easy, e.g.: # # Application uses Latin-1 for I/O, terminal uses UTF-8 # import codecs, sys # Make stdout translate Latin-1 output into UTF-8 output sys.stdout = codecs.EncodedFile(sys.stdout, 'latin-1', 'utf-8') # Have stdin translate Latin-1 input into UTF-8 input sys.stdin = codecs.EncodedFile(sys.stdin, 'utf-8', 'latin-1') We should probably extend the support in StreamRecoder (which is used by the above EncodedFile helper) to also support Unicode input to .write() and have a special codec 'unicode' that converts Unicode to Unicode, so that you can request the EncodedFile object to return Unicode for .read(). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 01 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From kiko at async.com.br Wed Dec 1 14:08:25 2004 From: kiko at async.com.br (Christian Robottom Reis) Date: Wed Dec 1 14:08:34 2004 Subject: [Python-Dev] Difflib modifications [reposted] Message-ID: <20041201130825.GQ4847@async.com.br> [Reposted to python-dev!] Hello there, We've has done some customizations to difflib to make it work well with pagetests we are running on a project at Canonical, and we are looking for some guidance as to what's the best way to do them. There are some tricky bits that have to do with how the class inheritance is put together, and since we would want to avoid duplicating difflib I figured we'd ask and see if some grand ideas come up. A [rough first cut of the] patch is inlined below. Essentially, it does: - Implements a custom Differ.fancy_compare function that supports ellipsis and omits equal content - Hacks _fancy_replace to skip ellipsis as well. - Hacks best_ratio and cutoff. I'm a bit fuzzy on why this was changed, to be honest, and Celso's travelling today, but IIRC it had to do with how difflib grouped changes. Essentially, what we aim for is: - Ignoring ellipsisized(!) content - Omitting content which is equal I initially thought the best way to do this would be to inherit from SequenceMatcher and make it not return opcodes for ellipsis. However, there is no easy way to replace the class short of rewriting major bits of Differ. I suspect this could be easily changed to use a class attribute that we could override, but let me know what you think of the whole thing. --- /usr/lib/python2.3/difflib.py 2004-11-18 20:05:38.720109040 -0200 +++ difflib.py 2004-11-18 20:24:06.731665680 -0200 @@ -885,6 +885,45 @@ for line in g: yield line + def fancy_compare(self, a, b): + """ + >>> import difflib + >>> engine = difflib.Differ() + >>> got = ['World is Cruel', 'Dudes are Cool'] + >>> want = ['World ... Cruel', 'Dudes ... Cool'] + >>> list(engine.fancy_compare(want, got)) + [] + + """ + cruncher = SequenceMatcher(self.linejunk, a, b) + for tag, alo, ahi, blo, bhi in cruncher.get_opcodes(): + + if tag == 'replace': + ## replace single line + if a[alo:ahi][0].rstrip() == '...' and ((ahi - alo) == 1): + g = None + ## two lines replaced + elif a[alo:ahi][0].rstrip() == '...' and ((ahi - alo) > 1): + g = self._fancy_replace(a, (ahi - 1), ahi, + b, (bhi - 1), bhi) + ## common + else: + g = self._fancy_replace(a, alo, ahi, b, blo, bhi) + elif tag == 'delete': + g = self._dump('-', a, alo, ahi) + elif tag == 'insert': + g = self._dump('+', b, blo, bhi) + elif tag == 'equal': + # do not show anything + g = None + else: + raise ValueError, 'unknown tag ' + `tag` + + if g: + for line in g: + yield line + + def _dump(self, tag, x, lo, hi): """Generate comparison results for a same-tagged range.""" for i in xrange(lo, hi): @@ -926,7 +965,13 @@ # don't synch up unless the lines have a similarity score of at # least cutoff; best_ratio tracks the best score seen so far - best_ratio, cutoff = 0.74, 0.75 + #best_ratio, cutoff = 0.74, 0.75 + + ## reduce the cutoff to have enough similarity + ## between ' ... ' and ' blabla ' + ## for example + best_ratio, cutoff = 0.009, 0.01 + cruncher = SequenceMatcher(self.charjunk) eqi, eqj = None, None # 1st indices of equal lines (if any) @@ -981,7 +1026,11 @@ cruncher.set_seqs(aelt, belt) for tag, ai1, ai2, bj1, bj2 in cruncher.get_opcodes(): la, lb = ai2 - ai1, bj2 - bj1 - if tag == 'replace': + + if aelt[ai1:ai2] == '...': + return + + if tag == 'replace': atags += '^' * la btags += '^' * lb elif tag == 'delete': Take care, -- Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 3361 2331 From anthony at interlink.com.au Wed Dec 1 14:32:07 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Dec 1 14:33:13 2004 Subject: [Python-Dev] Re: Small subprocess patch In-Reply-To: References: Message-ID: <200412020032.07794.anthony@interlink.com.au> On Wednesday 01 December 2004 09:00, Peter Astrand wrote: > I'm also wondering if patch 1071755 and 1071764 should go into > release24-maint: > > * 1071755 makes subprocess raise TypeError if Popen is called with a > bufsize that is not an integer. Since this isn't changing anything that's user facing (just making the error handling more explicit) this is suitable for the maint branch. > * 1071764 adds a new, small utility function. Please read PEP 6. Maintenance branches are not the place for new features. For an example why, consult almost any code that requires Python 2.2 or later. Chances are you'll find, all over the place, code like: try: True, False except NameError: True, False = 1, 0 From mal at egenix.com Wed Dec 1 16:10:10 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Wed Dec 1 16:10:15 2004 Subject: [Python-Dev] MS VC compiler versions Message-ID: <41ADDED2.30707@egenix.com> Preparing for the distutils patch to allow building extensions using the .NET SDK compilers, I am compiling a list of version numbers and MS compiler logo outputs in order to use these to identify the correct compiler to use for the extensions. These are the compilers I have found so far: * MS VC6 (German version; optimizing VC6 compiler): Optimierender Microsoft (R) 32-Bit C/C++-Compiler, Version 12.00.8804, fuer x86 Copyright (C) Microsoft Corp 1984-1998. Alle Rechte vorbehalten. * MS .NET SDK 1.1 Compiler (German version; non-optimizing VC7.1 compiler): Microsoft (R) 32-Bit C/C++-Standardcompiler Version 13.10.3077 f?r 80x86 Copyright (C) Microsoft Corporation 1984-2002. Alle Rechte vorbehalten. * MS VC7.1 (aka .NET 2003, US version; optimizing VC7.1 compiler) Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.10.3077 for 80x86 Copyright (C) Microsoft Corporation 1984-2002. All rights reserved. [It looks as if optimizing vs. non-optimizing is not something that you can detect by looking at the version number.] Could you please provide me with more version numbers and logo printouts ? Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 01 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From hyeshik at gmail.com Wed Dec 1 16:43:12 2004 From: hyeshik at gmail.com (Hye-Shik Chang) Date: Wed Dec 1 16:43:15 2004 Subject: [Python-Dev] MS VC compiler versions In-Reply-To: <41ADDED2.30707@egenix.com> References: <41ADDED2.30707@egenix.com> Message-ID: <4f0b69dc0412010743352b1a8e@mail.gmail.com> On Wed, 01 Dec 2004 16:10:10 +0100, M.-A. Lemburg wrote: > Could you please provide me with more version numbers and logo > printouts ? > * MS Windows XP DDK (International version, optimizing VC 7.0): Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.00.9176 for 80x86 Copyright (C) Microsoft Corporation 1984-2001. All rights reserved. * MS VS6 SP5 (International version, optimizing VC 6.0): Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8168 for 80x86 Copyright (C) Microsoft Corp 1984-1998. All rights reserved. Hye-Shik From theller at python.net Wed Dec 1 19:14:51 2004 From: theller at python.net (Thomas Heller) Date: Wed Dec 1 19:14:03 2004 Subject: [Python-Dev] Trouble installing 2.4 In-Reply-To: <000401c4d708$ba149dc0$6402a8c0@arkdesktop> (Andrew Koenig's message of "Tue, 30 Nov 2004 13:16:34 -0500") References: <000401c4d708$ba149dc0$6402a8c0@arkdesktop> Message-ID: "Andrew Koenig" writes: > Follow-up: When I install Python as Administrator, all is well. In that > case (but not when installing it as me), it asks whether I want to install > it for all users or for myself only. I then install pywin32 and it works. > > So it may be that a caveat is in order to people who do not install 2.4 as > Administrator. As Martin guessed, a distutils bug, triggered with non-admin Python installs, and when the add-on package uses the pre-install-script or post-install-script option. Please submit a report to SF. Thanks, Thomas From trentm at ActiveState.com Wed Dec 1 19:54:31 2004 From: trentm at ActiveState.com (Trent Mick) Date: Wed Dec 1 19:55:50 2004 Subject: [Python-Dev] MS VC compiler versions In-Reply-To: <41ADDED2.30707@egenix.com> References: <41ADDED2.30707@egenix.com> Message-ID: <41AE1367.5000805@activestate.com> M.-A. Lemburg wrote: > Preparing for the distutils patch to allow building extensions > using the .NET SDK compilers, I am compiling a list of version > numbers and MS compiler logo outputs in order to use these to > identify the correct compiler to use for the extensions. > > These are the compilers I have found so far: > > * MS VC6 (German version; optimizing VC6 compiler): > > Optimierender Microsoft (R) 32-Bit C/C++-Compiler, Version 12.00.8804, > fuer x86 > Copyright (C) Microsoft Corp 1984-1998. Alle Rechte vorbehalten. * MS VC6 (US version; optimizing VC6 compiler): Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8804 for 80x86 Copyright (C) Microsoft Corp 1984-1998. All rights reserved. Trent -- Trent Mick trentm@activestate.com From aahz at pythoncraft.com Wed Dec 1 20:25:57 2004 From: aahz at pythoncraft.com (Aahz) Date: Wed Dec 1 20:25:59 2004 Subject: [Python-Dev] Roster Deadline In-Reply-To: <41AD5822.2000501@ieee.org> References: <41AD5822.2000501@ieee.org> Message-ID: <20041201192557.GA24980@panix.com> On Tue, Nov 30, 2004, Tim Hochberg wrote: > > Hi Larry, > > FYI: I asked EB about the roster deadline and she says that she doesn't > know when it is either. Checking on the Lei Out web page didn't help > much either. > > So, you are no wiser now than at the start of this message. We're even less wise now given that you probably didn't intend this for python-dev. ;-) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From steven.bethard at gmail.com Wed Dec 1 22:03:50 2004 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed Dec 1 22:03:54 2004 Subject: [Python-Dev] adding key argument to min and max Message-ID: This is my first post to Python dev, so I figured I should introduce myself. My name's Steven Bethard and I'm a computer science Ph.D. student at the University of Colorado at Boulder working primarily in the areas of natural language processing and machine learning. During my undergrad at the University of Arizona, I worked as a teaching assistant teaching Java for 2 1/2 years, though now that I'm at CU Boulder I pretty much only work in Python. I started getting active on the Python list about 6 months ago, and I've been watching python-dev for the last few months. On to the real question... I posted a few notes about this on the python-list and didn't hear much of a response, so I thought that maybe python-dev is the more appropriate place (since it involves a change to some of the builtin functions). For Python 2.5, I'd like to add a keyword argument 'key' to min and max like we have now for list.sort and sorted. I've needed this a couple of times now, specifically when I have something like a dict of word counts, and I want the most frequent word, I'd like to do something like: >>> d = dict(aaa=3000, bbb=2000, ccc=1000) >>> max(d, key=d.__getitem__) 'aaa' I've implemented a patch that provides this functionality, but there are a few concerns about how it works. Here's some examples of what it does now: >>> d = dict(aaa=3000, bbb=2000, ccc=1000) >>> max(d) 'ccc' >>> max(d, key=d.__getitem__) 'aaa' >>> max(d, d.__getitem__) {'aaa': 3000, 'bbb': 2000, 'ccc': 1000} >>> max(('aaa', 3000), ('bbb', 2000), ('ccc', 1000)) ('ccc', 1000) >>> max(('aaa', 3000), ('bbb', 2000), ('ccc', 1000), key=operator.itemgetter(1)) ('aaa', 3000) >>> max(('aaa', 3000), ('bbb', 2000), ('ccc', 1000), operator.itemgetter(1)) ('ccc', 1000) Note the difference between the 2nd and 3rd use of max in each example. For backwards compatibility reasons, the 'key' argument cannot be specified as a positional argument or it will look like max is being used in the max(a, b, c, ...) form. This means that a 'key' argument can *only* be specified as a keyword parameter, thus giving us the asymmetry we see in these examples. My real question then is, is this asymmetry a problem? Is it okay to have a parameter that is *only* accessable as a keyword parameter? Thanks, Steve -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy From bac at OCF.Berkeley.EDU Wed Dec 1 22:42:02 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Wed Dec 1 22:42:20 2004 Subject: [Python-Dev] TRUNK UNFROZEN; release24-maint branch has been cut In-Reply-To: <200412010023.17728.anthony@interlink.com.au> References: <200412010023.17728.anthony@interlink.com.au> Message-ID: <41AE3AAA.7010402@ocf.berkeley.edu> Anthony Baxter wrote: > I've cut the release24-maint branch, and updated the Include/patchlevel.h > on trunk and branch (trunk is now 2.5a0, branch is 2.4+) > > The trunk and the branch are now both unfrozen and suitable for checkins. > The feature freeze on the trunk is lifted. Remember - if you're checking > bugfixes into the trunk, either backport them to the branch, or else mark > the commit message with 'bugfix candidate' or 'backport candidate' or the > like. > > Next up will be a 2.3.5 release. I'm going to be travelling for a large chunk > of December (at very short notice) so it's likely that this will happen at the > start of January. OK, I will send out an email to python-list and python-announce mentioning this to the community and that if they have fixes they need to go into 2.3.5 they need to get it in ASAP so there is enough time to consider them (with no guarantee they will get in) at the end of the week if no one objects. -Brett From pje at telecommunity.com Wed Dec 1 23:10:07 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Dec 1 23:08:35 2004 Subject: [Python-Dev] adding key argument to min and max In-Reply-To: Message-ID: <5.1.1.6.0.20041201170900.02532c40@mail.telecommunity.com> At 02:03 PM 12/1/04 -0700, Steven Bethard wrote: >Is it okay to >have a parameter that is *only* accessable as a keyword parameter? Yes. From python at rcn.com Thu Dec 2 00:23:29 2004 From: python at rcn.com (Raymond Hettinger) Date: Thu Dec 2 00:26:09 2004 Subject: [Python-Dev] adding key argument to min and max In-Reply-To: Message-ID: <001e01c4d7fc$c49efa20$e841fea9@oemcomputer> [Steven Bethard] > For Python 2.5, I'd like to add a keyword argument 'key' to min and > max like we have now for list.sort and sorted. . . . > This means that a 'key' > argument can *only* be specified as a keyword parameter, thus giving > us the asymmetry we see in these examples. FWIW, in Py2.5 I plan on adding a key= argument to heapq.nsmallest() and heapq.nlargest(). There is no "assymmetry" issue with those functions, so it can be implemented cleanly. And, since min/max are essentially the same nsmallest/nlargest with n==1, your use case is covered and there is no need to mess with the min/max builtins. > I've needed this a couple of times now, specifically when > I have something like a dict of word counts, and I want the most frequent word For Py2.4, you can cover your use cases readily adding the recipe for mostcommon() to a module of favorite utilities: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/347615 Alternatively, the recipe for a bag class is a more flexible and still reasonably efficient: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/259174 Raymond Hettinger From steven.bethard at gmail.com Thu Dec 2 02:02:33 2004 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu Dec 2 02:02:36 2004 Subject: [Python-Dev] adding key argument to min and max In-Reply-To: <001e01c4d7fc$c49efa20$e841fea9@oemcomputer> References: <001e01c4d7fc$c49efa20$e841fea9@oemcomputer> Message-ID: Raymond Hettinger wrote: > [Steven Bethard] > > For Python 2.5, I'd like to add a keyword argument 'key' to min and > > max like we have now for list.sort and sorted. > . . . > > This means that a 'key' > > argument can *only* be specified as a keyword parameter, thus giving > > us the asymmetry we see in these examples. > > FWIW, in Py2.5 I plan on adding a key= argument to heapq.nsmallest() and > heapq.nlargest(). There is no "assymmetry" issue with those functions, > so it can be implemented cleanly. And, since min/max are essentially > the same nsmallest/nlargest with n==1, your use case is covered and > there is no need to mess with the min/max builtins. I don't want to put words into your mouth, so is this a vote against a key= argument for min and max? If nsmallest/nlargest get key= arguments, this would definitely cover the same cases. If a key= argument gets vetoed for min and max, I'd at least like to add a bit of documentation pointing users of min/max to nsmallest/nlargest if they want a key= argument... Steve -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy From python at rcn.com Thu Dec 2 02:32:10 2004 From: python at rcn.com (Raymond Hettinger) Date: Thu Dec 2 02:34:51 2004 Subject: [Python-Dev] adding key argument to min and max In-Reply-To: Message-ID: <002301c4d80e$beb08e00$e841fea9@oemcomputer> > I don't want to put words into your mouth, so is this a vote against a > key= argument for min and max? Right. I don't think there is any need. > If nsmallest/nlargest get key= arguments, this would definitely cover > the same cases. Right. > If a key= argument gets vetoed for min and max, I'd > at least like to add a bit of documentation pointing users of min/max > to nsmallest/nlargest if they want a key= argument... Sounds reasonable. Raymond P.S. In case you're interested, here is the patch: Index: heapq.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/heapq.py,v retrieving revision 1.27 diff -u -r1.27 heapq.py --- heapq.py 29 Nov 2004 05:54:47 -0000 1.27 +++ heapq.py 2 Dec 2004 01:32:44 -0000 @@ -307,6 +307,31 @@ except ImportError: pass +# Extend the implementations of nsmallest and nlargest to use a key= argument +_nsmallest = nsmallest +def nsmallest(n, iterable, key=None): + """Find the n smallest elements in a dataset. + + Equivalent to: sorted(iterable, key=key)[:n] + """ + if key is None: + return _nsmallest(n, iterable) + it = ((key(r), i, r) for i, r in enumerate(iterable)) # decorate + result = _nsmallest(n, it) + return [r for k, i, r in result] # undecorate + +_nlargest = nlargest +def nlargest(n, iterable, key=None): + """Find the n largest elements in a dataset. + + Equivalent to: sorted(iterable, key=key, reverse=True)[:n] + """ + if key is None: + return _nlargest(n, iterable) + it = ((key(r), i, r) for i, r in enumerate(iterable)) # decorate + result = _nlargest(n, it) + return [r for k, i, r in result] # undecorate + if __name__ == "__main__": # Simple sanity test heap = [] Index: test/test_heapq.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_heapq.py,v retrieving revision 1.16 diff -u -r1.16 test_heapq.py --- test/test_heapq.py 29 Nov 2004 05:54:48 -0000 1.16 +++ test/test_heapq.py 2 Dec 2004 01:32:44 -0000 @@ -105,13 +105,19 @@ def test_nsmallest(self): data = [random.randrange(2000) for i in range(1000)] + f = lambda x: x * 547 % 2000 for n in (0, 1, 2, 10, 100, 400, 999, 1000, 1100): self.assertEqual(nsmallest(n, data), sorted(data)[:n]) + self.assertEqual(nsmallest(n, data, key=f), + sorted(data, key=f)[:n]) def test_largest(self): data = [random.randrange(2000) for i in range(1000)] + f = lambda x: x * 547 % 2000 for n in (0, 1, 2, 10, 100, 400, 999, 1000, 1100): self.assertEqual(nlargest(n, data), sorted(data, reverse=True)[:n]) + self.assertEqual(nlargest(n, data, key=f), + sorted(data, key=f, reverse=True)[:n]) #======================================================================= ======= From gvanrossum at gmail.com Thu Dec 2 02:58:48 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Thu Dec 2 02:58:51 2004 Subject: [Python-Dev] adding key argument to min and max In-Reply-To: <002301c4d80e$beb08e00$e841fea9@oemcomputer> References: <002301c4d80e$beb08e00$e841fea9@oemcomputer> Message-ID: > > I don't want to put words into your mouth, so is this a vote against a > > key= argument for min and max? > > Right. I don't think there is any need. Hm, min and max are probably needed 2-3 orders of magnitude more frequently than nsmallest/nlargest. So I think it's reasonable to add the key= argument to min and max as well. (We didn't leave it out of sorted() because you can already do it with list.sort().) > def test_largest(self): shouldn't that be test_nlargest? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From python at rcn.com Thu Dec 2 03:04:57 2004 From: python at rcn.com (Raymond Hettinger) Date: Thu Dec 2 03:07:36 2004 Subject: [Python-Dev] adding key argument to min and max In-Reply-To: Message-ID: <002e01c4d813$529cc300$e841fea9@oemcomputer> > -----Original Message----- > From: python-dev-bounces+python=rcn.com@python.org [mailto:python-dev- > bounces+python=rcn.com@python.org] On Behalf Of Steven Bethard > Sent: Wednesday, December 01, 2004 4:04 PM > To: python-dev@python.org > Subject: [Python-Dev] adding key argument to min and max > > This is my first post to Python dev, so I figured I should introduce > myself. > > My name's Steven Bethard and I'm a computer science Ph.D. student at > the University of Colorado at Boulder working primarily in the areas > of natural language processing and machine learning. During my > undergrad at the University of Arizona, I worked as a teaching > assistant teaching Java for 2 1/2 years, though now that I'm at CU > Boulder I pretty much only work in Python. I started getting active > on the Python list about 6 months ago, and I've been watching > python-dev for the last few months. > > For Python 2.5, I'd like to add a keyword argument 'key' to min and > max like we have now for list.sort and sorted. . . . > I've implemented a patch that provides this functionality, but there > are a few concerns about how it works. Guido says yes. So, load the patch to SF and assign to me for review, testing, and documentation. Raymond From kbk at shore.net Thu Dec 2 04:15:53 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Thu Dec 2 04:15:57 2004 Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <873byp1jbq.fsf@hydra.localdomain> Patch / Bug Summary ___________________ Patches : 258 open ( +4) / 2701 closed ( +1) / 2959 total ( +5) Bugs : 812 open (+28) / 4642 closed (+13) / 5454 total (+41) RFE : 160 open ( +4) / 136 closed ( +1) / 296 total ( +5) New / Reopened Patches ______________________ #1074261 gzip dies on gz files with many appended headers (2004-11-27) http://python.org/sf/1074381 opened by Mark Eichin Flush stdout/stderr if closed (fix for bug 1074011) (2004-11-29) http://python.org/sf/1075147 opened by Ben Hutchings gcc compiler on Windows (2004-11-30) http://python.org/sf/1075887 opened by Michiel de Hoon AUTH PLAIN in smtplib (2004-11-30) http://python.org/sf/1075928 opened by James Lan readline does not need termcap (2004-12-01) http://python.org/sf/1076826 opened by Michal Čihař Patches Closed ______________ bug#1021756: more informative error message (2004-11-23) http://python.org/sf/1071739 closed by effbot New / Reopened Bugs ___________________ Bugs in _csv module - lineterminator (2004-11-24) http://python.org/sf/1072404 opened by Chris Withers mkarg undocumented (2004-11-24) http://python.org/sf/1072410 opened by Gunter Ohrner email as_string() omits trailing newline (2004-11-24) CLOSED http://python.org/sf/1072623 opened by Tessa Lau dyld: ./python.exe multiple definitions of symbol _BC (2004-11-24) http://python.org/sf/1072642 opened by Marius thisid not intialized in pindent.py script (2004-11-24) http://python.org/sf/1072853 opened by Niraj Bajpai ^Z doesn't exit interpreter - 2.4c1 & Win2K (2004-11-26) http://python.org/sf/1073736 opened by Kent Johnson subprocess.py doc bug in 2.4c1 (2004-11-26) CLOSED http://python.org/sf/1073790 opened by Dan Christensen 2 XML parsing errors (2004-11-26) CLOSED http://python.org/sf/1073864 opened by Peer Janssen write failure ignored in Py_Finalize() (2004-11-27) http://python.org/sf/1074011 opened by Matthias Klose current directory in sys.path handles symlinks badly (2004-11-26) http://python.org/sf/1074015 opened by Eric M. Hopper xml.dom.minidom produces errors with certain unicode chars (2004-11-27) http://python.org/sf/1074200 opened by Peer Janssen gzip dies on gz files with many appended headers (2004-11-27) http://python.org/sf/1074261 opened by Mark Eichin input from numeric pad always dropped when numlock off (2004-11-27) http://python.org/sf/1074333 opened by Rick Graves FeedParser problem on end boundaries w/o newline (2004-11-24) http://python.org/sf/1072623 reopened by tlau Irregular behavior of datetime.__str__() (2004-11-27) http://python.org/sf/1074462 opened by Wai Yip Tung Errors and omissions in logging module documentation (2004-11-28) http://python.org/sf/1074693 opened by Joachim Boomberschloss Windows 2.4c1 installer default location issues (2004-11-28) http://python.org/sf/1074873 opened by dmerrill exceeding obscure weakproxy bug (2004-11-29) http://python.org/sf/1075356 opened by Michael Hudson urllib2.HTTPBasicAuthHandler problem with [HOST]:[PORT] (2004-11-29) http://python.org/sf/1075427 opened by O-Zone PyGILState_Ensure() deadlocks (ver 2.4c1) (2004-11-29) http://python.org/sf/1075703 opened by Andi Vajda Build Bug on Solaris. (2004-11-30) CLOSED http://python.org/sf/1075934 opened by Jeremy Whiting Memory fault pyexpat.so on SGI (2004-11-30) http://python.org/sf/1075984 opened by Maik Hertha Memory fault pyexpat.so on SGI (2004-11-30) http://python.org/sf/1075990 opened by Maik Hertha HTMLParser can't handle page with javascript (2004-11-30) http://python.org/sf/1076070 opened by Jeremy Hylton distutils.core.setup() with unicode arguments broken (2004-11-30) http://python.org/sf/1076233 opened by Walter D?rwald Whats New for 2.4 "SafeTemplate" patch. (2004-11-30) CLOSED http://python.org/sf/1076365 opened by Sean Reifschneider test_shutil fails on x86-64 // Suse 9.1 (2004-11-30) http://python.org/sf/1076467 opened by Ross G Baker Jr Another message that croaks email.FeedParser (2004-11-30) http://python.org/sf/1076485 opened by Skip Montanaro Sate/Save typo in Mac/scripts/BuildApplication.py (2004-11-30) http://python.org/sf/1076490 opened by Neil Mayhew BuildApplication includes many unneeded modules (2004-11-30) http://python.org/sf/1076492 opened by Neil Mayhew python24.msi install error (2004-12-01) http://python.org/sf/1076500 opened by guan zi jing shutil.move clobbers read-only files. (2004-12-01) http://python.org/sf/1076515 opened by Jeremy Fincher test test_codecs failed (2004-12-01) http://python.org/sf/1076790 opened by Michal Čihař test test_re produced unexpected output (2004-12-01) http://python.org/sf/1076791 opened by Michal Čihař test test_unicodedata failed (2004-12-01) http://python.org/sf/1076795 opened by Michal Čihař test test_shutil failed (2004-12-01) CLOSED http://python.org/sf/1076811 opened by Michal Čihař test_shelve failures (2004-12-01) http://python.org/sf/1076819 opened by Michal Čihař Bad IDLE shortcut by 2.4 installer on XP (2004-12-01) http://python.org/sf/1076861 opened by Jean M. Brouwers Tutorial corrections (2004-12-01) http://python.org/sf/1076955 opened by Fred L. Drake, Jr. Incorrect behaviour of StreamReader.readline leads to crash (2004-12-01) http://python.org/sf/1076985 opened by Sebastian Hartte Error building _bsddb extension (2004-12-01) http://python.org/sf/1077040 opened by Oleg Broytmann Problem testing python 2.4 (2004-12-01) http://python.org/sf/1077103 opened by Pierre Negative numbers to os.read() cause segfault (2004-12-01) http://python.org/sf/1077106 opened by Jp Calderone Bugs Closed ___________ coercing decimal to int doesn't work between -1 and 1 (2004-11-23) http://python.org/sf/1071588 closed by rhettinger OS X (Panther) Framework Install causes version mismatch (2004-11-18) http://python.org/sf/1069198 closed by bcannon urllib2 authentication redirection error(?) (2004-11-22) http://python.org/sf/1071147 closed by facundobatista shelve update fails on "large" entry (2002-02-27) http://python.org/sf/523425 closed by facundobatista new int overflow handling needs docs (2001-08-23) http://python.org/sf/454446 closed by facundobatista subprocess.py doc bug in 2.4c1 (2004-11-26) http://python.org/sf/1073790 closed by astrand 2 XML parsing errors (2004-11-26) http://python.org/sf/1073864 closed by peerjanssen gzip module cannot write large files (2003-02-08) http://python.org/sf/683061 closed by jlgijsbers FeedParser problem on end boundaries w/o newline (2004-11-24) http://python.org/sf/1072623 closed by bwarsaw FeedParser problem on end boundaries w/o newline (2004-11-24) http://python.org/sf/1072623 closed by bwarsaw Build Bug on Solaris. (2004-11-30) http://python.org/sf/1075934 closed by whitingjr Whats New for 2.4 "SafeTemplate" patch. (2004-11-30) http://python.org/sf/1076365 closed by akuchling 'Demo/embed/' , It crash (2003-12-04) http://python.org/sf/853862 closed by guanzijing test test_shutil failed (2004-12-01) http://python.org/sf/1076811 closed by nijel New / Reopened RFE __________________ Extension to optparse: options with facultative value (2004-11-25) http://python.org/sf/1073198 opened by pollastri Extension to optparse: options with facultative value (2004-11-25) CLOSED http://python.org/sf/1073305 opened by pollastri ignore element format character for PyArg_ParseTuple (2004-11-30) http://python.org/sf/1075902 opened by Sean Proctor Python Interpreter embedded in small memory system (2004-12-01) http://python.org/sf/1076478 opened by stoneabcwujx RFE Closed __________ Extension to optparse: options with facultative value (2004-11-25) http://python.org/sf/1073305 closed by nnorwitz From phd at phd.pp.ru Wed Dec 1 21:47:21 2004 From: phd at phd.pp.ru (Oleg Broytmann) Date: Thu Dec 2 11:46:39 2004 Subject: [Python-Dev] Python 2.4 README Message-ID: <20041201204721.GA6993@phd.pp.ru> Hello. Python 2.4 README still thinks it is 2.3. README lines 230-231: "A number of features are not supported in Python 2.3 anymore. Some support code is still present, but will be removed in Python 2.4." "Will be"? Isn't it 2.4 already? :) Line 322: "Tcl to support it. To compile Python2.3 with Tkinter, you will" Python2.4, I suppose? Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From bjoti at home.se Wed Dec 1 23:19:22 2004 From: bjoti at home.se (Bjorn Tillenius) Date: Thu Dec 2 11:46:40 2004 Subject: [Python-Dev] Unicode in doctests Message-ID: <20041201221922.GA26263@home.se> There are some issues regarding the use of unicode in doctests. Consider the following three tests. >>> foo = u'f??' >>> foo u'f\xf6\xf6' >>> foo u'f\u00f6\u00f6' >>> foo u'f??' To me, those are identical. At least string comparison shows that u'f\xf6\xf6' == u'f\u00f6\u00f6' == u'f??'. Yet, only the first one of the tests passes, the other two fail. And that's if the tests are within a doc string, where I can specify the encoding used. If DocFileSuite is being used, there's no way of specify the encoding, thus all tests will fail. Is it supposed to be like this, or have I missed something? If I could specify the encoding for DocFileSuite to use, I would at least be partially happy. Regards, Bjorn From niemeyer at conectiva.com Thu Dec 2 17:31:06 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Thu Dec 2 17:31:13 2004 Subject: [Python-Dev] SRE bug and notifications Message-ID: <20041202163106.GA16046@burma.localdomain> I'm really ashamed. The SRE bug #1072259, reported in 2004-11-24, and fixed a few minutes ago, got into 2.4 final. The only reason it wasn't fixed on time for 2.4 is because I wasn't aware about it. :-( Is there any way to get notified about certain kinds of bugs in SourceForge? Or, is there any way to follow all new bugs posted? Or even, what's the easiest way to avoid that problem by being notified of bugs as soon as possible? Thank you, and I'm sorry. -- Gustavo Niemeyer http://niemeyer.net From amk at amk.ca Thu Dec 2 17:46:07 2004 From: amk at amk.ca (A.M. Kuchling) Date: Thu Dec 2 17:46:12 2004 Subject: [Python-Dev] SRE bug and notifications In-Reply-To: <20041202163106.GA16046@burma.localdomain> References: <20041202163106.GA16046@burma.localdomain> Message-ID: <20041202164607.GA8027@rogue.amk.ca> On Thu, Dec 02, 2004 at 02:31:06PM -0200, Gustavo Niemeyer wrote: > in SourceForge? Or, is there any way to follow all new bugs > posted? Or even, what's the easiest way to avoid that problem > by being notified of bugs as soon as possible? Kurt's weekly bug summaries list all new bugs. For a sample, see http://mail.python.org/pipermail/python-list/2004-December/252968.html ; they get posted to python-dev, too. Perhaps the summaries should include an "unassigned bugs" list to nag us to look at bugs and assign them to the right person. --amk From niemeyer at conectiva.com Thu Dec 2 17:48:54 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Thu Dec 2 17:49:00 2004 Subject: [Python-Dev] Re: SRE bug and notifications In-Reply-To: <20041202163106.GA16046@burma.localdomain> References: <20041202163106.GA16046@burma.localdomain> Message-ID: <20041202164854.GA16821@burma.localdomain> > Is there any way to get notified about certain kinds of bugs > in SourceForge? Or, is there any way to follow all new bugs > posted? Or even, what's the easiest way to avoid that problem > by being notified of bugs as soon as possible? After some googling I've found python-bugs-list. Mentioning it in http://www.python.org/community/lists.html might be a good idea. Additional suggestions still accepted. Thanks! -- Gustavo Niemeyer http://niemeyer.net From tim.peters at gmail.com Thu Dec 2 18:02:51 2004 From: tim.peters at gmail.com (Tim Peters) Date: Thu Dec 2 18:02:53 2004 Subject: [Python-Dev] SRE bug and notifications In-Reply-To: <20041202163106.GA16046@burma.localdomain> References: <20041202163106.GA16046@burma.localdomain> Message-ID: <1f7befae041202090237b3d159@mail.gmail.com> FYI, I just changed SF so that bugs with Category "Regular Expressions" are auto-assigned to Gustavo (they were being auto-assigned to Fredrik, which doesn't appear to do much good anymore). [Gustavo Niemeyer] > I'm really ashamed. The SRE bug #1072259, reported in > 2004-11-24, and fixed a few minutes ago, got into 2.4 final. The > only reason it wasn't fixed on time for 2.4 is because I wasn't > aware about it. :-( Unfortunately, that particular bug was in a wrong Category, so the new auto-assign wouldn't have helped here anyway. From python at rcn.com Fri Dec 3 00:17:25 2004 From: python at rcn.com (Raymond Hettinger) Date: Fri Dec 3 00:20:05 2004 Subject: [Python-Dev] [Python-checkins] python/dist/src/Doc/lib liblogging.tex, 1.33, 1.34 Message-ID: <006301c4d8c5$15972380$e841fea9@oemcomputer> Reminder: The head is now for Py2.5. Please also do a checkin for release24-maint. Raymond From noamr at myrealbox.com Fri Dec 3 01:55:28 2004 From: noamr at myrealbox.com (Noam Raphael) Date: Fri Dec 3 01:55:16 2004 Subject: [Python-Dev] An import hook which does nothing Message-ID: <41AFB980.3050705@myrealbox.com> Hello, I'm currently writing an import hook which will cache information on the local disk, instead of retrieving it every time from the slow NFS where I work. To make sure that I understand how import hooks work, and as a starting point, I wrote a dummy import hook, which is supposed to behave like the python's built-in import mechanism. I post it here, for two reasons: 1. So that developers who understand the import mechanism better than I do may review it and find things which I didn't do exactly right. 2. Because I think other people might find it helpful when writing new import hooks, as a starting point as well as for better understanding -- there's nothing like a working example to help you settle up on what does which where. (Although perhaps a few more comments, in addition to those which I wrote, might help...) (Note: I wrote the "DEBUG" parts in order to make sure that my importer works, because if it fails things might be done by the built-in importer and I won't notice. These parts can be removed, of course.) Do you think that it might be useful? Maybe something like that can go into the "examples" section of the imp module? Thanks, Noam Raphael -------------- next part -------------- A non-text attachment was scrubbed... Name: importer.py Type: text/x-python Size: 1910 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20041203/378063b9/importer.py From sxanth at ceid.upatras.gr Fri Dec 3 10:54:25 2004 From: sxanth at ceid.upatras.gr (Stelios Xanthakis) Date: Fri Dec 3 10:54:31 2004 Subject: [Python-Dev] PEP: __source__ proposal Message-ID: Hi all. Now that 2.4 is out and everything maybe it's about time to start discussing the "use the __source__ Luke" feature which IMO will really boost python into a new domain of exciting possibilities. I've prepared a pre-PEP which is not very good but it is a base. In short, the feature is good and it enables editing of python code at runtime instead of the runfile-exit-edit-run-exit-edit-run cycle. We have the following possibilities as to whether __source__ data is marshalled and the feature is always enabled. [1] Command line switch and not marshalled [2] Always on and not marshalled [3] Always on and marshalled There is also [4] which doesn't make much sense. If I was BDFL I'd go for [1] so whoever wants it can enable it and whoever doesn't can't complain, and they'll all leave me alone. Phillip J. Eby expressed some concerns that the modules that depend on __source__ will eventually take over and it will become a standard. Anyway, the PEP is attached. You can mail me with votes on the feature and if you want on your preferred option from 1,2,3. If I get votes I'll post the results later. If this is accepted I'll try to come up with a good patch vs 2.4 Thanks, St. -------------------ATTACHED PYTHON ENHANCEMENT PROPOSAL--- PEP: XXX Title: The __source__ attribute Version: $Revision: 1.10 $ Last-Modified: $Date: 2003/09/22 04:51:49 $ Author: Stelios Xanthakis Status: Draft Type: Standards Track Content-Type: text/plain Created: 19-Nov-2004 Python-Version: 2.4.1 Post-History: Abstract This PEP suggests the implementation of __source__ attribute for functions and classes. The attribute is a read-only string which is generated by the parser and is a copy of the original source code of the function/class (including comments, indentation and whitespace). Motivation It is generally a tempting idea to use python as an interface to a program. The developers can implement all the functionality and instead of designing a user interface, provide a python interpreter to their users. Take for example one of the existing web browsers: they have everything that would be needed to write a script which downloads pages automatically or premutates the letters of web pages before they are displayed, but it is not possible for the user to do these things because the interface of these applications is static. A much more powerful approach would be an interface which is dynamically constructed by the user to meet the user's needs. The most common development cycle of python programs is: write .py file - execute .py file - exit - enhance .py file - execute .py file - etc. With the implementation of the __source__ attribute though the development/modification of python code can happen at run-time. Functions and classes can be defined, modified or enhanced while the python shell is running and all the changes can be saved by saving the __source__ attribute of globals before termination. Moreover, in such a system it is possible to modify the "code modification routines" and eventually we have a self-modifying interface. Using a program also means improving its usability. The current solution of using 'inspect' to get the source code of functions is not adequate because it doesn't work for code defined with "exec" and it doesn't have the source of functions/classes defined in the interactive mode. Generally, a "file" is something too abstract. What is more real is the data received by the python parser and that is what is stored in __source__. Specification The __source__ attribute is a read-only attribute of functions and classes. Its type is string or None. In the case of None it means that the source was not available. The indentation of the code block is the original identation obeying nested definitions. For example: >>> class A: ... def foo (self): ... print """Santa-Clauss ... is coming to town""" >>> def spam (): ... def closure (): ... pass ... return closure >>> print A.foo.__source__ def foo (self): print """Santa-Clauss is coming to town""" >>> print spam().__source__ def closure (): pass The attribute is not marshaled and therefore not stored in ".pyc" files. As a consequence, functions and classes of imported modules have __source__==None. We propose that the generation of __source__ will be controlled by a command line option. In the case this feature is not activated by the command line option, the attribute is absent. Rationale Generally, "import" refers to modules that either have a file in a standard location or they are distributed in ".pyc" form only. Therefore in the case of modules, getting the source with "inspect" is adequate. Moreover, it does not make sense saving __source__ in ".pyc" because the point would be to save modifications in the original ".py" file (if available). On the issue of the command-line option controlling the generation of __source__, please refer to the section about the overhead of this feature. The rationale is that those applications that do not wish to use this feature can avoid it (cgi scripts in python benchmarked against another language). Overhead The python's parser is not exactly well-suited for such a feature. Execution of python code goes through the stages of lexical analysis, tokenization, generation of AST and execution of bytecode. In order to implement __source__, the tokenizer has to be modified to store the lines of the current translation unit. Those lines are then attached the root node of the AST. While the AST is compiled we have to keep a reference of the current node in order to be able to find the next node after the node for which we wish to generate __source__, get the first and the last line of our block and then refer to the root node to extract these lines and make a string. All these actions add a minor overhead to some heavily optimized parts of python. However, once compilation to bytecode is done, this feature no longer affects the performance of the execution of the bytecode. There is also the issue of the memory spent to store __source__. In our opinion, this is worth the tradeoff for those who are willing to take advantage of it. Implementation There is a sample implementation at [2] which consists of a patch against python 2.3.4. The patch has to be improved to avoid generating __source__ for the case we are importing modules for the first time (not from .pyc). In the sample implementation there is also included a sample shell that takes advantage of __source__ and demonstrates some aspects that motivated us towards patching python and submitting this PEP. References [1] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html [2] Sample implementation http://students.ceid.upatras.gr/~sxanth/ISYSTEM/python-PIESS.tar.gz Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: From hpk at trillke.net Fri Dec 3 13:36:16 2004 From: hpk at trillke.net (holger krekel) Date: Fri Dec 3 13:36:20 2004 Subject: [Python-Dev] PEP: __source__ proposal In-Reply-To: References: Message-ID: <20041203123616.GB4708@solar.trillke.net> Hi Stelios, [Stelios Xanthakis Fri, Dec 03, 2004 at 11:54:25AM +0200] > Abstract > > This PEP suggests the implementation of __source__ attribute for > functions and classes. The attribute is a read-only string which > is generated by the parser and is a copy of the original source > code of the function/class (including comments, indentation and > whitespace). I've had similar ideas in the past as we are doing dynamic code generation in PyPy, as well as in other projects. After some discussion with Armin i think there is another possibility for storing "source" or any other such meta information with code/module objects: make __file__ and co_filename instances of a subclass of 'str' providing an extra attribute. For a simple example, they could have a 'source' attribute, which could be tried first by appropriate inspect functions and traceback related functionality. We are about to test out this approach with the py lib (http://codespeak.net/py) and want to have it work for for Python 2.2, 2.3. and 2.4. I suspect there may be some issues lurking (also in your proposed PEP) especially with respect to encodings. Also we have some use cases where we want to retrieve source code from non-local locations and want to integrate this seemlessly with the introspection facilities of Python which obviously is an important part of the equation. I can report back if there is interest. cheers, holger From fredrik at pythonware.com Fri Dec 3 14:38:04 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri Dec 3 14:38:13 2004 Subject: [Python-Dev] Re: Unicode in doctests References: <20041201221922.GA26263@home.se> Message-ID: Bjorn Tillenius wrote: > There are some issues regarding the use of unicode in doctests. Consider > the following three tests. > > >>> foo = u'föö' > >>> foo > u'f\xf6\xf6' > > >>> foo > u'f\u00f6\u00f6' > > >>> foo > u'föö' > > To me, those are identical. really? if a function is expected to print "öl", is it alright to print "\u00f6l" instead? wouldn't you complain if your newspaper used Unicode escapes in headlines instead of Swedish characters? > Is it supposed to be like this, or have I missed something? If I could > specify the encoding for DocFileSuite to use, I would at least be > partially happy. repr() always generates the same output, no matter what encoding you use. just use repr, and you're done. From mal at egenix.com Fri Dec 3 15:59:43 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Dec 3 15:59:48 2004 Subject: [Python-Dev] Removing --with-wctype-functions support Message-ID: <41B07F5F.9040207@egenix.com> I would like to remove the support for using libc wctype functions (e.g. towupper(), towlower(), etc.) from the code base. The reason is that compiling Python using this switch not only breaks the test suite, it also causes the functions .lower() and .upper() to become locale aware and creates wrong results as a result (which are hard to track down if you don't know whether Python was compiled with the wctype switch or not). The argument for having this switch at the time was to reduce the interpreter size. This was true for Python 1.6 since the Unicode type database was conditionally removed. Starting with Python 2.0 a different approach was taken which resulted in having to keep the type database in one piece - even with the switch enabled. As a result, performance became the only argument. However, because the libc functions are locale aware, the added overhead causes the libc functions not to perform better than the builtin Python versions. In the end, we loose complexity and reduce the Python configuration space by one dimension. Here's the bug that revealed the problem: https://sourceforge.net/tracker/index.php?func=detail&aid=1076790&group_id=5470&atid=105470 Question is: Should this be done in 2.4.1 or will it have to wait until 2.5 ? We might want to remove the support for 2.4.1 and simply ignore the compile time switch (or print a warning). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 03 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at egenix.com Fri Dec 3 16:08:54 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Dec 3 16:08:58 2004 Subject: [Python-Dev] MS VC compiler versions In-Reply-To: <41AE1367.5000805@activestate.com> References: <41ADDED2.30707@egenix.com> <41AE1367.5000805@activestate.com> Message-ID: <41B08186.5070600@egenix.com> Here's an update of the list I currently have: * MS VS6 SP5 (International version, optimizing VC6 compiler with SP5): Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8168 for 80x86 Copyright (C) Microsoft Corp 1984-1998. All rights reserved. * MS VC6 (German version; optimizing VC6 compiler with SP6): Optimierender Microsoft (R) 32-Bit C/C++-Compiler, Version 12.00.8804, fuer x86 Copyright (C) Microsoft Corp 1984-1998. Alle Rechte vorbehalten. * MS VC6 (US version; optimizing VC6 compiler with SP6): Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8804 for 80x86 Copyright (C) Microsoft Corp 1984-1998. All rights reserved. * MS .NET SDK 1.1 Compiler (German version; non-optimizing VC7.1 compiler): Microsoft (R) 32-Bit C/C++-Standardcompiler Version 13.10.3077 f?r 80x86 Copyright (C) Microsoft Corporation 1984-2002. Alle Rechte vorbehalten. * MS VC7.1 (aka .NET 2003, German version; optimizing VC7.1 compiler) Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.10.3077 for 80x86 Copyright (C) Microsoft Corporation 1984-2002. All rights reserved. * MS Windows XP DDK (International version, optimizing VC 7.0): Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.00.9176 for 80x86 Copyright (C) Microsoft Corporation 1984-2001. All rights reserved. Does someone have the details for the MS Toolkit compiler ? Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 03 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Scott.Daniels at Acm.Org Fri Dec 3 17:07:25 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Fri Dec 3 17:06:06 2004 Subject: [Python-Dev] Re: MS VC compiler versions In-Reply-To: <41B08186.5070600@egenix.com> References: <41ADDED2.30707@egenix.com> <41AE1367.5000805@activestate.com> <41B08186.5070600@egenix.com> Message-ID: M.-A. Lemburg wrote: > Here's an update of the list I currently have: > > * MS VS6 SP5 (International version, optimizing VC6 compiler with SP5): > > Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8168 for 80x86 > Copyright (C) Microsoft Corp 1984-1998. All rights reserved. > > * MS VC6 (German version; optimizing VC6 compiler with SP6): > > Optimierender Microsoft (R) 32-Bit C/C++-Compiler, Version 12.00.8804, > fuer x86 > Copyright (C) Microsoft Corp 1984-1998. Alle Rechte vorbehalten. > > * MS VC6 (US version; optimizing VC6 compiler with SP6): > > Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8804 for 80x86 > Copyright (C) Microsoft Corp 1984-1998. All rights reserved. > > * MS .NET SDK 1.1 Compiler (German version; non-optimizing VC7.1 compiler): > > Microsoft (R) 32-Bit C/C++-Standardcompiler Version 13.10.3077 f?r 80x86 > Copyright (C) Microsoft Corporation 1984-2002. Alle Rechte vorbehalten. > > * MS VC7.1 (aka .NET 2003, German version; optimizing VC7.1 compiler) > > Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.10.3077 for 80x86 > Copyright (C) Microsoft Corporation 1984-2002. All rights reserved. > > * MS Windows XP DDK (International version, optimizing VC 7.0): > > Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.00.9176 for 80x86 > Copyright (C) Microsoft Corporation 1984-2001. All rights reserved. > > > Does someone have the details for the MS Toolkit compiler ? > > Thanks, \Program Files\Microsoft Visual C++ Toolkit 2003\bin\cl.exe: Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.10.3077 for 80x86 Copyright (C) Microsoft Corporation 1984-2002. All rights reserved. -- -- Scott David Daniels Scott.Daniels@Acm.Org From bjoti at yahoo.se Fri Dec 3 18:02:52 2004 From: bjoti at yahoo.se (Bjorn Tillenius) Date: Fri Dec 3 18:20:43 2004 Subject: [Python-Dev] Re: Unicode in doctests References: <20041201221922.GA26263@home.se> Message-ID: Fredrik Lundh pythonware.com> writes: > Bjorn Tillenius wrote: > > There are some issues regarding the use of unicode in doctests. Consider > > the following three tests. > > > > >>> foo = u'f??' > > >>> foo > > u'f\xf6\xf6' > > > > >>> foo > > u'f\u00f6\u00f6' > > > > >>> foo > > u'f??' > > > > To me, those are identical. > > really? if a function is expected to print "?l", is it alright to print > "\u00f6l" instead? wouldn't you complain if your newspaper used > Unicode escapes in headlines instead of Swedish characters? No, I wouldn't like the newspaper to use Unicode escapes. For the same reason, I don't want my documentation to contain Unicode escapes. That's why I would like the latter test to pass. But I understand, it tries to match the output of repr(foo), I guess I can live with that. I can always do: >>> foo == u'f??' True On the other hand, since there already are some flags to modify the matching algorithm, one could argue for adding another flag... or at least provide the possibility for the user to alter the matching himself. Although it's not that important for me. > > Is it supposed to be like this, or have I missed something? If I could > > specify the encoding for DocFileSuite to use, I would at least be > > partially happy. > > repr() always generates the same output, no matter what encoding > you use. just use repr, and you're done. What is important for me, though, is to be able to specify an encoding to DocFileSuite. As you said, one doesn't want to read Unicode escapes. At the moment none of the tests I've given as example will pass in a DocFileSuite (given that the text file is encoded using UTF-8). I do find it a bit strange that I can't just copy a doctest within a docstring to a text file. I have to Unicode escape all non-ASCII characters, which produces ugly documentation. Regards, Bjorn From kbk at shore.net Fri Dec 3 18:40:42 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Fri Dec 3 18:42:14 2004 Subject: [Python-Dev] SRE bug and notifications In-Reply-To: <20041202164607.GA8027@rogue.amk.ca> (A. M. Kuchling's message of "Thu, 2 Dec 2004 11:46:07 -0500") References: <20041202163106.GA16046@burma.localdomain> <20041202164607.GA8027@rogue.amk.ca> Message-ID: <87653jxoth.fsf@hydra.localdomain> "A.M. Kuchling" writes: > Perhaps the summaries should include an "unassigned bugs" list to nag > us to look at bugs and assign them to the right person. The bug report is derived from the bug and patch email lists, so that information isn't available without scraping it off SF. However, the SF trackers can be set to query for Unassigned / Open. It looks like about 2/3 of the open bugs and 3/4 of the open patches are unassigned. I often unassign IDLE bugs that I'm not planning to fix right now, hoping that may encourage someone to step forward by removing the illusion that something is happening. Conversely, if I make progress on an unassigned bug/patch, I'll assign it to myself as an indication that it's being worked on, avoiding duplication of effort. I track IDLE bugs via the IDLE category. That's very useful, maybe we need more categories? -- KBK From sxanth at ceid.upatras.gr Fri Dec 3 22:59:30 2004 From: sxanth at ceid.upatras.gr (Stelios Xanthakis) Date: Fri Dec 3 22:59:40 2004 Subject: [Python-Dev] PEP: __source__ proposal In-Reply-To: <20041203123616.GB4708@solar.trillke.net> References: <20041203123616.GB4708@solar.trillke.net> Message-ID: On Fri, 3 Dec 2004, holger krekel wrote: > ... > there is another possibility for storing "source" or any other such meta > information with code/module objects: make __file__ and co_filename > instances of a subclass of 'str' providing an extra attribute. > For a simple example, they could have a 'source' attribute, which > could be tried first by appropriate inspect functions and traceback > related functionality. Attaching such info on 'code objects' is indeed a more general case. But, OTOH, AFAIK, a class is not a code object. At least by what I was able to figure out from python sources. It seems reasonable to make 'source' a dynamic object which will get its info from file/line if available. Now the thing is that if we had __source__ from the start, 'inspect' would have been much different. So the fact that we have some functionality with inspect does not mean that it's good enough. Probably inspect will be rewritten/improved if __source__ is implemented. > We are about to test out this approach with the py lib > (http://codespeak.net/py) and want to have it work for > for Python 2.2, 2.3. and 2.4. Do you plan hacking python ? It appears that tok_nextc() is the best place to catch all the source passed to the interpreter. A patch would be interesting. Stelios From martin at v.loewis.de Sat Dec 4 00:26:33 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Dec 4 00:26:32 2004 Subject: [Python-Dev] PEP: __source__ proposal In-Reply-To: References: Message-ID: <41B0F629.4040100@v.loewis.de> Stelios Xanthakis wrote: > Now that 2.4 is out and everything maybe it's > about time to start discussing the "use the > __source__ Luke" feature which IMO will really > boost python into a new domain of exciting > possibilities. I'm opposed to this idea. It creates overhead in the size of .pyc files, for no additional value that couldn't be obtained otherwise. As the rationale, the PEP lists: 1. > It is generally a tempting idea to use python as an interface to > a program. I cannot see how this rationale is related to the PEP. You can use Python as interface to a program with or without __source__. 2. > The developers can implement all the functionality > and instead of designing a user interface, provide a python > interpreter to their users. This does not require __source, either. 3. > A much more powerful approach would be an interface which is > dynamically constructed by the user to meet the user's needs. Dynamic code generation doesn't require __source__, either. 4. > The most common development cycle of python programs is: > write .py file - execute .py file - exit - enhance .py file - > execute .py file - etc. With the implementation of the __source__ > attribute though the development/modification of python code > can happen at run-time. This works just fine as well at the moment; see IDLE for an example. > Functions and classes can be defined, > modified or enhanced while the python shell is running and > all the changes can be saved by saving the __source__ attribute > of globals before termination. Currently, you can define classes dynamically, and you can also save the source code of the class to a file in case you need it later. > Moreover, in such a system > it is possible to modify the "code modification routines" and > eventually we have a self-modifying interface. Using a > program also means improving its usability. Self-modifying source code is currently also possible. Just read the old source code from a .py file, modify it, and recompile it. > The current solution of using 'inspect' to get the source > code of functions is not adequate because it doesn't work > for code defined with "exec" and it doesn't have the source > of functions/classes defined in the interactive mode. I fail to see why it isn't adequate. Anybody who wants to modify source code that was originally passed to exec just needs to preserve a copy of the source code, separately. > Generally, > a "file" is something too abstract. What is more real is the > data received by the python parser and that is what is stored > in __source__. Not at all. A file is precisely the level of granularity that is burnt into the Python language. A module is *always* a file, executed from top to bottom. It is not possible to recreate the source code of a module if you have only the source code of all functions, and all classes. Regards, Martin From edcjones at erols.com Fri Dec 3 03:24:45 2004 From: edcjones at erols.com (Edward C. Jones) Date: Sat Dec 4 03:23:03 2004 Subject: [Python-Dev] test_shutils.py fails for 2.4 install Message-ID: <41AFCE6D.10300@erols.com> I have a PC with an AMD cpu and Mandrake 10.1. While installing Python 2.4 "make test" failed in "test_shutils.py". Here is the output of "./python ./Lib/test/test_shutil.py": test_dont_copy_file_onto_link_to_itself (__main__.TestShutil) ... ok test_dont_move_dir_in_itself (__main__.TestShutil) ... ok test_on_error (__main__.TestShutil) ... FAIL test_rmtree_dont_delete_file (__main__.TestShutil) ... ok test_rmtree_errors (__main__.TestShutil) ... ok ====================================================================== FAIL: test_on_error (__main__.TestShutil) ---------------------------------------------------------------------- Traceback (most recent call last): File "./Lib/test/test_shutil.py", line 34, in test_on_error self.assertEqual(self.errorState, 2) AssertionError: 0 != 2 ---------------------------------------------------------------------- Ran 5 tests in 0.005s FAILED (failures=1) Traceback (most recent call last): File "./Lib/test/test_shutil.py", line 106, in ? test_main() File "./Lib/test/test_shutil.py", line 103, in test_main test_support.run_unittest(TestShutil) File "/usr/local/src/Python-2.4/Lib/test/test_support.py", line 290, in run_u run_suite(suite, testclass) File "/usr/local/src/Python-2.4/Lib/test/test_support.py", line 275, in run_s raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "./Lib/test/test_shutil.py", line 34, in test_on_error self.assertEqual(self.errorState, 2) AssertionError: 0 != 2 From michael.walter at gmail.com Sat Dec 4 06:19:20 2004 From: michael.walter at gmail.com (Michael Walter) Date: Sat Dec 4 06:19:23 2004 Subject: [Python-Dev] PEP: __source__ proposal In-Reply-To: <877e9a17041203211849479ab5@mail.gmail.com> References: <877e9a17041203211849479ab5@mail.gmail.com> Message-ID: <877e9a170412032119589f324e@mail.gmail.com> Hi! I think you were the omitting the more interesting closure examples (namely with free variables inside the closure's source): def foo(): pass def bar(x): def fiep(): return x() return fiep what's bar(foo).__source__? Generally, I'm opposed to the feature -- I don't think it provides a real advantage giving it's limitations (doesn't work for import'ed modules, classes + methods != module, etc.). Cheers, Michael On Fri, 3 Dec 2004 11:54:25 +0200 (EET), Stelios Xanthakis wrote: > > Hi all. > > Now that 2.4 is out and everything maybe it's > about time to start discussing the "use the > __source__ Luke" feature which IMO will really > boost python into a new domain of exciting > possibilities. > > I've prepared a pre-PEP which is not very good > but it is a base. > > In short, the feature is good and it enables > editing of python code at runtime instead of > the runfile-exit-edit-run-exit-edit-run cycle. > > We have the following possibilities as to whether > __source__ data is marshalled and the feature is > always enabled. > > [1] Command line switch and not marshalled > [2] Always on and not marshalled > [3] Always on and marshalled > > There is also [4] which doesn't make much sense. > > If I was BDFL I'd go for [1] so whoever wants it > can enable it and whoever doesn't can't complain, > and they'll all leave me alone. > Phillip J. Eby expressed some concerns that the > modules that depend on __source__ will eventually > take over and it will become a standard. > > Anyway, the PEP is attached. > You can mail me with votes on the feature and if you > want on your preferred option from 1,2,3. > If I get votes I'll post the results later. > > If this is accepted I'll try to come up with a good > patch vs 2.4 > > Thanks, > > St. > > -------------------ATTACHED PYTHON ENHANCEMENT PROPOSAL--- > PEP: XXX > Title: The __source__ attribute > Version: $Revision: 1.10 $ > Last-Modified: $Date: 2003/09/22 04:51:49 $ > Author: Stelios Xanthakis > Status: Draft > Type: Standards Track > Content-Type: text/plain > Created: 19-Nov-2004 > Python-Version: 2.4.1 > Post-History: > > Abstract > > This PEP suggests the implementation of __source__ attribute for > functions and classes. The attribute is a read-only string which > is generated by the parser and is a copy of the original source > code of the function/class (including comments, indentation and > whitespace). > > Motivation > > It is generally a tempting idea to use python as an interface to > a program. The developers can implement all the functionality > and instead of designing a user interface, provide a python > interpreter to their users. Take for example one of the existing > web browsers: they have everything that would be needed to write > a script which downloads pages automatically or premutates the > letters of web pages before they are displayed, but it is not > possible for the user to do these things because the interface > of these applications is static. > > A much more powerful approach would be an interface which is > dynamically constructed by the user to meet the user's needs. > The most common development cycle of python programs is: > write .py file - execute .py file - exit - enhance .py file - > execute .py file - etc. With the implementation of the __source__ > attribute though the development/modification of python code > can happen at run-time. Functions and classes can be defined, > modified or enhanced while the python shell is running and > all the changes can be saved by saving the __source__ attribute > of globals before termination. Moreover, in such a system > it is possible to modify the "code modification routines" and > eventually we have a self-modifying interface. Using a > program also means improving its usability. > > The current solution of using 'inspect' to get the source > code of functions is not adequate because it doesn't work > for code defined with "exec" and it doesn't have the source > of functions/classes defined in the interactive mode. Generally, > a "file" is something too abstract. What is more real is the > data received by the python parser and that is what is stored > in __source__. > > Specification > > The __source__ attribute is a read-only attribute of functions > and classes. Its type is string or None. In the case of None > it means that the source was not available. > > The indentation of the code block is the original identation > obeying nested definitions. For example: > > >>> class A: > ... def foo (self): > ... print """Santa-Clauss > ... is coming to town""" > >>> def spam (): > ... def closure (): > ... pass > ... return closure > >>> print A.foo.__source__ > def foo (self): > print """Santa-Clauss > is coming to town""" > >>> print spam().__source__ > def closure (): > pass > > The attribute is not marshaled and therefore not stored in > ".pyc" files. As a consequence, functions and classes of > imported modules have __source__==None. > > We propose that the generation of __source__ will be > controlled by a command line option. In the case this > feature is not activated by the command line option, the > attribute is absent. > > Rationale > > Generally, "import" refers to modules that either have a file in > a standard location or they are distributed in ".pyc" form only. > Therefore in the case of modules, getting the source with > "inspect" is adequate. Moreover, it does not make sense saving > __source__ in ".pyc" because the point would be to save > modifications in the original ".py" file (if available). > > On the issue of the command-line option controlling the generation > of __source__, please refer to the section about the overhead > of this feature. The rationale is that those applications that > do not wish to use this feature can avoid it (cgi scripts in > python benchmarked against another language). > > Overhead > > The python's parser is not exactly well-suited for such a feature. > Execution of python code goes through the stages of lexical > analysis, tokenization, generation of AST and execution of > bytecode. In order to implement __source__, the tokenizer has > to be modified to store the lines of the current translation > unit. Those lines are then attached the root node of the > AST. While the AST is compiled we have to keep a reference > of the current node in order to be able to find the next node > after the node for which we wish to generate __source__, get > the first and the last line of our block and then refer to > the root node to extract these lines and make a string. All > these actions add a minor overhead to some heavily optimized > parts of python. However, once compilation to bytecode is > done, this feature no longer affects the performance of the > execution of the bytecode. > > There is also the issue of the memory spent to store __source__. > In our opinion, this is worth the tradeoff for those who > are willing to take advantage of it. > > Implementation > > There is a sample implementation at [2] which consists of a > patch against python 2.3.4. The patch has to be improved > to avoid generating __source__ for the case we are importing > modules for the first time (not from .pyc). In the sample > implementation there is also included a sample shell that > takes advantage of __source__ and demonstrates some aspects > that motivated us towards patching python and submitting this > PEP. > > References > > [1] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton > http://www.python.org/peps/pep-0001.html > > [2] Sample implementation > http://students.ceid.upatras.gr/~sxanth/ISYSTEM/python-PIESS.tar.gz > > Copyright > > This document has been placed in the public domain. > > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > End: > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/michael.walter%40gmail.com > From hpk at trillke.net Sat Dec 4 09:59:38 2004 From: hpk at trillke.net (holger krekel) Date: Sat Dec 4 09:59:41 2004 Subject: [Python-Dev] PEP: __source__ proposal In-Reply-To: References: <20041203123616.GB4708@solar.trillke.net> Message-ID: <20041204085937.GC4708@solar.trillke.net> [Stelios Xanthakis Fri, Dec 03, 2004 at 11:59:30PM +0200] > On Fri, 3 Dec 2004, holger krekel wrote: > >We are about to test out this approach with the py lib > >(http://codespeak.net/py) and want to have it work for > >for Python 2.2, 2.3. and 2.4. > > Do you plan hacking python ? > It appears that tok_nextc() is the best place to > catch all the source passed to the interpreter. Well, as we want to have the library work on past python versions modifying CPython 2.5 does not make much sense. It's more about (like Martin pointed out) organizing dynamic code generation so that Python's introspect and traceback logic works as much as possible - with tiny runtime "monkey" patches if needed. Now Martin also correctly pointed out that you can store source code before/after you pass it to compile/parse. We are doing this already with an external dictionary. This has multithreading issues, though. So we think that hooking onto code's objects co_filename or a module's __file__ might be an interesting idea. cheers, holger From sxanth at ceid.upatras.gr Sat Dec 4 10:02:26 2004 From: sxanth at ceid.upatras.gr (Stelios Xanthakis) Date: Sat Dec 4 10:02:30 2004 Subject: [Python-Dev] PEP: __source__ proposal In-Reply-To: <41B0F629.4040100@v.loewis.de> References: <41B0F629.4040100@v.loewis.de> Message-ID: > I'm opposed to this idea. It creates overhead in > the size of .pyc files, No it doesn't. > for no additional value that couldn't be obtained otherwise. Martin: I know it is possible to do all this with existing python facilities. I did write such a dynamic code framework in python. Specifically I used a function 'deyndef(code)' which was exactly like 'def' but also stored the source string in a dictionary. The key point is that I think think should be the job of the parser and the functionality provided at the interactive prompt w/o the user having to write 'dyndef' or store the code of exec's or request from himself to use specific commands to create functions. It should be transparent built into python. > A file is precisely the level of granularity that is > burnt into the Python language. A module is *always* a file, executed > from top to bottom. It is not possible to recreate the source code > of a module if you have only the source code of all functions, and > all classes. That's exactly the rationale for NOT combining __source__ with import. It's in the PEP. It appears that there are the 'module people' who find this feature irrelevant. Indeed. If we are interested in distributing modules and increasing the number of people who use python programs,then __source__ is redundant. OTOH, programming python is easy and fun and I think the proposed feature will make it even more fun and it aims in increasing the number of people who program python for their every day tasks. It'd be interesting to hear if the developers of IDLE/ipython/etc could use this. Oh well. I guess I'm ahead of my time again:) St. From mwh at python.net Sat Dec 4 12:57:34 2004 From: mwh at python.net (Michael Hudson) Date: Sat Dec 4 12:57:35 2004 Subject: [Python-Dev] test_shutils.py fails for 2.4 install In-Reply-To: <41AFCE6D.10300@erols.com> (Edward C. Jones's message of "Thu, 02 Dec 2004 21:24:45 -0500") References: <41AFCE6D.10300@erols.com> Message-ID: <2m7jnys2c1.fsf@starship.python.net> "Edward C. Jones" writes: > I have a PC with an AMD cpu and Mandrake 10.1. While installing Python > 2.4 "make test" failed in "test_shutils.py". Here is the output of > "./python ./Lib/test/test_shutil.py": Are you running 'make test' as root? Don't do that (although possibly the test suite should guard against it). Also, did you search the bug tracker? Please do do that. Cheers, mwh -- CDATA is not an integration strategy. -- from Twisted.Quotes From martin at v.loewis.de Sat Dec 4 15:10:06 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Dec 4 15:10:03 2004 Subject: [Python-Dev] PEP: __source__ proposal In-Reply-To: References: <41B0F629.4040100@v.loewis.de> Message-ID: <41B1C53E.9050204@v.loewis.de> Stelios Xanthakis wrote: > The key point is that I think think should be the > job of the parser and the functionality provided > at the interactive prompt w/o the user having to > write 'dyndef' or store the code of exec's or request > from himself to use specific commands to create > functions. It should be transparent built into python. For the case of the interactive prompt, if preserving the source code is somehow desirable (which I doubt), it should be the job of the development environment to offer saving interactively-defined methods. Such IDE support is necessary even if __source__ was available, since users probably would not want to write open("demo.py").write(myfunc.__source__ + "\n" + myclass.__source) > OTOH, programming python is easy and fun and I think > the proposed feature will make it even more fun and it > aims in increasing the number of people who program > python for their every day tasks. It'd be interesting to > hear if the developers of IDLE/ipython/etc could use this. I think it is the other way 'round. If this is *only* for interactive mode, than you should *first* change the interactive environments. If you then find you absolutely need this feature to implement the IDEs correctly, I'd like to hear the (new) rationale for the change. Regards, Martin From ncoghlan at iinet.net.au Sat Dec 4 15:29:27 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Dec 4 15:29:34 2004 Subject: [Python-Dev] PEP: __source__ proposal In-Reply-To: References: <41B0F629.4040100@v.loewis.de> Message-ID: <41B1C9C7.1020604@iinet.net.au> [Resend, since a minor brain explosion caused me to send this to c.l.p instead of python-dev] Stelios Xanthakis wrote: > It appears that there are the 'module people' who > find this feature irrelevant. Indeed. If we are interested > in distributing modules and increasing the number of > people who use python programs,then __source__ is redundant. > OTOH, programming python is easy and fun and I think > the proposed feature will make it even more fun and it > aims in increasing the number of people who program > python for their every day tasks. It'd be interesting to > hear if the developers of IDLE/ipython/etc could use this. The feedback here (and the initial response on py-dev a while back) suggests to me that you should look at making this a feature of the interactive mode. Something that affects both Python's main interactive shell, plus the relevant class in the standard library (CommandInterpreter or whatever it is called). A late-night-train-of-thought example of what might be handy is below - keep in mind that I haven't looked at what enhanced Python shells like IPython can do, so it may be there are tools out there that do something like this already. It would be handy to have a standard library module that supported "on-the-fly" editing, though (this capability would then be available to anyone embedding Python as a scripting engine). Cheers, Nick. ============================== >>>import source >>>class bob: ... def mary(): ... pass ... def tim(): ... print 'Tim' ... >>>print bob.__source__ class bob: def mary(): pass def tim(): print 'Tim' >>>print bob.mary.__source__ def mary(): pass >>> source.edit(bob.mary) bob.mary(1)>def mary(text): # [1] bob.mary(2)> print "Mary:", text bob.mary(3)>\save >>> source.edit(bob.tim) bob.tim(1)>\help Commands: \help \cancel \save \deleteline bob.tim(2)>\cancel >>>print bob.__source__ "class bob: def mary(text): print "Mary:", text def tim(): print 'Tim' " >>> bob().mary("Hi!") Mary: Hi! The basic ideas of the above: "import source" triggers the storage of the __source__ attributes (e.g. via installation of appropriate hooks in the class and function definition process) The 'edit' function is then able to take advantage of the stored source code to present each line of the original source for modification (e.g. to fix a minor bug in one function of a class definition). When the 'edit' is complete, it can be saved or cancelled. 1. The feature mentioned in the last paragraph is hard to show in the expected output :) -- Nick Coghlan | ncoghlan@email.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net From tim.peters at gmail.com Sat Dec 4 17:23:26 2004 From: tim.peters at gmail.com (Tim Peters) Date: Sat Dec 4 17:23:30 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib whrandom.py, 1.21, NONE In-Reply-To: References: Message-ID: <1f7befae04120408235ffc9b8a@mail.gmail.com> [rhettinger@users.sourceforge.net] > Removed Files: > whrandom.py > Log Message: > Remove the deprecated whrandom module. Woo hoo! It's about friggin' time . From edcjones at erols.com Fri Dec 3 18:53:03 2004 From: edcjones at erols.com (Edward C. Jones) Date: Sat Dec 4 18:51:20 2004 Subject: [Python-Dev] RE: test_shutils.py fails for 2.4 install Message-ID: <41B0A7FF.9040305@erols.com> The relevant bug appears to be 1076467 at SourceForge. From raymond.hettinger at verizon.net Sun Dec 5 03:20:09 2004 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sun Dec 5 03:22:54 2004 Subject: [Python-Dev] Py2.5: Deprecated Cookie classes and interface Message-ID: <000901c4da70$f2201e40$ab2fc797@oemcomputer> Any objections to removing Cookie.Cookie, Cookie.SmartCookie, and Cookie.SerialCookie? Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20041204/e3c8679b/attachment.html From raymond.hettinger at verizon.net Sun Dec 5 03:30:03 2004 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sun Dec 5 03:32:47 2004 Subject: [Python-Dev] Deprecated xmllib module Message-ID: <001801c4da72$543801a0$ab2fc797@oemcomputer> Hmph. The xmllib module has been deprecated since Py2.0 but is not listed in PEP 4. Question: Do we have to keep it for another two years because of that omission? It seems somewhat silly to keep an obsolete, supplanted module that doesn?t full support XML 1.0. Raymond From fredrik at pythonware.com Sun Dec 5 08:40:47 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sun Dec 5 08:38:34 2004 Subject: [Python-Dev] Re: Deprecated xmllib module References: <001801c4da72$543801a0$ab2fc797@oemcomputer> Message-ID: Raymond Hettinger wrote: > Hmph. The xmllib module has been deprecated since Py2.0 but is not > listed in PEP 4. > > Question: Do we have to keep it for another two years because of that > omission? > > It seems somewhat silly to keep an obsolete, supplanted module that > doesn't full support XML 1.0. it's better to be somewhat silly than to be arrogant and stupid. if python-dev cares about existing users, xmllib should stay in there until 3.0. From bac at OCF.Berkeley.EDU Sun Dec 5 08:52:14 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sun Dec 5 08:52:21 2004 Subject: [Python-Dev] Any reason why CPPFLAGS not used in compiling? Message-ID: <41B2BE2E.3000907@ocf.berkeley.edu> I noticed that Makefile.pre.in uses the value from the environment variable LDFLAGS but not CPPFLAGS. Any reason for this? ``./configure -h`` lists both (and some other environment variables which are not really used either) so it seems a little misleading to have those listed but to not utilize them. The reason I ask is I plan on having setup.py add the directories specified in both of these environment variables for compiling the extension modules. It would be nice to be able to use the same values as used by the Makefile to build the core, but I can if I must just get the values directly from the environment variables themselves. This should allow for the removal of the direct support for Fink and DarwinPorts. It should also remove any hand-editing needed to get setup.py to pick up any non-standard library and header locations. -Brett From martin at v.loewis.de Sun Dec 5 09:36:00 2004 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Dec 5 09:35:59 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <001801c4da72$543801a0$ab2fc797@oemcomputer> References: <001801c4da72$543801a0$ab2fc797@oemcomputer> Message-ID: <41B2C870.5090609@v.loewis.de> Raymond Hettinger wrote: > Hmph. The xmllib module has been deprecated since Py2.0 but is not > listed in PEP 4. > > Question: Do we have to keep it for another two years because of that > omission? > > It seems somewhat silly to keep an obsolete, supplanted module that > doesn?t full support XML 1.0. I mostly agree with Fredrik. What good does removal of xmllib do? It's not that it is causing any maintenance burden, so we could just leave it in. Whether this means that we keep xmllib until P3k, I don't know. As for PEP 4: I don't know whether it needs to be listed there. It appears that the PEP is largely unmaintained (I, personally, do not really maintain it). So one option would be to just stop using PEP 4 for recording deprecations, since we now have the warnings module. If we want to keep PEP 4, we need to follow the procedures it requires (or modify them if we don't like them). Regards, Martin From martin at v.loewis.de Sun Dec 5 09:40:49 2004 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun Dec 5 09:40:45 2004 Subject: [Python-Dev] Any reason why CPPFLAGS not used in compiling? In-Reply-To: <41B2BE2E.3000907@ocf.berkeley.edu> References: <41B2BE2E.3000907@ocf.berkeley.edu> Message-ID: <41B2C991.4090502@v.loewis.de> Brett C. wrote: > I noticed that Makefile.pre.in uses the value from the environment > variable LDFLAGS but not CPPFLAGS. Any reason for this? How did you notice that? For LDFLAGS, Makefile.pre.in has LDFLAGS= @LDFLAGS@ This does *not* mean that the value from the environment is used. Instead, it means that configure computes the value of LDFLAGS when it generates Makefile.in. For CPPFLAGS, configure has nothing to compute, so Makefile.pre.in just has the static value for CPPFLAGS. Regards, Martin From python at rcn.com Sun Dec 5 10:27:15 2004 From: python at rcn.com (Raymond Hettinger) Date: Sun Dec 5 10:30:03 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <41B2C870.5090609@v.loewis.de> Message-ID: <000201c4daac$9bd871e0$3b01a044@oemcomputer> > As for PEP 4: I don't know whether it needs to be listed there. It > appears that the PEP is largely unmaintained (I, personally, do not > really maintain it). So one option would be to just stop using PEP 4 > for recording deprecations, since we now have the warnings module. +1 Raymond From mwh at python.net Sun Dec 5 11:40:08 2004 From: mwh at python.net (Michael Hudson) Date: Sun Dec 5 11:40:10 2004 Subject: [Python-Dev] Any reason why CPPFLAGS not used in compiling? In-Reply-To: <41B2BE2E.3000907@ocf.berkeley.edu> (Brett C.'s message of "Sat, 04 Dec 2004 23:52:14 -0800") References: <41B2BE2E.3000907@ocf.berkeley.edu> Message-ID: <2my8gdqb93.fsf@starship.python.net> "Brett C." writes: > I noticed that Makefile.pre.in uses the value from the environment > variable LDFLAGS but not CPPFLAGS. Any reason for this? > ./configure -h`` lists both (and some other environment variables > which are not really used either) so it seems a little misleading to > have those listed but to not utilize them. The whole story of how the compiling/linker flags get set up is a bit of a mess, AFAIK. I don't have the energy or the desire to try to make what we want to happen precise (the hard bit) and then make that happen (probably rather easier). > This should allow for the removal of the direct support for Fink and > DarwinPorts. +lots. I don't trust fink, but need it for latex... Cheers, mwh -- After a heavy night I travelled on, my face toward home - the comma being by no means guaranteed. -- paraphrased from cam.misc From python at rcn.com Sun Dec 5 11:40:16 2004 From: python at rcn.com (Raymond Hettinger) Date: Sun Dec 5 11:43:06 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <41B2C870.5090609@v.loewis.de> Message-ID: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> > > It seems somewhat silly to keep an obsolete, supplanted module that > > doesn?t full support XML 1.0. > > I mostly agree with Fredrik. What good does removal of xmllib do? A few thoughts: * Deprecated modules just get moved to the lib-old directory. If someone has ancient code relying on the module, it is a somewhat trivial maintenance step to add lib-old to their PYTHONPATH. IOW, I fail to see the harm. * For this particular module, xmllib, about six years will have elapsed between its original deprecation in Py2.0 and us taking it out in a Py2.5 release. * The number one current python complaint is the state of the standard library: http://www.internetnews.com/dev-news/article.php/3441931 . Some of this may just be the fruits of AMK's highly publicized journal entry; however, I think the concerns about library quality will persist. * The average quality of the library improves as we take out junk (the tzparse module for example) and put in high quality modules like logging, csv, decimal, etc. * After working through the tutorial, another huge task for aspiring intermediate Python programmer is to learn about the standard library. That task is made much difficult by the presence of obsolete, incomplete, and/or supplanted modules. For example, the high quality, actively maintained email package supplants several older, incomplete, or non-actively maintained modules. The library would be cleaner, more presentable, and easier to use if the old ones were to fall by the wayside. * Ideally, the library will develop in a way that doesn't have a cluttered concept space. This will help get it back to the point where you can hold it in your head all at once and still be able to think about the problem domain. Keeping cruft impedes that goal. my two cents, Raymond From martin at v.loewis.de Sun Dec 5 12:12:12 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Dec 5 12:12:08 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> References: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> Message-ID: <41B2ED0C.20509@v.loewis.de> Raymond Hettinger wrote: > * Deprecated modules just get moved to the lib-old directory. If > someone has ancient code relying on the module, it is a somewhat trivial > maintenance step to add lib-old to their PYTHONPATH. IOW, I fail to see > the harm. I have never considered this as an official policy. For example, when deprecated C modules are removed, they are never moved to lib-old. > * For this particular module, xmllib, about six years will have elapsed > between its original deprecation in Py2.0 and us taking it out in a > Py2.5 release. Correct. For regex, much more time has passed (it is deprecated since 1.5 or something). > * The number one current python complaint is the state of the standard > library: http://www.internetnews.com/dev-news/article.php/3441931 . > Some of this may just be the fruits of AMK's highly publicized journal > entry; however, I think the concerns about library quality will persist. I agree. Removing old modules might change this, indeed: the complaint about unstability and incompatibility might then become the number one complaint :-) > * The average quality of the library improves as we take out junk (the > tzparse module for example) and put in high quality modules like > logging, csv, decimal, etc. I am not convinced that all these more recent modules are really higher quality than the modules that have been added ten years ago. Some are (especially those which has seen extensive testing before being integrated), some are not (especially those added in an ad-hoc manner). Regards, Martin From anthony at interlink.com.au Sun Dec 5 16:42:56 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Sun Dec 5 16:43:28 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> References: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> Message-ID: <200412060243.01129.anthony@interlink.com.au> On Sunday 05 December 2004 21:40, Raymond Hettinger wrote: > * The number one current python complaint is the state of the standard > library: http://www.internetnews.com/dev-news/article.php/3441931 . > Some of this may just be the fruits of AMK's highly publicized journal > entry; however, I think the concerns about library quality will persist. > > * The average quality of the library improves as we take out junk (the > tzparse module for example) and put in high quality modules like > logging, csv, decimal, etc. We can't win - if we remove it, we'll get massive numbers of complaints bitching about us breaking backwards compatibility. I think the solution is to make sure the library docs say, very clearly, in big words at the top of the relevant page "DON'T DO NEW CODE WITH THIS MODULE". Perhaps Fred can make latex2html output a tag around it . The library docs for, e.g. xmllib already say deprecated at the top - maybe it should be larger? Should the global module index should grow a "(deprecated)" line next door to the module name of the modules that are known to be old and kinda awful (such as xmllib)? Or make a seperate category - normal modules, and modules-that-are-only-kept-for-backwards compat? I suspect that rfc822 will end up in that latter category, rather than ever being removed (at least, until 3.0). I went through just my own code, and found an amazing amount of use of that module. Python has a reputation for stability and not forcing people to rewrite their code - I think that this is a good thing. IOW, I think breaking code is far worse than some folks whining that some of the stdlib is old and crufty. Alternately, we could lock the folks complaining about the stdlib's state in a room with the folks who complain that every new thing has "wrecked the language that they knew and loved" and let them fight it out. > * After working through the tutorial, another huge task for aspiring > intermediate Python programmer is to learn about the standard library. > That task is made much difficult by the presence of obsolete, > incomplete, and/or supplanted modules. Surely this one is something that can be fixed in the documentation by marking obsolete modules as such, or making the marking much clearer? Anthony From barry at python.org Sun Dec 5 17:35:13 2004 From: barry at python.org (Barry Warsaw) Date: Sun Dec 5 17:35:17 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <41B2C870.5090609@v.loewis.de> References: <001801c4da72$543801a0$ab2fc797@oemcomputer> <41B2C870.5090609@v.loewis.de> Message-ID: <1102264513.29521.63.camel@presto.wooz.org> On Sun, 2004-12-05 at 03:36, "Martin v. L?wis" wrote: > I mostly agree with Fredrik. What good does removal of xmllib do? > It's not that it is causing any maintenance burden, so we could just > leave it in. Whether this means that we keep xmllib until P3k, I > don't know. > > As for PEP 4: I don't know whether it needs to be listed there. It > appears that the PEP is largely unmaintained (I, personally, do not > really maintain it). So one option would be to just stop using PEP 4 > for recording deprecations, since we now have the warnings module. > If we want to keep PEP 4, we need to follow the procedures it > requires (or modify them if we don't like them). I agree. We don't need to use PEP 4 for listing module deprecation schedules. It would be better if we included that information in the DeprecationWarning. Probably nobody reads PEP 4 when they get a DeprecationWarning, but everyone reads the message that's printed, so if it said something like "This module is deprecated and slated for removal in Python 2.6", I think we'd be providing better information to our users. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041205/094182eb/attachment.pgp From barry at python.org Sun Dec 5 17:50:22 2004 From: barry at python.org (Barry Warsaw) Date: Sun Dec 5 17:50:25 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> References: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> Message-ID: <1102265422.29524.74.camel@presto.wooz.org> On Sun, 2004-12-05 at 05:40, Raymond Hettinger wrote: > * The number one current python complaint is the state of the standard > library: http://www.internetnews.com/dev-news/article.php/3441931 . > Some of this may just be the fruits of AMK's highly publicized journal > entry; however, I think the concerns about library quality will persist. I don't know if I'd use one quote from this article to infer that standard library issues are the #1 complaint. But besides that, there's no question IMO that the standard library should be the place where future Python development efforts are focused (not exclusively, but predominately). > * The average quality of the library improves as we take out junk (the > tzparse module for example) and put in high quality modules like > logging, csv, decimal, etc. > > * After working through the tutorial, another huge task for aspiring > intermediate Python programmer is to learn about the standard library. > That task is made much difficult by the presence of obsolete, > incomplete, and/or supplanted modules. You don't have to remove modules to make this happen. Few developers find modules by doing an 'ls' on Lib -- they look to the library reference docs, books, and existing code. So for example, the re-org I did on the string module docs for 2.4 should help, because now you can't look up a string function such as atoi() without seeing that it's deprecated. A developer writing new code hopefully wouldn't even waste time learning about those functions, and instead hop right to the preferred string methods. Yet, there's no push in the community to actually remove those deprecated functions from string.py. > For example, the high quality, actively maintained email package > supplants several older, incomplete, or non-actively maintained modules. > The library would be cleaner, more presentable, and easier to use if the > old ones were to fall by the wayside. Yes, but there's still /a lot/ of work to do to update the existing standard library to use the email package. We debated this during 2.4 and decided we can't even add deprecation warnings to things like rfc822.py until we do. > * Ideally, the library will develop in a way that doesn't have a > cluttered concept space. This will help get it back to the point where > you can hold it in your head all at once and still be able to think > about the problem domain. Keeping cruft impedes that goal. Agreed, but I think it's mostly going to be an organic process over time. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041205/71ac1d23/attachment.pgp From carribeiro at gmail.com Sun Dec 5 18:42:45 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Sun Dec 5 18:42:48 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> References: <41B2C870.5090609@v.loewis.de> <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> Message-ID: <864d370904120509421ec3dd50@mail.gmail.com> On Sun, 5 Dec 2004 05:40:16 -0500, Raymond Hettinger wrote: > * Deprecated modules just get moved to the lib-old directory. If > someone has ancient code relying on the module, it is a somewhat trivial > maintenance step to add lib-old to their PYTHONPATH. IOW, I fail to see > the harm. In principle, that's a good idea, but I don't know if it's really practical. I fear that the distribution would end up simply adding lib-old to the PYTHONPATH anyway :-) But some variations may be worth considering: 1) Deprecated modules would first generate a warning (as it is today). On future versions, instead of being removed, the level of the warning would be raised, causing a fatal exception (an ImportError?), unless explicitly configured otherwise (using a config file or some similar mechanism). I believe that the current warnings module already provides a good basis for this implementation. (IOW - the module would still be there, and could be activated, but it's "off" by default. Turning it on again has to be easy, though. And the message on "ImportError" have to meaningful for non-technical users, allowing for easier support in these situations) 2) Split the documentation: the "old-and-deprecated" modules would be listed separated from the active & recommended ones, as a different section, or even as a separate book. That would be a clear sign to new users to keep in mind while they read the documentation, perhaps more effective than writing the deprecation warning on the top of the page. My half-a-cent. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From martin at v.loewis.de Sun Dec 5 18:54:42 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Dec 5 18:54:39 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <200412060243.01129.anthony@interlink.com.au> References: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> <200412060243.01129.anthony@interlink.com.au> Message-ID: <41B34B62.7040201@v.loewis.de> Anthony Baxter wrote: > The library docs for, e.g. xmllib already say deprecated at the top - maybe > it should be larger? I think it would be better to *remove* the xmllib documentation. Existing code which needs the module will continue to work even without documentation, and new code is unlikely to be written for a module that has no documentation, and where importing the module gives a DeprecationWarning. > Alternately, we could lock the folks complaining about the stdlib's state > in a room with the folks who complain that every new thing has "wrecked > the language that they knew and loved" and let them fight it out. That sounds like a fair and democratic way of solving the issue. Regards, Martin From allison at sumeru.stanford.EDU Sun Dec 5 19:30:28 2004 From: allison at sumeru.stanford.EDU (Dennis Allison) Date: Sun Dec 5 19:30:35 2004 Subject: [Python-Dev] Python 2.4 and Zope 2.7.X In-Reply-To: <1102265422.29524.74.camel@presto.wooz.org> Message-ID: A report on the zope@zope.org list suggests that Python 2.4 is not fully compatible with Zope 2.7.3. Has any tested against Zope? From lists at andreas-jung.com Sun Dec 5 21:06:26 2004 From: lists at andreas-jung.com (Andreas Jung) Date: Sun Dec 5 21:06:33 2004 Subject: [Python-Dev] Python 2.4 and Zope 2.7.X In-Reply-To: References: Message-ID: <17891C1A2E0C2C4C4580A33B@[192.168.0.102]> --On Sonntag, 5. Dezember 2004 10:30 Uhr -0800 Dennis Allison wrote: > > A report on the zope@zope.org list suggests that Python 2.4 is not fully > compatible with Zope 2.7.3. Has any tested against Zope? > To which report do you refer? The only thing I mentioned is that there has not been any audit for Python 2.4 + Zope 2.7 as with Python 2.3 + Zope 2.6/2.7. -aj From rwgk at cci.lbl.gov Mon Dec 6 00:37:12 2004 From: rwgk at cci.lbl.gov (Ralf W. Grosse-Kunstleve) Date: Mon Dec 6 00:37:19 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() Message-ID: <200412052337.iB5NbC5Y318887@boa.lbl.gov> Brett C. wrote: > Ralf W. Grosse-Kunstleve wrote: > > Brett C. wrote: > > > > >I have been mulling over this proposal and I think I am finally > > >settling on +0 as long as it can be shown that it does not hurt > > >performance at all (yes, it shouldn't but this is the eval loop we are > > >talking about and that thing is a touchy beast). > > > > > > Is there a standard benchmark I could use to demonstrate the > > impact of the two new additions on the runtime? > > Thanks! > > =) Parrotbench and PyBench would be enough for me to sign off on any > performance hit. There is also pystone. I have done my homework now, based on parrotbench. Timings are below. The most important conclusion from my viewpoint is that my original goal can be achieved with a small patch and with a runtime penalty that is in the noise. However, mysterious runtime behavior of Objects/longobject.c lead me to change my original patch, and to the unanswered question "could small rearrangements in longobject.c potentially boost performance by 45% on some platforms?" Please don't be put off by the size of this message. My patch is essentially just one line in ceval.c (_Py_TickerAccumulation++)! To restate my original goal: I am looking for a simple way to answer the question: How much of a speedup can I expect if I reimplement a piece of Python code in C or C++? Note that I am not asking how much time is spent in Python including all the extensions that come with it, but how much time is spent in the Python bytecode interpreter loop. To clarify, look at the timings for two tests in parrotbench: b1 5.8287 time 30049474 ticks 0.1940 us/tick b2 1.5944 time 482584 ticks 3.3039 us/tick Each of the tests was run with range(10) in b.py to increase accuracy. The first column shows time.time() differences, the second the number of bytecodes processed in ceval.c, and the last column shows micro-seconds per tick. Why is the us/tick value for b2 17 times larger than that for b1? The answer is that b2 makes heavy use of long integers, and that a lot of time is spent in Objects/longobject.c doing long-integer arithmetic. Compared to b1, relatively little time is spent interpreting bytecodes. Back to the original question: how much sense does it make to reimplement b1 or b2 in C? Simply looking at the us/tick values is telling me that I can expect much more of a performance boost by reimplementing b1 rather than b2. This is because b2 spends a lot of time in C already. A similar situation arises if extensions like Numeric are used. Computationally intensive operations like matrix multiplication and matrix inversion are already done at C speed. The us/tick let us quickly estimate how disappointed we would be after moving Python code to C or C++. We get this estimate without having to do detailed source code analysis, and before we have wasted our time doing the recoding (we have been through this a couple of times ...). The timings below show that the patched Python which counts the number of interpreted bytecodes is in the worst case 0.6% slower than the original Python, but some tests even run faster at face value: $gcc296_compiled/original/python -O b_timings.py | grep reporter b0 16.8831 time b1 5.9553 time b2 1.5914 time b3 10.8149 time b4 5.2528 time b5 11.6437 time b6 11.0248 time b7 27.6960 time all 90.9276 time $gcc296_compiled/work/python -O b_timings.py | grep reporter b0 16.9193 time 50979585 ticks 0.3319 us/tick b1 5.8287 time 30049474 ticks 0.1940 us/tick b2 1.5944 time 482584 ticks 3.3039 us/tick b3 10.7931 time 43577404 ticks 0.2477 us/tick b4 5.2865 time 18463044 ticks 0.2863 us/tick b5 11.6086 time 17065750 ticks 0.6802 us/tick b6 10.0992 time 60000464 ticks 0.1683 us/tick b7 27.6830 time 1928644 ticks 14.3536 us/tick all 89.8760 time 222548399 ticks 0.4038 us/tick The b7 test is the same as b2 but with xrange(10000) instead of xrange(1000), and with all print statements removed. Ratios (rounded to 3 decimals): 16.9193/16.8831=1.002 5.8287/5.9553 =0.979 1.5944/1.5914 =1.002 10.7931/10.8149=0.998 5.2865/5.2528 =1.006 11.6086/11.6437=0.997 10.0992/11.0248=0.916 27.6830/27.6960=1.000 Therefore I'd argue that the runtime penalty for the one additional long long increment in ceval.c (_Py_TickerAccumulation++) is in the noise. The timings were collected on a 2.8GHz Dual Xeon, Redhat WS 3. Python was compiled under Red Hat 7.3 with gcc 2.96. See below why. My latest patch can be found at the end of this posting. It can be applied to the Python-2.4 distribution like this: tar zxf Python-2.4.tgz patch --strip=1 -d Python-2.4 < patch_file ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ Now to the mysterious runtime behavior of Objects/longobject.c In the original Python 2.4 sources _Py_Ticker is decremented in two places: In the interpreter loop in Python/ceval.c: if (--_Py_Ticker < 0) { In Objects/longobject.c: #define SIGCHECK(PyTryBlock) \ if (--_Py_Ticker < 0) { \ _Py_Ticker = _Py_CheckInterval; \ if (PyErr_CheckSignals()) { PyTryBlock; } \ } This macro is used in four places. My initial patch was to insert updates of a new, global _Py_TickerAccumulation variable in these two places: Python/ceval.c: a try: finally: block uninterruptable. */ goto fast_next_opcode; } + _Py_TickerAccumulation += _Py_CheckInterval - _Py_Ticker; _Py_Ticker = _Py_CheckInterval; tstate->tick_counter++; Objects/longobject.c: #define SIGCHECK(PyTryBlock) \ if (--_Py_Ticker < 0) { \ + _Py_TickerAccumulation += _Py_CheckInterval - _Py_Ticker; \ _Py_Ticker = _Py_CheckInterval; \ if (PyErr_CheckSignals()) { PyTryBlock; } \ } This lead to the timings below. All timings were collected on the same machine, but the original and the patched Pythons were each compiled with two different compilers, gcc 2.96 and gcc 3.2.3. The important timings to look at are those for b2 and b7 again. $gcc296_compiled/original/python -O b_timings.py | grep reporter b0 16.8831 time b1 5.9553 time b2 1.5914 time b3 10.8149 time b4 5.2528 time b5 11.6437 time b6 11.0248 time b7 27.6960 time all 90.9276 time $gcc323_compiled/original/python -O b_timings.py | grep reporter b0 18.6390 time b1 6.4681 time b2 2.2588 time b3 11.0215 time b4 5.6490 time b5 12.3022 time b6 10.3815 time b7 40.2735 time all 107.0636 time This shows that the gcc 2.96 optimizer is generally a little bit better than the gcc 3.2.3 optimizer, but not by very much. Except for the b2 and b7 tests. E.g. 40.2735/27.6960 = 1.454! Now with the patches in both ceval.c and longobject.c: $gcc296_compiled/ticker/python -O b_timings.py | grep reporter b0 17.1190 time 60792625 ticks 0.2816 us/tick b1 6.1171 time 30049474 ticks 0.2036 us/tick b2 1.8764 time 705754 ticks 2.6587 us/tick b3 10.7070 time 43577404 ticks 0.2457 us/tick b4 5.2677 time 22944694 ticks 0.2296 us/tick b5 11.7448 time 17433190 ticks 0.6737 us/tick b6 10.9603 time 60000504 ticks 0.1827 us/tick b7 33.2320 time 2837124 ticks 11.7133 us/tick all 97.0893 time 238342235 ticks 0.4074 us/tick $gcc323_compiled/ticker/python -O b_timings.py | grep reporter b0 18.3115 time 60792625 ticks 0.3012 us/tick b1 6.1193 time 30049474 ticks 0.2036 us/tick b2 2.2522 time 705754 ticks 3.1912 us/tick b3 11.2202 time 43577404 ticks 0.2575 us/tick b4 5.4596 time 22944694 ticks 0.2379 us/tick b5 11.8197 time 17433190 ticks 0.6780 us/tick b6 10.7407 time 60000504 ticks 0.1790 us/tick b7 40.2397 time 2837124 ticks 14.1833 us/tick all 106.2283 time 238342235 ticks 0.4457 us/tick The gcc 3.2.3 timings for b7 are hardly affected by my patch in longobject.c, but the gcc 2.96 timing shoots up from 27.6960 to 33.2320. Still not as bad as the gcc 3.2.3 timing, but why? Interestingly, if I declare _Py_TickerAccumulation as long instead of PY_LONG_LONG, the gcc 2.96 timing is also hardly affected by my patch in longobject.c. Even more interestingly, if I change longobject.c like this... #define SIGCHECK(PyTryBlock) \ { \ if (PyErr_CheckSignals()) { PyTryBlock; } \ } the runtime is also hardly affected (times not shown in this posting) even though PyErr_CheckSignals() is executed 100 times more often. Curious to learn how other compilers deal with longobject.c I collected timings with this compiler: Compaq C V6.3-028 on Compaq Tru64 UNIX V5.1 (Rev. 732) $compaq_compiled/original/python -O b_timings.py | grep reporter b1 19.4492 time b2 4.9395 time b3 23.6787 time b5 28.2568 time b6 21.6230 time b7 86.5283 time all 192.0498 time $compaq_compiled/ticker/python -O b_timings.py | grep reporter b1 19.4775 time 30049474 ticks 0.6482 us/tick b2 4.9971 time 704784 ticks 7.0902 us/tick b3 24.2559 time 43577404 ticks 0.5566 us/tick b5 26.3320 time 17420746 ticks 1.5115 us/tick b6 21.7227 time 60000464 ticks 0.3620 us/tick b7 86.9395 time 2836154 ticks 30.6540 us/tick all 190.8936 time 165555937 ticks 1.1530 us/tick The b0 and b4 timings are missing because of exceptions that I didn't investigate. As with the gcc 3.2.3 compilation, my longobject.c patch hardly makes a difference. What does all this mean? Does the gcc 2.96 surprise tell us that small rearrangements in longobject.c could potentially boost performance by 45% on some platforms? Since I am not interested in optimizing longobject.c this is where I stopped. Eventually I decided that the problematic patch in longobject.c is not helping me with my original goal as stated near the beginning of this posting. I am only interested in counting the iterations of the interpreter loop. However, the _Py_Ticker decrements in longobject.c are not inside the interpreter loop, but in C loops! This means _Py_Ticker is useless for my purposes. Therefore I decoupled _Py_Ticker and _Py_TickerAccumulation. BTW: I think strictly speaking this documentation is incorrect because of the _Py_Ticker manipulations in longobject.c: sys.setcheckinterval.__doc__: Tell the Python interpreter to check for asynchronous events every n instructions. I feel very dizzy now after dealing with all the timings and the nasty details. A lot of fallout caused by the simple idea to add one innocent line to ceval.c. I hope it makes sense to somebody! Cheers, Ralf ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ diff -r -u original/Include/ceval.h work/Include/ceval.h --- original/Include/ceval.h 2004-10-10 19:40:35.000000000 -0700 +++ work/Include/ceval.h 2004-11-30 23:27:11.000000000 -0800 @@ -68,6 +68,11 @@ /* this used to be handled on a per-thread basis - now just two globals */ PyAPI_DATA(volatile int) _Py_Ticker; +#ifndef HAVE_LONG_LONG +PyAPI_DATA(volatile long) _Py_TickerAccumulation; +#else +PyAPI_DATA(volatile PY_LONG_LONG) _Py_TickerAccumulation; +#endif PyAPI_DATA(int) _Py_CheckInterval; /* Interface for threads. diff -r -u original/PC/os2emx/python24.def work/PC/os2emx/python24.def --- original/PC/os2emx/python24.def 2004-10-10 19:40:50.000000000 -0700 +++ work/PC/os2emx/python24.def 2004-11-30 23:27:11.000000000 -0800 @@ -743,6 +743,7 @@ "_Py_CheckRecursionLimit" "_Py_CheckInterval" "_Py_Ticker" + "_Py_TickerAccumulation" ; From python24_s.lib(compile) "PyCode_New" diff -r -u original/Python/ceval.c work/Python/ceval.c --- original/Python/ceval.c 2004-11-23 10:06:08.000000000 -0800 +++ work/Python/ceval.c 2004-12-03 19:39:36.000000000 -0800 @@ -373,6 +373,7 @@ pendinglast = j; _Py_Ticker = 0; + _Py_TickerAccumulation = 0; things_to_do = 1; /* Signal main loop */ busy = 0; /* XXX End critical section */ @@ -476,6 +477,11 @@ per thread, now just a pair o' globals */ int _Py_CheckInterval = 100; volatile int _Py_Ticker = 100; +#ifndef HAVE_LONG_LONG +volatile long _Py_TickerAccumulation = 0; +#else +volatile PY_LONG_LONG _Py_TickerAccumulation = 0; +#endif PyObject * PyEval_EvalCode(PyCodeObject *co, PyObject *globals, PyObject *locals) @@ -770,6 +776,7 @@ async I/O handler); see Py_AddPendingCall() and Py_MakePendingCalls() above. */ + _Py_TickerAccumulation++; if (--_Py_Ticker < 0) { if (*next_instr == SETUP_FINALLY) { /* Make the last opcode before diff -r -u original/Python/sysmodule.c work/Python/sysmodule.c --- original/Python/sysmodule.c 2004-08-12 11:19:17.000000000 -0700 +++ work/Python/sysmodule.c 2004-12-03 19:36:52.000000000 -0800 @@ -442,6 +442,20 @@ "getcheckinterval() -> current check interval; see setcheckinterval()." ); +static PyObject * +sys_gettickeraccumulation(PyObject *self, PyObject *args) +{ +#ifndef HAVE_LONG_LONG + return PyInt_FromLong(_Py_TickerAccumulation); +#else + return PyLong_FromLongLong(_Py_TickerAccumulation); +#endif +} + +PyDoc_STRVAR(gettickeraccumulation_doc, +"gettickeraccumulation() -> current count of bytecodes processed by the interpreter." +); + #ifdef WITH_TSC static PyObject * sys_settscdump(PyObject *self, PyObject *args) @@ -763,6 +777,8 @@ setcheckinterval_doc}, {"getcheckinterval", sys_getcheckinterval, METH_NOARGS, getcheckinterval_doc}, + {"gettickeraccumulation", sys_gettickeraccumulation, METH_NOARGS, + gettickeraccumulation_doc}, #ifdef HAVE_DLOPEN {"setdlopenflags", sys_setdlopenflags, METH_VARARGS, setdlopenflags_doc}, From bac at OCF.Berkeley.EDU Mon Dec 6 00:49:56 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon Dec 6 00:51:03 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <41B34B62.7040201@v.loewis.de> References: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> <200412060243.01129.anthony@interlink.com.au> <41B34B62.7040201@v.loewis.de> Message-ID: <41B39EA4.9030105@ocf.berkeley.edu> Martin v. L?wis wrote: > Anthony Baxter wrote: > >> The library docs for, e.g. xmllib already say deprecated at the top - >> maybe >> it should be larger? > > > I think it would be better to *remove* the xmllib documentation. > Existing code which needs the module will continue to work even > without documentation, and new code is unlikely to be written for > a module that has no documentation, and where importing the module > gives a DeprecationWarning. > I like this solution. And if anyone cares about having good docs still for a deprecated module they can just read the docstrings. We don't get new people using it but others don't have to change their code. And as for the mention of dropping PEP 4, I agree with the running consensus that it isn't needed thanks to the warning module. Do we need to have a more formal vote to drop PEP 4, or should one the PEP maintainers just delete it? -Brett From bac at OCF.Berkeley.EDU Mon Dec 6 00:55:54 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon Dec 6 00:57:05 2004 Subject: [Python-Dev] Any reason why CPPFLAGS not used in compiling? In-Reply-To: <41B2C991.4090502@v.loewis.de> References: <41B2BE2E.3000907@ocf.berkeley.edu> <41B2C991.4090502@v.loewis.de> Message-ID: <41B3A00A.30500@ocf.berkeley.edu> Martin v. L?wis wrote: > Brett C. wrote: > >> I noticed that Makefile.pre.in uses the value from the environment >> variable LDFLAGS but not CPPFLAGS. Any reason for this? > > > How did you notice that? For LDFLAGS, Makefile.pre.in has > > LDFLAGS= @LDFLAGS@ > > This does *not* mean that the value from the environment is used. > Instead, it means that configure computes the value of LDFLAGS > when it generates Makefile.in. For CPPFLAGS, configure has nothing > to compute, so Makefile.pre.in just has the static value for > CPPFLAGS. > I realize that much. But if you look in configure.in it seems to use the previous value of LDFLAGS every time it is redefined as a base and thus gets its initial value from the environment variable the first time it is tweaked. Not a big deal, though. I will just use the environment variables in setup.py . -Brett From python at rcn.com Mon Dec 6 01:05:56 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon Dec 6 01:09:05 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() In-Reply-To: <200412052337.iB5NbC5Y318887@boa.lbl.gov> Message-ID: <000601c4db27$6b1bad80$e841fea9@oemcomputer> > To restate my original goal: > > I am looking for a simple way to answer the question: How much of a > speedup can I expect if I reimplement a piece of Python code in C or > C++? . . . > Ratios (rounded to 3 decimals): > 16.9193/16.8831=1.002 > 5.8287/5.9553 =0.979 > 1.5944/1.5914 =1.002 > 10.7931/10.8149=0.998 > 5.2865/5.2528 =1.006 > 11.6086/11.6437=0.997 > 10.0992/11.0248=0.916 > 27.6830/27.6960=1.000 > > Therefore I'd argue that the runtime penalty for the one additional > long long increment in ceval.c (_Py_TickerAccumulation++) is in the > noise. The measurements are too imprecise to draw any worthwhile conclusions. Try running: python timeit.py -r9 "pass" That ought to give more stable measurements. The proposed "analysis tool" has no benefit to a majority of Python users. Even a 1% hit is not worth it. > I am only interested in counting the > iterations of the interpreter loop. However, the _Py_Ticker decrements > in longobject.c are not inside the interpreter loop, but in C loops! > This means _Py_Ticker is useless for my purposes. Therefore I decoupled > _Py_Ticker and _Py_TickerAccumulation. Why add this to everyone's build? Just put it in when doing your own analysis. The eval loop already pays a penalty for Py2.4's extra function tracing code. And ceval.c has been cluttered with #ifdefs for hardware timestamps. And there have been other additions for signal handling and whatnot. This is enough. >. A lot of fallout caused by the simple idea to add one innocent > line to ceval.c. I do not find it to be innocent. A great of work was expended over the years just trying to eliminate a small step or two from the eval-loop. Those efforts should not be discarded lightly. -1 on adding it directly. -0 on adding it as a #ifdeffed compile option (with the default being to exclude it). Raymond Hettinger From python at rcn.com Mon Dec 6 01:33:17 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon Dec 6 01:36:01 2004 Subject: [Python-Dev] pystone rant [was] proposal+patch: sys.gettickeraccumulation() In-Reply-To: <200412052337.iB5NbC5Y318887@boa.lbl.gov> Message-ID: <000f01c4db2b$2e119f40$e841fea9@oemcomputer> (Old message) > > =) Parrotbench and PyBench would be enough for me to sign off on any > > performance hit. There is also pystone. pystone is good a predicting the relative performance of python apps on difference hardware/software environments. As a tool for evaluating proposed language changes, it completely sucks and should *never* be used for anything other than an extra informational datapoint. The foremost issue is that it exercises only a tiny portion of the language. Its design makes it worse than totally useless for evaluating eval-loop overhead. It runs a loop twice, once with benchmarking code and once without. Only the difference is reported. In theory, that means that all eval-loop speedups and slowdowns are netted out of the result. In practice, the reported result may indicate exactly the opposite of reality because the empty loop has vastly different cache effects than the benchmarked loop. For useful timings, run timeit on the specific feature in question. Then check for overall impact using pybench, parrotbench, and test_decimal. Raymond From barry at python.org Mon Dec 6 02:10:28 2004 From: barry at python.org (Barry Warsaw) Date: Mon Dec 6 02:10:31 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <41B39EA4.9030105@ocf.berkeley.edu> References: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> <200412060243.01129.anthony@interlink.com.au> <41B34B62.7040201@v.loewis.de> <41B39EA4.9030105@ocf.berkeley.edu> Message-ID: <1102295428.8361.8.camel@presto.wooz.org> On Sun, 2004-12-05 at 18:49, Brett C. wrote: > And as for the mention of dropping PEP 4, I agree with the running consensus > that it isn't needed thanks to the warning module. Do we need to have a more > formal vote to drop PEP 4, or should one the PEP maintainers just delete it? There's really no such thing as "dropping" a PEP, but in any event, we should still keep PEP 4 to document the procedure for deprecating modules. It just doesn't need to list any modules (i.e. remove the sections labeled Obsolete modules, Deprecated modules, and Undeprecated modules). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041205/c1ad06b4/attachment.pgp From anthony at interlink.com.au Mon Dec 6 02:19:31 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Dec 6 02:41:22 2004 Subject: [Python-Dev] pystone rant [was] proposal+patch: sys.gettickeraccumulation() In-Reply-To: <000f01c4db2b$2e119f40$e841fea9@oemcomputer> References: <000f01c4db2b$2e119f40$e841fea9@oemcomputer> Message-ID: <200412061219.32190.anthony@interlink.com.au> On Monday 06 December 2004 11:33, Raymond Hettinger wrote: > pystone is good a predicting the relative performance of python apps on > difference hardware/software environments. This comes up so often that I'm almost tempted to add a disclaimer to the pystone output. I can't count the number of times people have tried to claim that pystone showed that a change was good/neutral. From ncoghlan at iinet.net.au Mon Dec 6 09:53:42 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Mon Dec 6 09:53:49 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <41B34B62.7040201@v.loewis.de> References: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> <200412060243.01129.anthony@interlink.com.au> <41B34B62.7040201@v.loewis.de> Message-ID: <41B41E16.90100@iinet.net.au> Martin v. L?wis wrote: > Anthony Baxter wrote: > >> The library docs for, e.g. xmllib already say deprecated at the top - >> maybe >> it should be larger? > > > I think it would be better to *remove* the xmllib documentation. > Existing code which needs the module will continue to work even > without documentation, and new code is unlikely to be written for > a module that has no documentation, and where importing the module > gives a DeprecationWarning. If we went down this path, I would suggest we include a list of deprecated modules in the docs to forestall "where is the documentation for these modules?" questions. Such a list would also make life easier for anyone stuck with maintaining legacy Python code. E.g. X.X Deprecated modules The following modules exist in the standard library solely for backwards compatibility with older versions of Python. They are no longer supported, and should not be used for new code. Anyone maintaining code using these modules should refer to the documentation for the Python version listed in the table below. Module Name | Last documented in: ------------------------------------- lib1 | Python 2.2 lib2 | Python 2.3 etc. . . I'd also suggest including a link to this list at the bottom of the global module index. >> Alternately, we could lock the folks complaining about the stdlib's state >> in a room with the folks who complain that every new thing has "wrecked >> the language that they knew and loved" and let them fight it out. > > That sounds like a fair and democratic way of solving the issue. If we sold tickets, it could even be a nice little earner for the PSF ;) Cheers, Nick. -- Nick Coghlan | ncoghlan@email.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net From mwh at python.net Mon Dec 6 13:12:48 2004 From: mwh at python.net (Michael Hudson) Date: Mon Dec 6 13:12:50 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <41B34B62.7040201@v.loewis.de> ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "Sun, 05 Dec 2004 18:54:42 +0100") References: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> <200412060243.01129.anthony@interlink.com.au> <41B34B62.7040201@v.loewis.de> Message-ID: <2mhdmzr5fj.fsf@starship.python.net> "Martin v. L?wis" writes: > Anthony Baxter wrote: >> The library docs for, e.g. xmllib already say deprecated at the top - maybe >> it should be larger? > > I think it would be better to *remove* the xmllib documentation. > Existing code which needs the module will continue to work even > without documentation, and new code is unlikely to be written for > a module that has no documentation, and where importing the module > gives a DeprecationWarning. +1, at least for *hiding* the documentation of xmllib + similar modules. I'm not sure wholesale removal is really a good idea. It's like the new "Non-essential Built-in Functions" section of the lib ref. Cheers, mwh -- jemfinch: What's to parse? A numeric code, perhaps a chicken, and some arguments -- from Twisted.Quotes From carribeiro at gmail.com Mon Dec 6 13:23:09 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Mon Dec 6 13:23:11 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <2mhdmzr5fj.fsf@starship.python.net> References: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> <200412060243.01129.anthony@interlink.com.au> <41B34B62.7040201@v.loewis.de> <2mhdmzr5fj.fsf@starship.python.net> Message-ID: <864d3709041206042352920501@mail.gmail.com> On Mon, 06 Dec 2004 12:12:48 +0000, Michael Hudson wrote: > "Martin v. L?wis" writes: > > > Anthony Baxter wrote: > >> The library docs for, e.g. xmllib already say deprecated at the top - maybe > >> it should be larger? > > > > I think it would be better to *remove* the xmllib documentation. > > Existing code which needs the module will continue to work even > > without documentation, and new code is unlikely to be written for > > a module that has no documentation, and where importing the module > > gives a DeprecationWarning. > > +1, at least for *hiding* the documentation of xmllib + similar > modules. I'm not sure wholesale removal is really a good idea. 1) Move the deprecated documentation into a separate book. 2) Don't include the "Deprecated Modules Reference" in the standard distribution, but make it available as a separate download. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From ncoghlan at iinet.net.au Mon Dec 6 13:35:11 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Mon Dec 6 13:35:18 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <864d3709041206042352920501@mail.gmail.com> References: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> <200412060243.01129.anthony@interlink.com.au> <41B34B62.7040201@v.loewis.de> <2mhdmzr5fj.fsf@starship.python.net> <864d3709041206042352920501@mail.gmail.com> Message-ID: <41B451FF.1000909@iinet.net.au> Carlos Ribeiro wrote: > On Mon, 06 Dec 2004 12:12:48 +0000, Michael Hudson wrote: >>+1, at least for *hiding* the documentation of xmllib + similar >>modules. I'm not sure wholesale removal is really a good idea. > > > 1) Move the deprecated documentation into a separate book. > > 2) Don't include the "Deprecated Modules Reference" in the standard > distribution, but make it available as a separate download. What about the idea of pointing to the last version of the docs that contained the relevant documentation? Cheers, Nick. -- Nick Coghlan | ncoghlan@email.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net From python at rcn.com Mon Dec 6 13:44:53 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon Dec 6 13:47:37 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <864d3709041206042352920501@mail.gmail.com> Message-ID: <001101c4db91$627e4600$e841fea9@oemcomputer> > > > I think it would be better to *remove* the xmllib documentation. > > > Existing code which needs the module will continue to work even > > > without documentation, and new code is unlikely to be written for > > > a module that has no documentation, and where importing the module > > > gives a DeprecationWarning. > > > > +1, at least for *hiding* the documentation of xmllib + similar > > modules. I'm not sure wholesale removal is really a good idea. > 1) Move the deprecated documentation into a separate book. > > 2) Don't include the "Deprecated Modules Reference" in the standard > distribution, but make it available as a separate download. -1 We are trying to remove clutter, not keep it in perpetuity. Leaving the docs means continuing to have to test it, field bug reports, be aware of its existence, etc. Do not keep this on a respirator. Let it die in peace. Raymond From barry at python.org Mon Dec 6 13:55:35 2004 From: barry at python.org (Barry Warsaw) Date: Mon Dec 6 13:55:37 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <001101c4db91$627e4600$e841fea9@oemcomputer> References: <001101c4db91$627e4600$e841fea9@oemcomputer> Message-ID: <1102337735.8362.36.camel@presto.wooz.org> On Mon, 2004-12-06 at 07:44, Raymond Hettinger wrote: > We are trying to remove clutter, not keep it in perpetuity. Leaving the > docs means continuing to have to test it, field bug reports, be aware of > its existence, etc. > > Do not keep this on a respirator. Let it die in peace. Old documentation lives on in http://www.python.org/doc/versions.html which seems sufficient for looking up deprecated modules. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041206/5e733f08/attachment.pgp From bac at OCF.Berkeley.EDU Mon Dec 6 21:32:02 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon Dec 6 21:32:24 2004 Subject: [Python-Dev] Any reason why CPPFLAGS not used in compiling? In-Reply-To: <41B2C991.4090502@v.loewis.de> References: <41B2BE2E.3000907@ocf.berkeley.edu> <41B2C991.4090502@v.loewis.de> Message-ID: <41B4C1C2.7060207@ocf.berkeley.edu> Martin v. L?wis wrote: > Brett C. wrote: > >> I noticed that Makefile.pre.in uses the value from the environment >> variable LDFLAGS but not CPPFLAGS. Any reason for this? > > > How did you notice that? For LDFLAGS, Makefile.pre.in has > > LDFLAGS= @LDFLAGS@ > > This does *not* mean that the value from the environment is used. > Instead, it means that configure computes the value of LDFLAGS > when it generates Makefile.in. For CPPFLAGS, configure has nothing > to compute, so Makefile.pre.in just has the static value for > CPPFLAGS. > I am not so sure that is true. Checking configure.in, there is no mention of CPPFLAGS anywhere. And yet if I modify the definition of CPPFLAGS in Makefile.pre.in to ``-I. -I./Include @CPPFLAGS@`` it ends up containing the value I have for the environment variable at the end of it. I think the '@@' syntax uses a value from configure.in if it is defined else it defaults to the value the shell has. -Brett From bac at OCF.Berkeley.EDU Mon Dec 6 07:36:14 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon Dec 6 21:37:41 2004 Subject: [Python-Dev] Ptyon 2.3.5 probably coming in January; get your bugs/patches reported! Message-ID: <41B3FDDE.3040302@ocf.berkeley.edu> Anthony Baxter, our ever-diligent release manager, mentioned this past week that Python 2.3.5 will most likely come to fruition some time in January (this is not guaranteed date). This means that in order to have enough time to proper evaluate new patches and bugs they must be reported **now**! A one month lead time is necessary to properly look at, test, and commit patches, let alone coming up with solutions to any reported bugs. Please realize, though, that reporting a bug or submitting a patch now does not guarantee that it will committed in time! The free time of the development team is limited. If you want to help a bug or patch along to make it easier to be evaluated and thus raise its chances of being dealt with please see the "Helping Out" section of the 'Intro to Development' essay at http://www.python.org/dev/dev_intro.html . As always, both bugs and patches should be reported to Python's SourceForge tracker at http://sourceforge.net/bugs/?group_id=5470 and http://sourceforge.net/patch/?group_id=5470, respectively. -Brett Cannon From martin at v.loewis.de Mon Dec 6 22:28:39 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon Dec 6 22:28:33 2004 Subject: [Python-Dev] Rewriting PEP4 In-Reply-To: <41B39EA4.9030105@ocf.berkeley.edu> References: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> <200412060243.01129.anthony@interlink.com.au> <41B34B62.7040201@v.loewis.de> <41B39EA4.9030105@ocf.berkeley.edu> Message-ID: <41B4CF07.4050108@v.loewis.de> Brett C. wrote: > And as for the mention of dropping PEP 4, I agree with the running > consensus that it isn't needed thanks to the warning module. Do we need > to have a more formal vote to drop PEP 4, or should one the PEP > maintainers just delete it? I would do what Barry suggests: rephrase the module to document the deprecation/removal procedure. This, of course, needs consensus/pronouncement, too, but I think I would put the following aspects into it: Requirements ============ Removal of module needs proper advance warning for users; the module should be removed only if - continued usage poses a security threat to the application using it, or - there are no reported usages of the module anymore for quite some time - the module is unmaintained (i.e. there are serious outstanding unfixed bugs for it), and there are alternatives which are maintained. There must be a way for users to stop removal of the module, e.g. by volunteering to maintain an unmaintained module. Deprecation =========== If a module is candidate for removal, it must be deprecated first. In order to deprecate the module 1. a warning must be added to the module, indicating the expected version when the module is removed, and a reference to the PEP 2. a comment must be added just above the warning, giving a the date and version of deprecation, and a rationale for removal (e.g.: no known users, is security risk) 3. the module documentation must be removed from the Python documentation (alternatively: moved into a "deprecated" section) Undeprecation ============= A module can be undeprecated if the deprecation reasons turns out to be invalid, e.g. if users of the module appear when the module was believed to have no users, or if a maintainer shows up for a previously unmaintained module. This fact must be recorded in the module; the original state (documentation, no warning) be restored. Removal ======= If the module has been deprecated for atleast a year and atleast a version, it can be removed. Removal should move it to old-libs for pure Python modules; a removal procedure for pure C modules has not been defined yet. If the module was deprecated because it was unmaintained, the module should only be removed if there have been no recent reports about usage of the module. Bug reports against deprecated modules ====================================== If a bug is reported against a deprecated module, the bug report can be closed without further consideration, pointing to the deprecation status. If a patch is submitted against the module, it should be considered whether the patch could undeprecate the module; if not, it can be rejected with the same rationale. If this policy can be agreed, I will replace PEP4 with it. Modules that have currently deprecation messages in them often fail to identify the Python version in which removal will occur. For modules that have been deprecated since 2.1, I would suggest to remove them for 2.5, with the option of bringing them back in 2.5.1 if people complain. Regards, Martin From martin at v.loewis.de Mon Dec 6 22:31:30 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon Dec 6 22:31:23 2004 Subject: [Python-Dev] Any reason why CPPFLAGS not used in compiling? In-Reply-To: <41B3A00A.30500@ocf.berkeley.edu> References: <41B2BE2E.3000907@ocf.berkeley.edu> <41B2C991.4090502@v.loewis.de> <41B3A00A.30500@ocf.berkeley.edu> Message-ID: <41B4CFB2.6080907@v.loewis.de> Brett C. wrote: > I realize that much. But if you look in configure.in it seems to use > the previous value of LDFLAGS every time it is redefined as a base and > thus gets its initial value from the environment variable the first time > it is tweaked. Yes, it would be possible to do the same thing for CPPFLAGS. However, you really need to combine the values. If you assume you don't know anything about the current value of CPPFLAGS, this might get tricky - you might cause build failures if you honor CPPFLAGS too much. > Not a big deal, though. I will just use the environment variables in > setup.py . You shouldn't do this - you do need to consider the values in the Makefile as well. If you think you need both, you should modify configure.in appropriately. Regards, Martin From martin at v.loewis.de Mon Dec 6 22:39:30 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon Dec 6 22:39:25 2004 Subject: [Python-Dev] Any reason why CPPFLAGS not used in compiling? In-Reply-To: <41B4C1C2.7060207@ocf.berkeley.edu> References: <41B2BE2E.3000907@ocf.berkeley.edu> <41B2C991.4090502@v.loewis.de> <41B4C1C2.7060207@ocf.berkeley.edu> Message-ID: <41B4D192.1040307@v.loewis.de> Brett C. wrote: >> How did you notice that? For LDFLAGS, Makefile.pre.in has >> >> LDFLAGS= @LDFLAGS@ >> >> This does *not* mean that the value from the environment is used. >> Instead, it means that configure computes the value of LDFLAGS >> when it generates Makefile.in. For CPPFLAGS, configure has nothing >> to compute, so Makefile.pre.in just has the static value for >> CPPFLAGS. >> > > I am not so sure that is true. Checking configure.in, there is no > mention of CPPFLAGS anywhere. Right. That's what I meant when I said "has nothing to compute", so it does not even mention CPPFLAGS. > And yet if I modify the definition of > CPPFLAGS in Makefile.pre.in to ``-I. -I./Include @CPPFLAGS@`` it ends up > containing the value I have for the environment variable at the end of > it. I think the '@@' syntax uses a value from configure.in if it is > defined else it defaults to the value the shell has. Indeed, that seems to be the case. However, absence of @CPPFLAGS@ means that Makefile.pre will just use the static value from Makefile.pre.in. Whether or not adding @CPPFLAGS@ to the end is the right thing, I don't know. Regards, Martin From barry at python.org Mon Dec 6 22:54:52 2004 From: barry at python.org (Barry Warsaw) Date: Mon Dec 6 22:55:04 2004 Subject: [Python-Dev] Rewriting PEP4 In-Reply-To: <41B4CF07.4050108@v.loewis.de> References: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> <200412060243.01129.anthony@interlink.com.au> <41B34B62.7040201@v.loewis.de> <41B39EA4.9030105@ocf.berkeley.edu> <41B4CF07.4050108@v.loewis.de> Message-ID: <1102370092.9565.30.camel@geddy.wooz.org> On Mon, 2004-12-06 at 16:28, "Martin v. L?wis" wrote: Martin, +1 on everything you wrote, with one minor quibble. > Removal > ======= > If the module has been deprecated for atleast a year and atleast > a version, it can be removed. Removal should move it to old-libs > for pure Python modules; a removal procedure for pure C modules > has not been defined yet. I wonder if the one year/one version rule is too short. Given that new versions come about every 18 months, I'd probably give it a 2 year/one version limit. > Modules that have currently deprecation messages in them often > fail to identify the Python version in which removal will occur. > For modules that have been deprecated since 2.1, I would suggest > to remove them for 2.5, with the option of bringing them back > in 2.5.1 if people complain +1 here too. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041206/54057bcd/attachment.pgp From eirik.mikkelsen at unix.net Mon Dec 6 23:58:40 2004 From: eirik.mikkelsen at unix.net (Eirik Mikkelsen) Date: Mon Dec 6 23:58:42 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: <41B34B62.7040201@v.loewis.de> References: <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> <200412060243.01129.anthony@interlink.com.au> <41B34B62.7040201@v.loewis.de> Message-ID: <1102373920.29451.4.camel@localhost> On Sun, 2004-12-05 at 18:54 +0100, "Martin v. L?wis" wrote: > I think it would be better to *remove* the xmllib documentation. > Existing code which needs the module will continue to work even > without documentation, and new code is unlikely to be written for > a module that has no documentation, and where importing the module > gives a DeprecationWarning. Not sure I'm a registered voter, but I'll chip in with a +1 If someone needs the documentation for xmllib it will still be available in the older versions. Eirik. From janssen at parc.com Tue Dec 7 01:25:07 2004 From: janssen at parc.com (Bill Janssen) Date: Tue Dec 7 01:25:23 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: Your message of "Sun, 05 Dec 2004 02:40:16 PST." <000701c4dab6$cf5e8e00$3b01a044@oemcomputer> Message-ID: <04Dec6.162511pst."58617"@synergy1.parc.xerox.com> > * The average quality of the library improves as we take out junk (the > tzparse module for example) and put in high quality modules like > logging, csv, decimal, etc. Yes and no. The added modules have to be relevant to what users want to do. While (relatively) minor stuff like csv and decimal are good ideas, of course, logging is kind of an "insider's" module. What many more users want, however, are things like an XML parser, or a CSS parser, or a usable HTTP server, or... The fact that XML 1.1 can't be parsed with already-written Python is a *big* strike against. So removing highly demanded functionality like XML parsing, and adding specialist modules such as csv, do not overall help what users see as the quality of the standard library. Bill From t-meyer at ihug.co.nz Tue Dec 7 01:35:16 2004 From: t-meyer at ihug.co.nz (Tony Meyer) Date: Tue Dec 7 01:36:03 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: Message-ID: >> * The average quality of the library improves as we take >> out junk (the tzparse module for example) and put in high >> quality modules like logging, csv, decimal, etc. > > Yes and no. The added modules have to be relevant to what > users want to do. While (relatively) minor stuff like csv > and decimal are good ideas, of course, logging is kind of an > "insider's" module. What do you mean by "insiders"? The logging module is great (ok, it could be simpler to use in some cases) for any Python programmer. > What many more users want, however, are things like an XML > parser, or a CSS parser, or a usable HTTP server, or... Statements like this are pretty common, but there's no evidence (that I've ever seen pointed to) that someone has *measured* how many people want modules for X. People who work with HTML at lot are probably keen on those things you list, yes. OTOH, other people (e.g. me) have no use for any of those, but use CSV and logging daily. Others want something completely different. There's quite a difference between quality and relevance. It's certainly worthwhile to ensure that all the standard library modules are as of high a quality as possible (e.g. email > rfc822). You'll never be able to get everyone to agree on the same set of modules that are relevant. If there are that many people that want (e.g.) a CSS parser, wouldn't there be a 3rd party one that everyone is using that could be proposed for addition into the standard library? =Tony.Meyer From python at rcn.com Tue Dec 7 01:43:07 2004 From: python at rcn.com (Raymond Hettinger) Date: Tue Dec 7 01:46:02 2004 Subject: [Python-Dev] Rewriting PEP4 In-Reply-To: <41B4CF07.4050108@v.loewis.de> Message-ID: <002401c4dbf5$b847cc00$f301a044@oemcomputer> > I would do what Barry suggests: rephrase the module to document the > deprecation/removal procedure. This, of course, needs > consensus/pronouncement, too, but I think I would put the following > aspects into it: > > Requirements > ============ > Removal of module needs proper advance warning for users; the > module should be removed only if > - continued usage poses a security threat to the application > using it, or > - there are no reported usages of the module anymore for quite > some time > - the module is unmaintained (i.e. there are serious outstanding > unfixed bugs for it), and there are alternatives which are > maintained. > There must be a way for users to stop removal of the module, > e.g. by volunteering to maintain an unmaintained module. The effect of coding this into the PEP is to make deprecation unnecessarily involved. Also, the list of reasons is insufficiently comprehensive. In the past we've gotten along fine with discussing a situation on python-dev and either reaching a consensus or having Guido give an okay. > 2. a comment must be added just above the warning, giving a > the date and version of deprecation, and a > rationale for removal (e.g.: no known users, is security risk) We should only talk in terms of versions. Dates are irrelevant and confusing. In the last version of the PEP, it was far from clear the significane of deprecating or removing on date xx-xx-xx. All that matters is the version the warning first appears and the version where the module is removed. > Modules that have currently deprecation messages in them often > fail to identify the Python version in which removal will occur. > For modules that have been deprecated since 2.1, I would suggest > to remove them for 2.5, with the option of bringing them back > in 2.5.1 if people complain. I do not think the version number messages are useful. When someone imports a file and gets a warning, they know a deprecation is pending and to stop writing code with that module. Further, I want to avoid the previous PEP 4 situation where one nit or another wasn't followed to the letter so we had to keep the module around for another two versions. It can and should be as simple as having us agree to deprecation, inserting a warning, and then removing it two versions later. Also, we need to shy away from the idea of having 2.5.1 with different capabilities than 2.5. This sort of thing is a portability disaster. Anthony, please speak up. General comments: I had thought the purpose of rewording the PEP was just to take out language naming specific modules and to add language on using the warning module. Instead, it looks like an attempt to make it much more difficult and slow to deprecate a module. If so, I would like to understand what situation is motivating it. Is there some module that had been deprecated but should not have been? Why the change in procedure? It doesn't affect me personally (I haven't proposed any modules for deprecation and do not have any in mind); however, I think the discussion should take place in the broader context of what we intend to do with the standard library -- do we want regex still there in 2010. Also, as proposed, the PEP lists many forces against deprecation and none of the forces for it. If that is the reality, then we could just never deprecate anything because someone, somewhere is bound to be irritated by it. That would certainly simplify the discussion. Going back to specifics, it may be worthwhile to think through the reason we kept the xmllib code but not whrandom. Both were documented, non-buggy, tested, marked as deprecated for a long time, and it was within the realm of possibility that some code still used them. Also, the PEP should discuss the use of the lib-old directory and it should include Barry's link to old documentation. Raymond From bac at OCF.Berkeley.EDU Tue Dec 7 01:52:08 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Tue Dec 7 01:52:15 2004 Subject: [Python-Dev] Any reason why CPPFLAGS not used in compiling? In-Reply-To: <41B4D192.1040307@v.loewis.de> References: <41B2BE2E.3000907@ocf.berkeley.edu> <41B2C991.4090502@v.loewis.de> <41B4C1C2.7060207@ocf.berkeley.edu> <41B4D192.1040307@v.loewis.de> Message-ID: <41B4FEB8.90906@ocf.berkeley.edu> Martin v. L?wis wrote: > Brett C. wrote: > >>> How did you notice that? For LDFLAGS, Makefile.pre.in has >>> >>> LDFLAGS= @LDFLAGS@ >>> >>> This does *not* mean that the value from the environment is used. >>> Instead, it means that configure computes the value of LDFLAGS >>> when it generates Makefile.in. For CPPFLAGS, configure has nothing >>> to compute, so Makefile.pre.in just has the static value for >>> CPPFLAGS. >>> >> >> I am not so sure that is true. Checking configure.in, there is no >> mention of CPPFLAGS anywhere. > > > Right. That's what I meant when I said "has nothing to compute", so > it does not even mention CPPFLAGS. > >> And yet if I modify the definition of CPPFLAGS in Makefile.pre.in to >> ``-I. -I./Include @CPPFLAGS@`` it ends up containing the value I have >> for the environment variable at the end of it. I think the '@@' >> syntax uses a value from configure.in if it is defined else it >> defaults to the value the shell has. > > > Indeed, that seems to be the case. However, absence of @CPPFLAGS@ > means that Makefile.pre will just use the static value from > Makefile.pre.in. > That's basically the functionality I need, so I am going with it. > Whether or not adding @CPPFLAGS@ to the end is the right thing, > I don't know. > Well, we will soon find out. =) My checkin made this change and everything seems fine. If it doesn't work out I will have to create another environment variable to store the value. Michael's desire of getting the Fink and DarwinPorts special casing in setup.py is now solved; setup.py now uses the directories specified in LDFLAGS and CPPFLAGS for compiling the extension modules. I didn't bother with CFLAGS or CC since the former is mostly handled by BASECFLAGS it seems and the latter is specified by arguments to configure. -Brett From python at rcn.com Tue Dec 7 01:53:05 2004 From: python at rcn.com (Raymond Hettinger) Date: Tue Dec 7 01:55:58 2004 Subject: [Python-Dev] Rewriting PEP4 In-Reply-To: <002401c4dbf5$b847cc00$f301a044@oemcomputer> Message-ID: <002801c4dbf7$1ce6a860$f301a044@oemcomputer> One other thought: In deciding how long to leave module in, we should consider that Python books are updated infrequently, if at all. It would be a bummer if code in them stopped working as advertised. Raymond From anthony at interlink.com.au Tue Dec 7 02:31:13 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Dec 7 02:31:29 2004 Subject: [Python-Dev] Rewriting PEP4 In-Reply-To: <002401c4dbf5$b847cc00$f301a044@oemcomputer> References: <002401c4dbf5$b847cc00$f301a044@oemcomputer> Message-ID: <200412071231.13878.anthony@interlink.com.au> On Tuesday 07 December 2004 11:43, Raymond Hettinger wrote: > > Modules that have currently deprecation messages in them often > > fail to identify the Python version in which removal will occur. > > For modules that have been deprecated since 2.1, I would suggest > > to remove them for 2.5, with the option of bringing them back > > in 2.5.1 if people complain. > > [...] > > Also, we need to shy away from the idea of having 2.5.1 with different > capabilities than 2.5. This sort of thing is a portability disaster. > Anthony, please speak up. Sorry - been a bit busy the last few days. Raymond is right here - there's no way we can or should do this. A bugfix release contains _bugfixes_. Making new modules available in a bugfix is a sucky sucky approach. Worse - once we remove a module, it should _stay_ removed. Otherwise, we're going to end up with crap like: if sys.version < (2,5) or sys.version >= (2,6): import froggie else: # froggie was removed in 2.5 and reinstated in 2.6 from compat import froggie and people will be shipping their own versions of the code to get around our misfeature. -- Anthony Baxter It's never too late to have a happy childhood. From janssen at parc.com Tue Dec 7 02:49:40 2004 From: janssen at parc.com (Bill Janssen) Date: Tue Dec 7 02:50:15 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: Your message of "Mon, 06 Dec 2004 16:35:16 PST." Message-ID: <04Dec6.174948pst."58617"@synergy1.parc.xerox.com> > Statements like this are pretty common, but there's no evidence (that I've > ever seen pointed to) that someone has *measured* how many people want > modules for X. I almost didn't send this in, because I figured someone would have to argue with it. > If there are that many people that want (e.g.) a CSS parser, wouldn't there > be a 3rd party one that everyone is using that could be proposed for > addition into the standard library? As far as I can tell, there are no CSS or XML 1.1 parsers for Python, period. Bill From t-meyer at ihug.co.nz Tue Dec 7 03:04:49 2004 From: t-meyer at ihug.co.nz (Tony Meyer) Date: Tue Dec 7 03:05:23 2004 Subject: [Python-Dev] Deprecated xmllib module In-Reply-To: Message-ID: > As far as I can tell, there are no CSS or XML 1.1 parsers for > Python, period. This belongs on c.l.p, I suppose, but the first page of google results includes: =Tony.Meyer From sjoerd at acm.org Mon Dec 6 21:47:37 2004 From: sjoerd at acm.org (Sjoerd Mullender) Date: Tue Dec 7 03:19:07 2004 Subject: [Python-Dev] Any reason why CPPFLAGS not used in compiling? In-Reply-To: <41B4C1C2.7060207@ocf.berkeley.edu> References: <41B2BE2E.3000907@ocf.berkeley.edu> <41B2C991.4090502@v.loewis.de> <41B4C1C2.7060207@ocf.berkeley.edu> Message-ID: <41B4C569.8060608@acm.org> Brett C. wrote: > Martin v. L?wis wrote: > >> Brett C. wrote: >> >>> I noticed that Makefile.pre.in uses the value from the environment >>> variable LDFLAGS but not CPPFLAGS. Any reason for this? >> >> >> >> How did you notice that? For LDFLAGS, Makefile.pre.in has >> >> LDFLAGS= @LDFLAGS@ >> >> This does *not* mean that the value from the environment is used. >> Instead, it means that configure computes the value of LDFLAGS >> when it generates Makefile.in. For CPPFLAGS, configure has nothing >> to compute, so Makefile.pre.in just has the static value for >> CPPFLAGS. >> > > I am not so sure that is true. Checking configure.in, there is no > mention of CPPFLAGS anywhere. And yet if I modify the definition of > CPPFLAGS in Makefile.pre.in to ``-I. -I./Include @CPPFLAGS@`` it ends up > containing the value I have for the environment variable at the end of > it. I think the '@@' syntax uses a value from configure.in if it is > defined else it defaults to the value the shell has. It's autoconf that deals with these flags. See the output of "configure --help". -- Sjoerd Mullender -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 374 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041206/bfe3b473/signature.pgp From kbk at shore.net Tue Dec 7 05:33:45 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Tue Dec 7 05:35:36 2004 Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200412070433.iB74XjGf019407@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 259 open ( +1) / 2705 closed ( +4) / 2964 total ( +5) Bugs : 800 open (-12) / 4662 closed (+20) / 5462 total ( +8) RFE : 160 open ( +0) / 137 closed ( +1) / 297 total ( +1) New / Reopened Patches ______________________ add key (2004-12-01) CLOSED http://python.org/sf/1077353 opened by Steven Bethard Simple webbrowser fix for netscape -remote (2004-12-02) http://python.org/sf/1077979 opened by Josh Cherry readline does not need termcap (2004-12-01) http://python.org/sf/1076826 reopened by nijel Make cgi.py use logging module (2004-12-05) http://python.org/sf/1079729 opened by Josh Hoyt Make cgi.py use email instead of rfc822 or mimetools (2004-12-05) http://python.org/sf/1079734 opened by Josh Hoyt list sort is not "in place" (2004-12-06) http://python.org/sf/1080078 reopened by rhettinger list sort is not "in place" (2004-12-06) CLOSED http://python.org/sf/1080078 opened by Heikki Orsila Patches Closed ______________ add key (2004-12-02) http://python.org/sf/1077353 closed by rhettinger readline does not need termcap (2004-12-01) http://python.org/sf/1076826 closed by loewis raise error for common mistake with subprocess (2004-11-23) http://python.org/sf/1071755 closed by astrand list sort is not "in place" (2004-12-06) http://python.org/sf/1080078 closed by rhettinger list sort is not "in place" (2004-12-06) http://python.org/sf/1080078 closed by tim_one AUTH PLAIN in smtplib (2004-11-30) http://python.org/sf/1075928 closed by jlgijsbers New / Reopened Bugs ___________________ problem with make test on OS/X (2004-12-02) CLOSED http://python.org/sf/1077302 opened by Ian Holsman threads: segfault or Py_FatalError at exit (2004-11-08) http://python.org/sf/1061968 reopened by mhammond assert fail to threw exception when run python with '-O' (2004-12-02) CLOSED http://python.org/sf/1077862 opened by tj128 Python2.4: building '_socket' extension fails with `INET_ADD (2004-12-03) http://python.org/sf/1078245 opened by Michael Str?der Docs for run() methods are misleading (2004-12-04) CLOSED http://python.org/sf/1078905 opened by Kent Johnson Email.Header encodes non-ASCII content incorrectly (2004-12-04) http://python.org/sf/1078919 opened by Tessa Lau Incorrect error message (somewhat) (2004-12-04) http://python.org/sf/1079011 opened by Gerrit Holl datetime changes missing from "Porting from 2.3 to 2.4" (2004-12-04) http://python.org/sf/1079134 opened by Sadruddin Rejeb python-2.4.msi install error (2004-12-05) http://python.org/sf/1079545 opened by maharal IDLE and old color themes (2004-12-06) http://python.org/sf/1080387 opened by projecktzero Bugs Closed ___________ threads: segfault or Py_FatalError at exit (2004-11-07) http://python.org/sf/1061968 closed by arigo Tutorial corrections (2004-12-01) http://python.org/sf/1076955 closed by rhettinger test_shutil fails on x86-64 // Suse 9.1 (2004-12-01) http://python.org/sf/1076467 closed by jlgijsbers test test_unicodedata failed (2004-12-01) http://python.org/sf/1076795 closed by lemburg Problem testing python 2.4 (2004-12-01) http://python.org/sf/1077103 closed by lemburg re module segfaulting in large regular expression (2004-11-24) http://python.org/sf/1072259 closed by niemeyer Memory fault pyexpat.so on SGI (2004-11-30) http://python.org/sf/1075990 closed by akuchling assert fail to threw exception when run python with '-O' (2004-12-02) http://python.org/sf/1077862 closed by tim_one Errors and omissions in logging module documentation (2004-11-28) http://python.org/sf/1074693 closed by vsajip xml.dom.minidom produces errors with certain unicode chars (2004-11-27) http://python.org/sf/1074200 closed by effbot test test_re produced unexpected output (2004-12-01) http://python.org/sf/1076791 closed by nijel ^Z doesn't exit interpreter - 2.4c1 & Win2K (2004-11-26) http://python.org/sf/1073736 closed by facundobatista test_macostools failure on OS X (2004-12-01) http://python.org/sf/1077302 closed by bcannon PyGILState_Ensure() deadlocks (ver 2.4) (2004-11-29) http://python.org/sf/1075703 closed by andivajda calendar.weekheader() undocumented (2004-05-04) http://python.org/sf/947894 closed by doerwalter Docs for unittest run() methods are misleading (2004-12-04) http://python.org/sf/1078905 closed by rhettinger Another message that croaks email.FeedParser (2004-11-30) http://python.org/sf/1076485 closed by bwarsaw Windows msi doesn't install site-packages directory (2004-11-23) http://python.org/sf/1071594 closed by loewis win32con missing codes VK_VOLUME_MUTE, VK_BROWSER_BACK, ... (2004-11-16) http://python.org/sf/1067153 closed by loewis test_sutil fails on cygwin (2004-11-23) http://python.org/sf/1071513 closed by jlgijsbers CFLAGS, CPPFLAGS, LDFLAGS are NOT used when building modules (2004-05-17) http://python.org/sf/955197 closed by bcannon RFE Closed __________ Python Interpreter embedded in small memory system (2004-11-30) http://python.org/sf/1076478 closed by rhettinger From martin at v.loewis.de Tue Dec 7 08:08:46 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Dec 7 08:08:39 2004 Subject: [Python-Dev] Rewriting PEP4 In-Reply-To: <002401c4dbf5$b847cc00$f301a044@oemcomputer> References: <002401c4dbf5$b847cc00$f301a044@oemcomputer> Message-ID: <41B556FE.20504@v.loewis.de> Raymond Hettinger wrote: > The effect of coding this into the PEP is to make deprecation > unnecessarily involved. Can you please explain why you consider this involvement unnecessary? I think it is important that we do not prematurely remove (or even deprecate) modules that are still being actively used. > Also, the list of reasons is insufficiently > comprehensive. In the past we've gotten along fine with discussing a > situation on python-dev and either reaching a consensus or having Guido > give an okay. I don't think we "got along", but rather "got away". Discussion on python-dev does not cause sufficient user involvement, IMO. And indeed, some people (even on python-dev) think that we *never* should deprecate any modules, as it breaks existing code. They might not be as vocal as supporters of early removal of "ugly" code, but I think their opinion is just as valuable. > We should only talk in terms of versions. Dates are irrelevant and > confusing. Why is a date confusing? I think everybody on Earth understands the notion of a date, there is nothing confusing in this notion. Why are they irrelevant? If we release Python 2.5 next month (and stop developing 2.4), are we entitled to remove all features that were scheduled for removal? I think not: users would have not gotten sufficient advance warning. Users need *time* to react on proposed feature changes. > In the last version of the PEP, it was far from clear the > significane of deprecating or removing on date xx-xx-xx. All that > matters is the version the warning first appears and the version where > the module is removed. No. It also matters how much time users have had to react. > I do not think the version number messages are useful. When someone > imports a file and gets a warning, they know a deprecation is pending > and to stop writing code with that module. Yes, but what about code that currently uses the module? People will not start rewriting (perhaps they aren't the authors in the first place) just because they see a deprecation warning. Instead, they will silence the warning, and go on. If we plan to remove the feature at some point, we need to give these users an indication that they need to act beyond silencing the warning. Such actions could be - have the author of the software update it appropriately - look for a replacement software - ask for an extension of the grace period on python-dev Currently, users do not really have these options, since they do not have an indication that action on their side is really needed if they want their software continue to work with future Python versions. > Further, I want to avoid the > previous PEP 4 situation where one nit or another wasn't followed to the > letter so we had to keep the module around for another two versions. Why do you want to avoid that situation? What is the problem with waiting for two more versions? No harm is done in doing so. > It > can and should be as simple as having us agree to deprecation, inserting > a warning, and then removing it two versions later. I strongly disagree with that position. > I had thought the purpose of rewording the PEP was just to take out > language naming specific modules and to add language on using the > warning module. Instead, it looks like an attempt to make it much more > difficult and slow to deprecate a module. I want the PEP to be followed. For that, it needs to be precise, and the procedures need to be agreed upon (or, if we cannot agree, it needs to be Pronounced). > If so, I would like to understand what situation is motivating it. Is > there some module that had been deprecated but should not have been? Deprecation is fine for all the modules so far. Generating the warning is a bit of a problem, removal would be a serious problem, for the following modules: - regex - xmllib - whrandom since I expect that these modules are still widely used, in existing code, by users who are not the authors of that code. This is a problem, since these users cannot do anything about the usage of the deprecated code. They just find that the software that ran happily with the previous Python version first shouts at them in the next Python version, then totally breaks with the subsequent version. They will hate Python for that. > Why the change in procedure? I think we agree that the previously-documented procedure needs to be changed - it was not followed. > It doesn't affect me personally (I haven't proposed any modules for > deprecation and do not have any in mind); however, I think the > discussion should take place in the broader context of what we intend to > do with the standard library -- do we want regex still there in 2010. As to the latter question: Definitely. We don't want regex to be used for new code, and we want developers to fully understand that, but we also do want code that currently works correctly with regex continue to do so in 2010. The only exception would be that 2010 no code is left in active use that uses regex, which I doubt. > Also, as proposed, the PEP lists many forces against deprecation and > none of the forces for it. That is not true. Security flaws are giving priority. They can only be undeprecated if the reason for deprecation goes away, which means that somebody fixes the security flaw. > If that is the reality, then we could just > never deprecate anything because someone, somewhere is bound to be > irritated by it. That would certainly simplify the discussion. Well, we do want to move developers to the new direction. However, many people use Python code who are *not* developers. We need to take their concerns into account as well. > Going back to specifics, it may be worthwhile to think through the > reason we kept the xmllib code but not whrandom. Both were documented, > non-buggy, tested, marked as deprecated for a long time, and it was > within the realm of possibility that some code still used them. Indeed. If it is in the realm of possibility that code still uses a module, and that this would not raise security concerns, the module should not be removed. > Also, the PEP should discuss the use of the lib-old directory and it > should include Barry's link to old documentation. I'm happy to adjust details of the procedures - but it seems we disagree on the principles. Regards, Martin From martin at v.loewis.de Tue Dec 7 08:12:21 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Dec 7 08:12:14 2004 Subject: [Python-Dev] Rewriting PEP4 In-Reply-To: <002801c4dbf7$1ce6a860$f301a044@oemcomputer> References: <002801c4dbf7$1ce6a860$f301a044@oemcomputer> Message-ID: <41B557D5.2080700@v.loewis.de> Raymond Hettinger wrote: > One other thought: In deciding how long to leave module in, we should > consider that Python books are updated infrequently, if at all. It > would be a bummer if code in them stopped working as advertised. Correct. Thanks for reminding me - that is indeed a reasoning that is typically brought up by book authors, so I should have brought it up myself :-) Of course, developers would first see the deprecation warning, which would tell them that their books are outdated. However, if the code is removed, they get an import error, which leaves them guessing what the problem might have been. Regards, Martin From python at rcn.com Tue Dec 7 08:50:55 2004 From: python at rcn.com (Raymond Hettinger) Date: Tue Dec 7 08:53:39 2004 Subject: [Python-Dev] Rewriting PEP4 In-Reply-To: <41B556FE.20504@v.loewis.de> Message-ID: <000701c4dc31$7bf6da20$e841fea9@oemcomputer> > I'm happy to adjust details of the procedures - but it seems we disagree > on the principles. Not really. I have no objection to making module deprecation/removal rare or even not doing it at all. Personally, I've never asked for a module deprecation and don't expect to. I also agree that more public participation is a wise step. I would just like to see a clean, actionable PEP. To me, the draft text outlined a timid, faltering process that would be hard to follow, prone to reversal, and accomplish little. I especially find burdensome the obligation to undo a deprecation the moment some random user sends a grumpy email. Instead, there should be a clear decision to deprecate or not. If that entails a comp.lang.python.announce notice and comment period, so be it. Once a decision is made, document it, add a warning, and wait. Once a couple versions pass, some useful action needs to take place. If the code is left in-place and the doc index is just re-arranged, then we would have been better off not doing anything at all. Ideally, the PEP should also provide some informational value. It should list Barry's link as a reference for old docs. It should describe the appropriate use of lib-old (like whrandom) vs. renaming a module (like stringold) vs. leaving it in-place (like xmllib) vs. removing the module The questions of dates was somewhat minor. I was unclear on the implication of, for example, "remove on 15Jan2005". Since Py2.5 won't go out for at least a year, does that mean that the work can't be done now while I have time instead of later when I don't. The only time the removal becomes visible is when a new version goes out the door. Further, if a version is going to have something removed, we should do it at the outset rather than just before a release goes out the door (to allow for early surfacing of any issues). > > Further, I want to avoid the > > previous PEP 4 situation where one nit or another wasn't followed to the > > letter so we had to keep the module around for another two versions. > > Why do you want to avoid that situation? What is the problem with > waiting for two more versions? No harm is done in doing so. If we really don't care whether it gets done, then we shouldn't bother in the first place. Raymond From python at rcn.com Tue Dec 7 09:12:20 2004 From: python at rcn.com (Raymond Hettinger) Date: Tue Dec 7 09:15:03 2004 Subject: [Python-Dev] Rewriting PEP4 In-Reply-To: <000701c4dc31$7bf6da20$e841fea9@oemcomputer> Message-ID: <000801c4dc34$7949b420$e841fea9@oemcomputer> There is one other issue that should get addressed, modules in limbo. I'm concerned about the old email modules. They haven't been deprecated and may stay around for a good while. However, some SF requests have been passed over on the basis that the modules are supplanted and will be phased out. I don't think it is good to leave active modules as orphans. Raymond From martin at v.loewis.de Tue Dec 7 09:16:05 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Dec 7 09:15:58 2004 Subject: [Python-Dev] Rewriting PEP4 In-Reply-To: <000701c4dc31$7bf6da20$e841fea9@oemcomputer> References: <000701c4dc31$7bf6da20$e841fea9@oemcomputer> Message-ID: <41B566C5.2040102@v.loewis.de> Raymond Hettinger wrote: > Instead, there should be a clear decision to deprecate or not. If that > entails a comp.lang.python.announce notice and comment period, so be it. > Once a decision is made, document it, add a warning, and wait. Ok, that might be reasonable. > Once a couple versions pass, some useful action needs to take place. If > the code is left in-place and the doc index is just re-arranged, then we > would have been better off not doing anything at all. I disagree. The primary purpose (move developers to the alternative approach) is achieved as soon as the warning is added, and the documentation is deleted. Whether the module is actually deleted is of minor importance: it costs nothing to continue including it; disk space is cheap. > The questions of dates was somewhat minor. I was unclear on the > implication of, for example, "remove on 15Jan2005". Since Py2.5 won't > go out for at least a year, does that mean that the work can't be done > now while I have time instead of later when I don't. The only time the > removal becomes visible is when a new version goes out the door. You could remove it now, but if we release Py2.5 in two months, you would have to put it back in. So if you don't have time then, maybe somebody else will. If nobody has time to remove the module, the next release will ship with it, again - no big issue. > Further, if a version is going to have something removed, we should do > it at the outset rather than just before a release goes out the door (to > allow for early surfacing of any issues). That is true. > If we really don't care whether it gets done, then we shouldn't bother > in the first place. What do you mean by "bother"? Not put the deprecation warning in? We do want users to move to the new approach of doing something. However, two version are just not enough - it may take 10 or 20 years to remove a widely used feature (e.g. the string module). That it will take so long does not mean we should not start the process *now* - if we start the process in five years, it will *still* take 10 or 20 years. Just be patient. Regards, Martin From python at rcn.com Tue Dec 7 09:35:57 2004 From: python at rcn.com (Raymond Hettinger) Date: Tue Dec 7 09:38:42 2004 Subject: [Python-Dev] Rewriting PEP4 In-Reply-To: <41B566C5.2040102@v.loewis.de> Message-ID: <000e01c4dc37$c673b400$e841fea9@oemcomputer> > > Instead, there should be a clear decision to deprecate or not. If that > > entails a comp.lang.python.announce notice and comment period, so be it. > > Once a decision is made, document it, add a warning, and wait. > > Ok, that might be reasonable. Please word the PEP accordingly. > > Once a couple versions pass, some useful action needs to take place. If > > the code is left in-place and the doc index is just re-arranged, then we > > would have been better off not doing anything at all. > > I disagree. The primary purpose (move developers to the alternative > approach) is achieved as soon as the warning is added, and the > documentation is deleted. Whether the module is actually deleted is > of minor importance: it costs nothing to continue including it; disk > space is cheap. That is reasonable. Please make that goal/assumption explicit in the PEP. > > The questions of dates was somewhat minor. I was unclear on the > > implication of, for example, "remove on 15Jan2005". Since Py2.5 won't > > go out for at least a year, does that mean that the work can't be done > > now while I have time instead of later when I don't. The only time the > > removal becomes visible is when a new version goes out the door. > > You could remove it now, but if we release Py2.5 in two months, you > would have to put it back in. So if you don't have time then, maybe > somebody else will. If nobody has time to remove the module, the next > release will ship with it, again - no big issue. Okay. Again, please say that in the PEP. > > Further, if a version is going to have something removed, we should do > > it at the outset rather than just before a release goes out the door (to > > allow for early surfacing of any issues). > > That is true. That should also be recommended in the PEP. > > If we really don't care whether it gets done, then we shouldn't bother > > in the first place. > > What do you mean by "bother"? Not put the deprecation warning in? We > do want users to move to the new approach of doing something. However, > two version are just not enough - it may take 10 or 20 years to remove > a widely used feature (e.g. the string module). That it will take so > long does not mean we should not start the process *now* - if we start > the process in five years, it will *still* take 10 or 20 years. Just > be patient. I see. That also may useful to include in the motivation section of the PEP. With these adjustments, I think the PEP will be fine. Also, be sure get rid of the mid-version undo (see Anthony's note) and to address the situation with Python books. I look forward to a revised draft. Raymond From ncoghlan at iinet.net.au Tue Dec 7 09:39:06 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Tue Dec 7 09:39:12 2004 Subject: [Python-Dev] Re: long number multiplication In-Reply-To: References: Message-ID: <41B56C2A.3030609@iinet.net.au> Christopher A. Craig wrote: >>i needed to implement this myself and was thinking of storing the digits >>of an integer in a list. > > > That's sort of what Python does except the "digits" are 15 bits, > not base 10. Doing it in base 10 would be a huge pain because of the > problems with base 10->base 2 conversion. > Indeed. Python's Decimal data type stores the digits in a list for ease of power of 10 multiplication & rounding (the precision rules means a *lot* of that goes on). However, it converts that list of digits to a long integer in order to do addition, multiplication or division. Cheers, Nick. -- Nick Coghlan | ncoghlan@email.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net From ncoghlan at iinet.net.au Tue Dec 7 10:46:41 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Tue Dec 7 10:46:49 2004 Subject: [Python-Dev] Rewriting PEP4 In-Reply-To: <000e01c4dc37$c673b400$e841fea9@oemcomputer> References: <000e01c4dc37$c673b400$e841fea9@oemcomputer> Message-ID: <41B57C01.9050404@iinet.net.au> One other question occurred to me for "deprecate X in favour of Y" situations - should the deprecation warning added to X point developers towards Y? Also, should the PEP include example wordings for deprecation warnings, rather than being completely freeform? Martin listed some information that should be included, but it seems an example or two would make it easy to get right. E.g.: Unmaintained, with a maintained alternative: "Due to lack of maintenance, this module has been deprecated in favour of module Y and will be removed in Python 2.6 (see PEP 4 for information on the deprecation process)" Security problems, no alternative: "Due to security concerns, this module has been deprecated and will be removed in Python 2.6 (see PEP 4 for information on the deprecation process)" Cheers, Nick. -- Nick Coghlan | ncoghlan@email.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net From ncoghlan at iinet.net.au Tue Dec 7 10:51:01 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Tue Dec 7 10:51:10 2004 Subject: [Python-Dev] Re: long number multiplication In-Reply-To: <41B56C2A.3030609@iinet.net.au> References: <41B56C2A.3030609@iinet.net.au> Message-ID: <41B57D05.1090509@iinet.net.au> Gah, sorry folks. I really ought to pay more attention to that send line. Cheers, Nick. -- Nick Coghlan | ncoghlan@email.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net From barry at python.org Tue Dec 7 15:06:10 2004 From: barry at python.org (Barry Warsaw) Date: Tue Dec 7 15:06:15 2004 Subject: [Python-Dev] Rewriting PEP4 In-Reply-To: <000801c4dc34$7949b420$e841fea9@oemcomputer> References: <000801c4dc34$7949b420$e841fea9@oemcomputer> Message-ID: <1102428369.25008.31.camel@geddy.wooz.org> On Tue, 2004-12-07 at 03:12, Raymond Hettinger wrote: > I'm concerned about the old email modules. They haven't been deprecated > and may stay around for a good while. However, some SF requests have > been passed over on the basis that the modules are supplanted and will > be phased out. I don't think it is good to leave active modules as > orphans. I would like to add deprecation warnings to some of the old email-related modules for Python 2.5, but I doubt we can remove any of them until Python 3.0 (or 2014, whichever comes first ). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041207/2226a04a/attachment.pgp From fdrake at acm.org Tue Dec 7 15:47:37 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue Dec 7 15:48:02 2004 Subject: [Python-Dev] Rewriting PEP4 In-Reply-To: <41B57C01.9050404@iinet.net.au> References: <000e01c4dc37$c673b400$e841fea9@oemcomputer> <41B57C01.9050404@iinet.net.au> Message-ID: <200412070947.38173.fdrake@acm.org> On Tuesday 07 December 2004 04:46 am, Nick Coghlan wrote: > One other question occurred to me for "deprecate X in favour of Y" > situations - should the deprecation warning added to X point developers > towards Y? Not sure about the warning, but the documentation certainly should. When the mapping from the old module to the new is not obvious and trivial, sufficient information should be included in the documentation for the old module to make it fairly easy to perform the conversion. (Placing this in the docs for the old module ensures that it goes away when the documentation for the old module goes away, and the docs for the new module need never be loaded with information about the old module.) > Also, should the PEP include example wordings for deprecation warnings, > rather than being completely freeform? Martin listed some information that > should be included, but it seems an example or two would make it easy to > get right. Examples would be good, though specific wording should not be proscribed. -Fred -- Fred L. Drake, Jr. From mwh at python.net Tue Dec 7 15:54:50 2004 From: mwh at python.net (Michael Hudson) Date: Tue Dec 7 15:54:53 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib/test test_shutil.py, 1.10, 1.11 In-Reply-To: (jlgijsbers@users.sourceforge.net's message of "Mon, 06 Dec 2004 12:50:17 -0800") References: Message-ID: <2m4qiyqhtx.fsf@starship.python.net> jlgijsbers@users.sourceforge.net writes: > Update of /cvsroot/python/python/dist/src/Lib/test > In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv20531 > > Modified Files: > test_shutil.py > Log Message: > SF bug #1076467: don't run test_on_error as root, as the permission > errors don't get provoked that way. Also add a bunch of cross-references > to bugs. > > > Index: test_shutil.py > =================================================================== > RCS file: /cvsroot/python/python/dist/src/Lib/test/test_shutil.py,v > retrieving revision 1.10 > retrieving revision 1.11 > diff -u -d -r1.10 -r1.11 > --- test_shutil.py 23 Nov 2004 09:27:27 -0000 1.10 > +++ test_shutil.py 6 Dec 2004 20:50:15 -0000 1.11 > @@ -16,7 +16,10 @@ > filename = tempfile.mktemp() > self.assertRaises(OSError, shutil.rmtree, filename) > > - if hasattr(os, 'chmod') and sys.platform[:6] != 'cygwin': > + # See bug #1071513 for why we don't run this on cygwin > + # and bug #1076467 for why we don't run this as root. > + if (hasattr(os, 'chmod') and sys.platform[:6] != 'cygwin' > + and os.getenv('USER') != 'root'): Is that really the best way to check for root? It may be, I guess, but I'd have expected os.geteuid() to be more reliable... Or is it os.getuid()? I always get confused by these functions... Cheers, mwh -- CDATA is not an integration strategy. -- from Twisted.Quotes From lists at andreas-jung.com Tue Dec 7 18:24:31 2004 From: lists at andreas-jung.com (Andreas Jung) Date: Tue Dec 7 18:24:37 2004 Subject: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL Message-ID: <9DFB75B2DC037BC6AAD05F9E@[192.168.0.102]> While using Zope 2.7 with Python 2.4 we discovered some strange behaviour of the security machinery. I could track this down to some Zope code in cAccessControl.c where an Unauthorized exception is raised because of a call to PyInt_FromLong(1) which returns NULL. What could be the reason that such a "stupid" call return NULL in a reproducable way? -aj From mwh at python.net Tue Dec 7 18:28:45 2004 From: mwh at python.net (Michael Hudson) Date: Tue Dec 7 18:28:47 2004 Subject: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL In-Reply-To: <9DFB75B2DC037BC6AAD05F9E@[192.168.0.102]> (Andreas Jung's message of "Tue, 07 Dec 2004 18:24:31 +0100") References: <9DFB75B2DC037BC6AAD05F9E@[192.168.0.102]> Message-ID: <2mzn0qow4y.fsf@starship.python.net> Andreas Jung writes: > While using Zope 2.7 with Python 2.4 we discovered some strange > behaviour of the security machinery. > I could track this down to some Zope code in cAccessControl.c where an > Unauthorized exception is > raised because of a call to PyInt_FromLong(1) which returns NULL. What > could be the reason that > such a "stupid" call return NULL in a reproducable way? A memory scribble somewhere else? Possibly a DECREF too many somewhere? Is an exception set? Have you tried a debug build? Etc. Cheers, mwh -- All obscurity will buy you is time enough to contract venereal diseases. -- Tim Peters, python-dev From jim at zope.com Tue Dec 7 18:30:19 2004 From: jim at zope.com (Jim Fulton) Date: Tue Dec 7 18:30:24 2004 Subject: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL In-Reply-To: <9DFB75B2DC037BC6AAD05F9E@[192.168.0.102]> References: <9DFB75B2DC037BC6AAD05F9E@[192.168.0.102]> Message-ID: <41B5E8AB.8060405@zope.com> Andreas Jung wrote: > While using Zope 2.7 with Python 2.4 we discovered some strange > behaviour of the security machinery. > I could track this down to some Zope code in cAccessControl.c where an > Unauthorized exception is > raised because of a call to PyInt_FromLong(1) which returns NULL. What > could be the reason that > such a "stupid" call return NULL in a reproducable way? Ugh. Part of the problem is that all of those calls are unchecked, Dang us. If they were checked, then, who knows, we might have gotten informative exceptions. I'd say the first step should be to add checks. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From tim.peters at gmail.com Tue Dec 7 18:58:12 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Dec 7 18:59:03 2004 Subject: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL In-Reply-To: <9DFB75B2DC037BC6AAD05F9E@192.168.0.102> References: <9DFB75B2DC037BC6AAD05F9E@192.168.0.102> Message-ID: <1f7befae041207095831b00ae0@mail.gmail.com> [Andreas Jung] > While using Zope 2.7 Do you mean 2.7, or do you mean 2.7.3, or ...? > with Python 2.4 we discovered some strange behaviour > of the security machinery. > I could track this down to some Zope code in cAccessControl.c > where an Unauthorized exception is raised because of a call to > PyInt_FromLong(1) which returns NULL. What could be the > reason that such a "stupid" call return NULL in a reproducable > way? Any C function that returns a Python object can return NULL if malloc() says there's not enough memory to create a new object. PyInt_FromLong() actually allocates about 1KB at a time, and will return NULL if malloc() can't find that much. OTOH, 1KB isn't big, and PyInt_FromLong(1) specifically should be returning a shared reference to a pre-existing PyIntObject (a number of small integer objects are constructed at Python initialization time, and PyInt_FromLong() returns references to them instead of allocating new memory). So PyInt_FromLong(1) should have taken this path: v = small_ints[ival + NSMALLNEGINTS]; But if a wild store had stuffed NULL into 1's slot in the small_ints vector, the *next* line would have blown up with a NULL-pointer dereference before PyInt_FromLong() could have returned: Py_INCREF(v); return (PyObject *) v; So, in all, it appears impossible for PyInt_FromLong(1) to return NULL. If it's reproducible, run it under a debugger and step into the errant PyInt_FromLong(1) to see what's happening? Could be a compiler optimization bug (while rare, they have happened in Python). From tim.peters at gmail.com Tue Dec 7 19:02:51 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Dec 7 19:03:52 2004 Subject: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL In-Reply-To: <41B5E8AB.8060405@zope.com> References: <9DFB75B2DC037BC6AAD05F9E@192.168.0.102> <41B5E8AB.8060405@zope.com> Message-ID: <1f7befae04120710026748686c@mail.gmail.com> [Jim Fulton] > Ugh. Part of the problem is that all of those calls are unchecked, > Dang us. If they were checked, then, who knows, we might have > gotten informative exceptions. They certainly "should" be checked, but as a pragmatic matter PyInt_FromLong(1) can't fail -- Python allocates an int object for 1 (and for about 100 other popular little integers) when it starts up, and PyInt_FromLong() just returns a new reference to these pre-existing objects whenever possible. So, wrt: > I'd say the first step should be to add checks that's probably not going to help. I'd make it the fourth thing . From jim at zope.com Tue Dec 7 19:06:54 2004 From: jim at zope.com (Jim Fulton) Date: Tue Dec 7 19:07:00 2004 Subject: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL In-Reply-To: <1f7befae04120710026748686c@mail.gmail.com> References: <9DFB75B2DC037BC6AAD05F9E@192.168.0.102> <41B5E8AB.8060405@zope.com> <1f7befae04120710026748686c@mail.gmail.com> Message-ID: <41B5F13E.5080901@zope.com> Tim Peters wrote: > [Jim Fulton] > >>Ugh. Part of the problem is that all of those calls are unchecked, >>Dang us. If they were checked, then, who knows, we might have >>gotten informative exceptions. > > > They certainly "should" be checked, but as a pragmatic matter > PyInt_FromLong(1) can't fail -- Python allocates an int object for 1 > (and for about 100 other popular little integers) when it starts up, > and PyInt_FromLong() just returns a new reference to these > pre-existing objects whenever possible. I know. I'm sure that's why we don't bother. But, obviously, it can fail. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From tim.peters at gmail.com Tue Dec 7 19:17:19 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Dec 7 19:18:30 2004 Subject: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL In-Reply-To: <41B5F13E.5080901@zope.com> References: <9DFB75B2DC037BC6AAD05F9E@192.168.0.102> <41B5E8AB.8060405@zope.com> <1f7befae04120710026748686c@mail.gmail.com> <41B5F13E.5080901@zope.com> Message-ID: <1f7befae04120710175ae5882@mail.gmail.com> [Tim] >> ... but as a pragmatic matter PyInt_FromLong(1) can't fail -- [Jim] > I know. I'm sure that's why we don't bother. But, obviously, > it can fail. I disagree -- it's not obvious at all. Looking at the code, it's far more likely that Andreas misunderstood the cause of the failure than that PyInt_FromLong(1) actually returned NULL. If it did return NULL, then it's got to be something as rare as bad code generation (for reasons I explained earlier), or a non-standard compilation that fiddled the NSMALLPOSINTS and/or NSMALLNEGINTS #defines to insane values. This is the entire expected path in PyInt_FromLong(1): PyObject * PyInt_FromLong(long ival) { register PyIntObject *v; #if NSMALLNEGINTS + NSMALLPOSINTS > 0 if (-NSMALLNEGINTS <= ival && ival < NSMALLPOSINTS) { v = small_ints[ival + NSMALLNEGINTS]; Py_INCREF(v); #ifdef COUNT_ALLOCS if (ival >= 0) quick_int_allocs++; else quick_neg_int_allocs++; #endif return (PyObject *) v; } #endif It's not possible for that to return NULL -- even if small_ints[] got corrupted, so that v == NULL, the Py_INCREF(v) would have blown up before the function could have returned. From amk at amk.ca Tue Dec 7 19:30:20 2004 From: amk at amk.ca (A.M. Kuchling) Date: Tue Dec 7 19:30:41 2004 Subject: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL In-Reply-To: <1f7befae04120710026748686c@mail.gmail.com> References: <9DFB75B2DC037BC6AAD05F9E@192.168.0.102> <41B5E8AB.8060405@zope.com> <1f7befae04120710026748686c@mail.gmail.com> Message-ID: <20041207183020.GA22351@rogue.amk.ca> On Tue, Dec 07, 2004 at 01:02:51PM -0500, Tim Peters wrote: > pre-existing objects whenever possible. So, wrt: > > I'd say the first step should be to add checks > that's probably not going to help. I'd make it the fourth thing . Is it possible that some other Python API call is raising an exception, but a NULL return isn't being checked for, and PyInt_FromLong() happens to be the first bit of code that does 'if (PyErr_Occurred())'? Though from a quick glance at PyInt_FromLong() and the macros it uses, I don't see any references to PyErr_Occurred()... --amk From lists at andreas-jung.com Tue Dec 7 19:34:32 2004 From: lists at andreas-jung.com (Andreas Jung) Date: Tue Dec 7 19:34:37 2004 Subject: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL In-Reply-To: <1f7befae04120710175ae5882@mail.gmail.com> References: <9DFB75B2DC037BC6AAD05F9E@192.168.0.102> <41B5E8AB.8060405@zope.com> <1f7befae04120710026748686c@mail.gmail.com> <41B5F13E.5080901@zope.com> <1f7befae04120710175ae5882@mail.gmail.com> Message-ID: <47517AA25D790FF4C7D2A450@[192.168.0.102]> Sorry, false alarm :-( There assignment of the NULL occurs in the if-clause of the corresponding code (I have overseen the ASSIGN call): if (! PyInt_Check(p)) { if (PyDict_Check(p)) { if (PyString_Check(name) || PyUnicode_Check(name)) { ASSIGN(p, PyObject_GetItem(p, name)); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ if (p == NULL) { puts("PyObject returned NULL"); PyErr_Clear(); } } else p = PyInt_FromLong((long)1); ...doing some further investigations on that. -aj From tim.peters at gmail.com Tue Dec 7 19:40:01 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Dec 7 19:40:16 2004 Subject: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL In-Reply-To: <20041207183020.GA22351@rogue.amk.ca> References: <9DFB75B2DC037BC6AAD05F9E@192.168.0.102> <41B5E8AB.8060405@zope.com> <1f7befae04120710026748686c@mail.gmail.com> <20041207183020.GA22351@rogue.amk.ca> Message-ID: <1f7befae0412071040346940c8@mail.gmail.com> [A.M. Kuchling] > Is it possible that some other Python API call is raising an > exception, but a NULL return isn't being checked for, and > PyInt_FromLong() happens to be the first bit of code that does 'if > (PyErr_Occurred())'? Though from a quick glance at > PyInt_FromLong() and the macros it uses, I don't see any > references to PyErr_Occurred()... A long stare wouldn't uncover any either . This isn't it. That 1 is passed as an argument is important too, which cuts out all but the simplest path thru this code. The (current Zope 2.7 branch) source for cAccessControl.c contains six calls to PyInt_FromLong(). Five of them pass the literal 1. The other passes a variable with value -1, 0, or 1. So regardless of which of these Andreas is talking about, it's going thru PyInt_FromLong's fast path. From tim.peters at gmail.com Tue Dec 7 20:10:20 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Dec 7 20:10:54 2004 Subject: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL In-Reply-To: <47517AA25D790FF4C7D2A450@192.168.0.102> References: <9DFB75B2DC037BC6AAD05F9E@192.168.0.102> <41B5E8AB.8060405@zope.com> <1f7befae04120710026748686c@mail.gmail.com> <41B5F13E.5080901@zope.com> <1f7befae04120710175ae5882@mail.gmail.com> <47517AA25D790FF4C7D2A450@192.168.0.102> Message-ID: <1f7befae0412071110707ff1db@mail.gmail.com> [Andreas Jung] > Sorry, false alarm :-( There assignment of the NULL occurs in the > if-clause of the corresponding code (I have overseen the ASSIGN > call): Thanks for the followup! > if (! PyInt_Check(p)) > { > if (PyDict_Check(p)) > { > if (PyString_Check(name) || > PyUnicode_Check(name)) > { > ASSIGN(p, PyObject_GetItem(p, name)); > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > if (p == NULL) { > puts("PyObject returned NULL"); > PyErr_Clear(); > } > } > else > p = PyInt_FromLong((long)1); > > ...doing some further investigations on that. I note that all of this is nested inside another "if (p) {...}" block. That implies the "p = PyInt_FromLong((long)1);" line is at least a memory leak: it overwrites p without decref'ing p first. From jim at zope.com Tue Dec 7 20:19:43 2004 From: jim at zope.com (Jim Fulton) Date: Tue Dec 7 20:19:46 2004 Subject: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL In-Reply-To: <1f7befae0412071110707ff1db@mail.gmail.com> References: <9DFB75B2DC037BC6AAD05F9E@192.168.0.102> <41B5E8AB.8060405@zope.com> <1f7befae04120710026748686c@mail.gmail.com> <41B5F13E.5080901@zope.com> <1f7befae04120710175ae5882@mail.gmail.com> <47517AA25D790FF4C7D2A450@192.168.0.102> <1f7befae0412071110707ff1db@mail.gmail.com> Message-ID: <41B6024F.4070201@zope.com> Tim Peters wrote: > [Andreas Jung] > >>Sorry, false alarm :-( There assignment of the NULL occurs in the >>if-clause of the corresponding code (I have overseen the ASSIGN >>call): > > > Thanks for the followup! > > >> if (! PyInt_Check(p)) >> { >> if (PyDict_Check(p)) >> { >> if (PyString_Check(name) || >>PyUnicode_Check(name)) >> { >> ASSIGN(p, PyObject_GetItem(p, name)); >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> >> if (p == NULL) { >> puts("PyObject returned NULL"); >> PyErr_Clear(); >> } >> } >> else >> p = PyInt_FromLong((long)1); >> >>...doing some further investigations on that. > > > I note that all of this is nested inside another "if (p) {...}" block. > That implies the "p = PyInt_FromLong((long)1);" line is at least a > memory leak: it overwrites p without decref'ing p first. The ASSIGN macro DECREFs it's first argument if it is non-NULL. It loosly models a Python assignment, assuming that it owns the reference to the second argument. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From tim.peters at gmail.com Tue Dec 7 21:25:06 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Dec 7 21:25:19 2004 Subject: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL In-Reply-To: <41B6024F.4070201@zope.com> References: <9DFB75B2DC037BC6AAD05F9E@192.168.0.102> <41B5E8AB.8060405@zope.com> <1f7befae04120710026748686c@mail.gmail.com> <41B5F13E.5080901@zope.com> <1f7befae04120710175ae5882@mail.gmail.com> <47517AA25D790FF4C7D2A450@192.168.0.102> <1f7befae0412071110707ff1db@mail.gmail.com> <41B6024F.4070201@zope.com> Message-ID: <1f7befae04120712255df4861e@mail.gmail.com> >>> if (! PyInt_Check(p)) >>> { >>> if (PyDict_Check(p)) >>> { >>> if (PyString_Check(name) || PyUnicode_Check(name)) >>> { >>> ASSIGN(p, PyObject_GetItem(p, name)); >>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>> >>> if (p == NULL) { >>> puts("PyObject returned NULL"); >>> PyErr_Clear(); >>> } >>> } >>> else >>> p = PyInt_FromLong((long)1); >>> [Tim] >> I note that all of this is nested inside another "if (p) {...}" block. >> That implies the "p = PyInt_FromLong((long)1);" line is at least a >> memory leak: it overwrites p without decref'ing p first. [Jim] > The ASSIGN macro DECREFs it's first argument if it is non-NULL. > > It loosly models a Python assignment, assuming that it owns the > reference to the second argument. ASSIGN isn't executed on the path in question. I really can't follow nesting with that indentation style. Reformatting in Python style makes it obvious to me: if (p) { if (! PyInt_Check(p)) { if (PyDict_Check(p)) { if (PyString_Check(name) || PyUnicode_Check(name)) { ASSIGN(p, PyObject_GetItem(p, name)); if (p == NULL) PyErr_Clear(); } else p = PyInt_FromLong(1); } else { ASSIGN(p, callfunction2(p, name, value)); if (p == NULL) goto err; } } } "p = PyInt_FromLong(1)" is in an ``else`` block. The only ASSIGN before it is in that ``else`` block's disjoint ``if`` block. From jlg at dds.nl Tue Dec 7 23:52:45 2004 From: jlg at dds.nl (Johannes Gijsbers) Date: Tue Dec 7 23:49:55 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib/test test_shutil.py, 1.10, 1.11 In-Reply-To: <2m4qiyqhtx.fsf@starship.python.net> References: <2m4qiyqhtx.fsf@starship.python.net> Message-ID: <20041207225245.GB9734@authsmtp.dds.nl> On Tue, Dec 07, 2004 at 02:54:50PM +0000, Michael Hudson wrote: > Is that really the best way to check for root? It may be, I guess, > but I'd have expected os.geteuid() to be more reliable... Ah yes, I suppose it is. I'll change it tomorrow night if no Unix-guru steps up with an alternative solution. > Or is it os.getuid()? I always get confused by these functions... Based on my reading of the difference between geteuid and getuid, I'd go for geteuid. Cheers, Johannes From tjreedy at udel.edu Wed Dec 8 02:41:45 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Wed Dec 8 02:41:41 2004 Subject: [Python-Dev] Re: Rewriting PEP4 References: <002401c4dbf5$b847cc00$f301a044@oemcomputer> <41B556FE.20504@v.loewis.de> Message-ID: ""Martin v. Löwis"" wrote in message news:41B556FE.20504@v.loewis.de... As a (currently) casual user of Python, this is my view of the standard library dilemma and its solution: 1. It is too small: I may someday want to use a module not yet added. 2. It is too big: I cannot keep everything in mind at once and cannot remember, without referring back to the lib manual or pydev, which modules are currently or slated to become deprecated. 3. I do not wish to have other people's code broken,or books made obsolete, without sufficient reason (ie, the code is actively dangerous versus merely broken or superceded). My preferred solution is to apply the 'out of sight, out of mind' principle. 1. Move obsolete modules to a separate directory (lib_old sounds fine) and put that directory in pythonpath. When I ran Python (1.3) from a 20 meg disk, I would have preferred complete removal, but with 60+ gigs, the small extra space is no longer an issue. If someone needs the space or wants to guarantee non-use of old modules, deleting lib_old is easy enough. 2. Remove the docs for obsolete modules from the main part of the current lib reference. Sub-choices for legacy chapters are either complete removal or segregation into a separate document or apppendix to current one. Terry J. Reedy From gvanrossum at gmail.com Wed Dec 8 23:18:48 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed Dec 8 23:18:50 2004 Subject: [Python-Dev] 2.4 news reaches interesting places Message-ID: I was pleasantly surprised to find a pointer to this article in a news digest that the ACM emails me regularly (ACM TechNews). http://gcn.com/vol1_no1/daily-updates/28026-1.html One thing that bugs me: the article says 3 or 4 times that Python is slow, each time with a refutation ("but it's so flexible", "but it's fast enough") but still, they sure seem to harp on the point. This is a PR issue that Python needs to fight -- any ideas? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Wed Dec 8 23:39:31 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Dec 8 23:39:34 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: Message-ID: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> At 02:18 PM 12/8/04 -0800, Guido van Rossum wrote: >I was pleasantly surprised to find a pointer to this article in a news >digest that the ACM emails me regularly (ACM TechNews). > >http://gcn.com/vol1_no1/daily-updates/28026-1.html > >One thing that bugs me: the article says 3 or 4 times that Python is >slow, each time with a refutation ("but it's so flexible", "but it's >fast enough") but still, they sure seem to harp on the point. This is >a PR issue that Python needs to fight -- any ideas? The only thing that will fix the PR issue is to have a Python compiler distributed as part of the language. It doesn't matter if it doesn't support the full generality of Python, or even if it doesn't speed many operations up much. The only real requirements are that it can be used to produce "native" executables, and that it be an official part of the language, not a separately-distributed tool like Psyco or Pyrex. Then, it will perhaps be a sufficient "security blanket" to stop people FUDding about it. I imagine you could speed up the Python interpreter until it's faster than half the Java JIT's out there, and people will still ask, "But can I compile to an .exe?" On the other hand, if you add a compiler, we'll see articles like the above talking about how Python can now be compiled and so therefore it's suitable for all kinds of things it wasn't before. :) Of course, it would be *really* useful if the compiler were coupled with optional type declarations for Python-the-language, because then we could ultimately dispense with the current syntax warts of Pyrex (and the runtime/distribution warts of ctypes) for interfacing with C. But I don't think that having the compiler actually be useful is a prerequisite for solving the PR issue. :) From pythondev at bitfurnace.com Wed Dec 8 23:59:05 2004 From: pythondev at bitfurnace.com (damien morton) Date: Wed Dec 8 23:59:11 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: References: Message-ID: <41B78739.1080200@bitfurnace.com> Guido van Rossum wrote: > I was pleasantly surprised to find a pointer to this article in a news > digest that the ACM emails me regularly (ACM TechNews). > > http://gcn.com/vol1_no1/daily-updates/28026-1.html > > One thing that bugs me: the article says 3 or 4 times that Python is > slow, each time with a refutation ("but it's so flexible", "but it's > fast enough") but still, they sure seem to harp on the point. This is > a PR issue that Python needs to fight -- any ideas? In public documents - documentation, release notes, etc etc.: Remove all references to the performance of Python. From now on, Python has always been extremely fast, has no need to get any faster, and any speed increases are of so little importance that they barely rate a mention (no percentage increases mentioned in release notes). If you must mention performance, do it in terms of the use of C modules such as numarray, and for benchmarks that other languages cant compete in. Write a benchmark which uses common C extensions, that Java cant compete with. Replace all references to a 'Python interpreter' with 'Python Virtual Machine' Ensure that the terms 'Python compiler' and 'compiled python' are liberally sprinkled around. Start emphasising the compilation step from python source to python bytecodes instead of making it transparent. From sxanth at ceid.upatras.gr Thu Dec 9 00:25:20 2004 From: sxanth at ceid.upatras.gr (Stelios Xanthakis) Date: Thu Dec 9 00:25:24 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> References: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> Message-ID: > The only thing that will fix the PR issue is to have a Python compiler > distributed as part of the language. It doesn't matter if it doesn't > support the full generality of Python, or even if it doesn't speed many > operations up much. The only real requirements are that it can be used > to produce "native" executables I don't hink it's a matter of native executables. Hopefully all the real algorithms are already in native executables as builtins (dictionary lookup, etc). And generally, one can wrap all the tough code in C and take advantage of the flexibility of python at a higher level. pygame is the best example of this! Basic stuff from complexity theory: the real complex parts are few and isolated in C. For the other we can take advantage of a higher level dynamic language. Stelios From nas at arctrix.com Thu Dec 9 00:29:09 2004 From: nas at arctrix.com (Neil Schemenauer) Date: Thu Dec 9 00:29:13 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: References: Message-ID: <20041208232909.GA6354@mems-exchange.org> On Wed, Dec 08, 2004 at 02:18:48PM -0800, Guido van Rossum wrote: > This is a PR issue that Python needs to fight -- any ideas? I'm not good at PR so I will continue to try to make it faster. In my copious free time I plan to: * finish the AST compiler (no performance benefit but makes other things easier) * optimize module namespaces as per the PEPs (whose numbers I cannot recall) * optimize function/method invocation (we must be able to do something better here, they are currently very expensive). Fredrik has mentioned "call-site caching". * write a new, register based VM From hpk at trillke.net Thu Dec 9 01:39:38 2004 From: hpk at trillke.net (holger krekel) Date: Thu Dec 9 01:39:40 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: References: Message-ID: <20041209003938.GE4708@solar.trillke.net> [Guido van Rossum Wed, Dec 08, 2004 at 02:18:48PM -0800] > I was pleasantly surprised to find a pointer to this article in a news > digest that the ACM emails me regularly (ACM TechNews). > > http://gcn.com/vol1_no1/daily-updates/28026-1.html > > One thing that bugs me: the article says 3 or 4 times that Python is > slow, each time with a refutation ("but it's so flexible", "but it's > fast enough") but still, they sure seem to harp on the point. This is > a PR issue that Python needs to fight -- any ideas? What about doing a survey on c.l.py asking questions like: what do you find runs slowly in Python that should run faster? Could you help with some concrete - preferably real life examples with speed problems?". If python-dev takes the time to seriously debate (and debunk :-) upcoming code and suggestions then such a process could be very useful for all sides and also for PR purposes. IMO the biggest PR problem is that python programmers themselves [*] tend to say that Python is slow and it's interesting to find out why they think so, document and discuss the "answers" if any. I am not saying that such questioning/discussion doesn't already happen sometimes. But doing a survey in a more systematic way might let us find out how pervasive the perception of "Python is too slow" really is. Maybe it turns out that not many people have concrete problems to offer? Anyway, this would probably also be interesting for the various alternative Python implementations currently in the works. just an idea, holger [*] For example, something i stumbled upon today: http://unununium.org/articles/languages where it states (without providing any details!): But what about that fast system? Python isn't a slow language; it just has a slow implementation. There are many projects underway to correct this situation: Psyco, PyPy, Starkiller, IronPython, and Parrotcode are among them. It's likely these projects will be nearing completion when the time comes for Unununium to look at optimizations. From pje at telecommunity.com Thu Dec 9 02:30:02 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Dec 9 02:30:11 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: References: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20041208201940.0210b270@mail.telecommunity.com> At 01:25 AM 12/9/04 +0200, Stelios Xanthakis wrote: >>The only thing that will fix the PR issue is to have a Python compiler >>distributed as part of the language. It doesn't matter if it doesn't >>support the full generality of Python, or even if it doesn't speed many >>operations up much. The only real requirements are that it can be used >>to produce "native" executables > >I don't hink it's a matter of native executables. As I explained later in that message, "native" simply means, "has an .exe extension on Windows". For PR purposes, it would suffice to bundle py2exe with Python 2.5 and say that Python "now includes a compiler to produce executable files". This will then be picked up and misinterpreted by the trade press in exactly the same way that the article Guido cited picked up and misinterpreted what was said about 2.4. If you read the article carefully, you will notice that the author translated "we rewrote a few modules in C" into "we made Python faster by switching to C". If you ask anybody what language is faster, language X or C, most everybody will answer "C", regardless of what X is (unless it's assembly, of course). All of the discussion about *actually* improving Python's performance is moot for PR purposes. Public perception is not swayed by mere facts (as one might cynically observe of the U.S. political system). If the goal is to achieve a PR win, the important thing is to pick a meme that's capable of succeeding, and stay "on message" with it. The *only* meme that's organically capable of trumping "Python is slow because it's interpreted" is "Python is compiled now". Me, I don't really care one way or the other. I used to sell software I wrote in TRS-80 Basic, so Python's performance is fine for me, and I'm certainly not a compiler bigot. I'm just responding to Guido's inquiry about what might work to increase Python's *perceived* speed in popular consciousness, not its actual speed. From andrew at indranet.co.nz Thu Dec 9 03:32:10 2004 From: andrew at indranet.co.nz (Andrew McGregor) Date: Thu Dec 9 03:32:28 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: References: Message-ID: <86F33E75-498A-11D9-BB31-000D93C603C0@indranet.co.nz> Well, for a lot of applications for Python, the performance that really counts is time from no code but a pile of data to code and processed data. Python shines at that because nearly always the time to write the code dominates, so it doesn't matter what the run time is. I wrote a little tool to translate a bunch of free data into scenery for the X-Plane flight simulator. If I'd tried to do it in C, I'd still be debugging it, whereas I released it and the scenery I was building six months ago. The run time of the C version would be, I speculate, about 5 times faster than the Python (given that psyco speeds it up about 8 times, and I would not have been so fancy with the algorithms in C). But a 5x improvement on a 24 hour runtime is not 6 months of improvement. The other thing we can do is finish the portable backend for psyco and make it a standard module. Then Python won't be slow, it will be compiled, and py2exe will be able to make a single-file executable. Andrew On 9/12/2004, at 11:18 AM, Guido van Rossum wrote: > I was pleasantly surprised to find a pointer to this article in a news > digest that the ACM emails me regularly (ACM TechNews). > > http://gcn.com/vol1_no1/daily-updates/28026-1.html > > One thing that bugs me: the article says 3 or 4 times that Python is > slow, each time with a refutation ("but it's so flexible", "but it's > fast enough") but still, they sure seem to harp on the point. This is > a PR issue that Python needs to fight -- any ideas? > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ > andrew%40indranet.co.nz > > From ilya at bluefir.net Thu Dec 9 04:57:25 2004 From: ilya at bluefir.net (Ilya Sandler) Date: Thu Dec 9 04:55:08 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: References: Message-ID: On Wed, 8 Dec 2004, Guido van Rossum wrote: > One thing that bugs me: the article says 3 or 4 times that Python is > slow, each time with a refutation ("but it's so flexible", "but it's > fast enough") but still, they sure seem to harp on the point. This is > a PR issue that Python needs to fight -- any ideas? > On PR side of the issue one idea would be to avoid using the words interpreted/intepreter less and use the words compiler/byte-compilation/virtual machine instead... On non-PR side of the issue (I do think that python slowness is a real problem for at least some people/domains): would bundling of Psyco with Python be a good idea? Ilya From python at rcn.com Thu Dec 9 04:55:59 2004 From: python at rcn.com (Raymond Hettinger) Date: Thu Dec 9 05:00:16 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: Message-ID: <000b01c4dda2$fe7a87e0$e841fea9@oemcomputer> > One thing that bugs me: the article says 3 or 4 times that Python is > slow, each time with a refutation ("but it's so flexible", "but it's > fast enough") but still, they sure seem to harp on the point. This is > a PR issue that Python needs to fight -- any ideas? * Ask a prominent PSF member, ESR, to stop bagging on that point in his book (python is much faster since he wrote it and Moore's law has made it less of an issue over time). Py1.5.2 running on a 133Mhz 386 is not the same beast as well designed Py2.4 code running on a 3Ghz Pentium IV (also I would argue that many apps are computationally bound anyway). * Have python.org prominently feature an article of Python's use in high-performance environments. IIRC, somebody wrote a realtime voice over internet system and found that with good design, there was no speed issue. Also, the cellphone apps may make a good example. * Another prominent developer and PSF member wrote a much publicized, unfavorable log entry basically saying that he was unimpressed with Py2.4. While that log entry was subsequently revised, it did serve as anti-advocacy. Also, my request was denied for better presentation of performance related benchmarks in the widely read What's New document. Right now, it features somewhat unimpressive and misleading pystone results. Were we to show comparative pybench results 2.0 to 2.1 to 2.2 to 2.3 to 2.4, it would become clear that some of the performance concerns are out of date. Parrotbench and test_decimal also reveal some modest gains over 2.3. * A ZDNet reporter had setup a phone interview with me but did not follow through. I had planned to nip the issue in the bud by focusing on the Py2.4's vastly improved scalability: - The trend toward use of iterators, generators, generator expressions, and itertools scale-up well because of their superb memory performance and ability to save state. Apps using generators and genexps save all the time that was formerly lost to accessing instance variables. And apps keying off of itertools can sometimes completely avoid interpreter overhead. IOW, Py2.4 provides a strong toolset for writing clean, high-performance apps with a minimum of development time. - Behind the scenes, almost every important building block has either improved algorithms, memory optimizations, speed optimizations, and/or has been rewritten in C (while still preserving their pure Python equivalents). The bytecodes get optimized for speed and readability, and the eval-loop itself is tighter. Lists initialize, append, pop, and extend much faster and are more conservative with memory. List comps are up to twice as fast. Dictionary iteration is speedier. Sets, bisect, and heaps now have C implementations like other basic building blocks. All of the fifo queues in the library now use an O(1) algorithm instead of O(n) -- this improves scalability for everything from threading and queues to mutexs and asynchat. * Any PR effort should also emphasize that no usability trade-offs were made along the way. A number of features make Py2.4 easier to use than 1.5.6: list comps, genexps, generators, sets, nested scopes, int/long unification, decorators, etc. Some might argue that it takes a modicum effort to learn listcomps, generators, and decorators, but they cannot make any bones about the cleanliness, simplicity, beauty, and speed of the resulting code. Python will always lead in maintainability and development time. * Python's website has traditionally been self-centered, leaving others to have to make the case for their own products. Perhaps, it is time to change that. Those who really care about speed cannot make a balanced decision about Python without considering psyco, pyrex, numpy, and the like as part of the total development environment. * There is another angle that is difficult to present but a reality never-the-less. It is easy to write toy C / C++ code that is blazingly fast. However, when you add error checking, fault tolerance, interaction with real world, etc, the code gets slower. Also because that code is harder to write, it is somewhat common to use crummy algorithms (linear searches for example). I really liked Tim's comment that code using Python's dictionaries runs 10,000 times faster than anything written in any other language. That seems to get to the point across well enough. * A bit of good news is that the upcoming Python cookbook will make a strong case for Py2.4 being a tool of choice for those who care speed, space, maintainability, and development time. It is a nice, highly visible bit of advocacy. * One last thought. Articles will always pick on something. It's not really a terrible thing to have someone say the superb productivity gains come at the price of running slower than C. I would much rather hear that than have people bag on the docs or standard library or launch into a diatribe @decocrator destroying the beauty of the language. Raymond From mdehoon at ims.u-tokyo.ac.jp Thu Dec 9 05:47:01 2004 From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon) Date: Thu Dec 9 05:43:46 2004 Subject: [Python-Dev] PyOS_InputHook enhancement proposal Message-ID: <41B7D8C5.9000008@ims.u-tokyo.ac.jp> PyOS_InputHook is a pointer to a function that is called periodically (ten times per second) while Python is idle, for example, when waiting for a user command. Python C extension modules can set this pointer to a hook function defined in the extension module. For example, _tkinter.c makes use of PyOS_InputHook to get messages delivered to its widgets. A problem arises when two or more extension modules want to set PyOS_InputHook. For example, the scientific plotting package pygist needs PyOS_InputHook to get messages delivered to its graphics windows, and may therefore conflict with the Python GUI IDLE, which uses Tkinter. Chaining won't work, as it will fail when an extension module wants to remove its hook function. My suggestion is therefore to replace PyOS_InputHook by two functions PyOS_AddInputHook and PyOS_RemoveInputHook, and let Python keep track of which hooks are installed. This way, an extension module can add a hook function without having to worry about other extension modules trying to use the same hook. Any comments? Would I need to submit a PEP for this proposal? --Michiel, U Tokyo. -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon From python at rcn.com Thu Dec 9 05:43:49 2004 From: python at rcn.com (Raymond Hettinger) Date: Thu Dec 9 05:48:03 2004 Subject: [Python-Dev] The other Py2.4 issue Message-ID: <001401c4dda9$ad605680$e841fea9@oemcomputer> Acceptance for Py2.4 partially hinges on how quickly third party apps have their binaries updated. I wonder if there is anything we can do to help. Raymond -----Original Message----- From: image-sig-bounces@python.org [mailto:image-sig-bounces@python.org] On Behalf Of Tuure Laurinolli Sent: Tuesday, December 07, 2004 2:51 AM To: image-sig@python.org Subject: [Image-SIG] Re: Python 2.4 Spencer Ogden wrote: > I was wondering if there was a time frame for a windows binary for Py > 2.4. Is it possible to compile the source against 2.4 myself? There didn't seem to be any problems compiling it against 2.4. I managed to get some version up and running, but problems with the required shared libraries(I seem to have many, incompatible versions of jpeg around...) killed my enthusiasm and I returned to 2.3. _______________________________________________ Image-SIG maillist - Image-SIG@python.org http://mail.python.org/mailman/listinfo/image-sig From phd at mail2.phd.pp.ru Thu Dec 9 11:17:02 2004 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Thu Dec 9 11:17:05 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <5.1.1.6.0.20041208201940.0210b270@mail.telecommunity.com> References: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041208201940.0210b270@mail.telecommunity.com> Message-ID: <20041209101702.GA12506@phd.pp.ru> On Wed, Dec 08, 2004 at 08:30:02PM -0500, Phillip J. Eby wrote: > As I explained later in that message, "native" simply means, "has an .exe > extension on Windows". And very soon that strategy will backfire - people will start PRing "but those .exe's are nothing more than a python interpreter in disguise" which in my opnion is even worse. > All of the discussion about *actually* improving Python's performance is > moot for PR purposes. Hence we must stop spending our very valuable time thinking about PR and return to actually improving Python (not only performance). > If the goal is > to achieve a PR win, the important thing is to pick a meme that's capable > of succeeding, and stay "on message" with it. Translating to a plain langauge: "PSF should spend money spreading around a counter-PR". I am afraid PSF doesn't have enough money, and even if it has - should we really run down that path? Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From phd at mail2.phd.pp.ru Thu Dec 9 11:22:50 2004 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Thu Dec 9 11:22:52 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <000b01c4dda2$fe7a87e0$e841fea9@oemcomputer> References: <000b01c4dda2$fe7a87e0$e841fea9@oemcomputer> Message-ID: <20041209102250.GB12506@phd.pp.ru> On Wed, Dec 08, 2004 at 10:55:59PM -0500, Raymond Hettinger wrote: > * Have python.org prominently feature an article of Python's use in > high-performance environments. IIRC, somebody wrote a realtime voice > over internet system and found that with good design, there was no speed > issue. Also, the cellphone apps may make a good example. +Games. > * Python's website has traditionally been self-centered, leaving others > to have to make the case for their own products. Perhaps, it is time to > change that. Those who really care about speed cannot make a balanced > decision about Python without considering psyco, pyrex, numpy, and the > like as part of the total development environment. That's overreaction, I think. People always say this and that about python - python is slow, and python is not like java, and python does not have static type checks... And what? Should the webmasters adapt the site to every complain? Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From aahz at pythoncraft.com Thu Dec 9 14:15:55 2004 From: aahz at pythoncraft.com (Aahz) Date: Thu Dec 9 14:15:58 2004 Subject: [Python-Dev] PyOS_InputHook enhancement proposal In-Reply-To: <41B7D8C5.9000008@ims.u-tokyo.ac.jp> References: <41B7D8C5.9000008@ims.u-tokyo.ac.jp> Message-ID: <20041209131555.GB10465@panix.com> On Thu, Dec 09, 2004, Michiel Jan Laurens de Hoon wrote: > > My suggestion is therefore to replace PyOS_InputHook by two functions > PyOS_AddInputHook and PyOS_RemoveInputHook, and let Python keep track of > which hooks are installed. This way, an extension module can add a hook > function without having to worry about other extension modules trying > to use the same hook. > > Any comments? Would I need to submit a PEP for this proposal? Because this is only for the C API, your best bet is to write a patch and submit it to SF. If people whine or it gets rejected, then write a PEP. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "19. A language that doesn't affect the way you think about programming, is not worth knowing." --Alan Perlis From stephen.kennedy at havok.com Thu Dec 9 14:20:38 2004 From: stephen.kennedy at havok.com (Stephen Kennedy) Date: Thu Dec 9 14:32:15 2004 Subject: [Python-Dev] How to install tile (or any other tcl module) Message-ID: I've been trying to get Tile to work with python. It can make your tkinter apps look like http://tktable.sourceforge.net/tile/screenshots/demo-alt-unix.png See http://tktable.sourceforge.net/tile/ Under linux I built tile from source, installed and it just works. import Tkinter root = Tkinter.Tk() root.tk.call('package', 'require', 'tile') root.tk.call('namespace', 'import', '-force', 'ttk::*') root.tk.call('tile::setTheme', 'alt') ### Widgets are now pretty! Under win32, I installed the binary package into python/tcl (i.e. python/tcl/tile0.5) with all the other tcl packages, but tcl can't find it. Any ideas? Traceback (most recent call last): File "Script1.py", line 5, in ? root.tk.call('package', 'require', 'tile') _tkinter.TclError: can't find package tile Stephen. From anthony at interlink.com.au Thu Dec 9 14:36:16 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Dec 9 14:36:33 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <000b01c4dda2$fe7a87e0$e841fea9@oemcomputer> References: <000b01c4dda2$fe7a87e0$e841fea9@oemcomputer> Message-ID: <200412100036.17253.anthony@interlink.com.au> On Thursday 09 December 2004 14:55, Raymond Hettinger wrote: > * Have python.org prominently feature an article of Python's use in > high-performance environments. IIRC, somebody wrote a realtime voice > over internet system and found that with good design, there was no speed > issue. Also, the cellphone apps may make a good example. That was me. I gave a keynote (45 minute) version of the talk last week at OSDC, and I believe it was videoed and will be available eventually. This is good propaganda. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From Scott.Daniels at Acm.Org Thu Dec 9 15:07:41 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Thu Dec 9 15:06:19 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: <20041209102250.GB12506@phd.pp.ru> References: <000b01c4dda2$fe7a87e0$e841fea9@oemcomputer> <20041209102250.GB12506@phd.pp.ru> Message-ID: Oleg Broytmann wrote: > Raymond Hettinger wrote: >>* Python's website has traditionally been self-centered, leaving others >>to have to make the case for their own products. Perhaps, it is time to >>change that. Those who really care about speed cannot make a balanced >>decision about Python without considering psyco, pyrex, numpy, and the >>like as part of the total development environment. > > That's overreaction, I think. People always say this and that about > python - python is slow, and python is not like java, and python does > not have static type checks... And what? Should the webmasters adapt the > site to every complain? > Perhaps a link on the main page to a "for even more speed" page where we can combine advice on how to improve application performance (measure, abstract, choose good data structures) with references to these other projects for particular applications. This is the place to say things like "operating on every character of a string is seldom efficient." -- -- Scott David Daniels Scott.Daniels@Acm.Org From phd at mail2.phd.pp.ru Thu Dec 9 15:11:04 2004 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Thu Dec 9 15:11:07 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: References: <000b01c4dda2$fe7a87e0$e841fea9@oemcomputer> <20041209102250.GB12506@phd.pp.ru> Message-ID: <20041209141104.GA18348@phd.pp.ru> On Thu, Dec 09, 2004 at 06:07:41AM -0800, Scott David Daniels wrote: > Oleg Broytmann wrote: > > That's overreaction, I think. > > Perhaps a link on the main page Why on the main page? There are Topics Guide at http://python.org/topics/ that describes the ways Python can be used in some popular areas. Let's add another topic, "Making things fast". Let's even make it the first topic, though I personnaly dont see a need for this. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From anthony at interlink.com.au Thu Dec 9 15:17:50 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Dec 9 15:18:49 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: <20041209141104.GA18348@phd.pp.ru> References: <20041209141104.GA18348@phd.pp.ru> Message-ID: <200412100117.50910.anthony@interlink.com.au> FWIW, I was planning on writing a tutorial (working title: "Making Python Code Not Suck") for some conference or another... talked to a bunch of people last week at OSDC, and it seems like something people are interested in. Got a bunch of stuff already down from various notes I've written in the past when helping coworkers and the like. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From aahz at pythoncraft.com Thu Dec 9 15:55:40 2004 From: aahz at pythoncraft.com (Aahz) Date: Thu Dec 9 15:55:54 2004 Subject: [Python-Dev] How to install tile (or any other tcl module) In-Reply-To: References: Message-ID: <20041209145540.GB13555@panix.com> On Thu, Dec 09, 2004, Stephen Kennedy wrote: > > I've been trying to get Tile to work with python. > It can make your tkinter apps look like > http://tktable.sourceforge.net/tile/screenshots/demo-alt-unix.png > See http://tktable.sourceforge.net/tile/ Sorry, this is not a good place to get Python support; python-dev is for people actively working on Python language development. Please use comp.lang.python or one of the other resources. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "19. A language that doesn't affect the way you think about programming, is not worth knowing." --Alan Perlis From pje at telecommunity.com Thu Dec 9 16:30:10 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Dec 9 16:30:50 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <20041209101702.GA12506@phd.pp.ru> References: <5.1.1.6.0.20041208201940.0210b270@mail.telecommunity.com> <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041208201940.0210b270@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20041209102402.042c8580@mail.telecommunity.com> At 01:17 PM 12/9/04 +0300, Oleg Broytmann wrote: >On Wed, Dec 08, 2004 at 08:30:02PM -0500, Phillip J. Eby wrote: > > As I explained later in that message, "native" simply means, "has an .exe > > extension on Windows". > > And very soon that strategy will backfire - people will start PRing >"but those .exe's are nothing more than a python interpreter in >disguise" which in my opnion is even worse. ISTR that for a long time, Visual Basic actually did the same thing. A few magazines mentioned the fact, but nobody really cared. However, if this is really a concern, bundle Pyrex as well. Both Pyrex and py2exe are distutils-based, so at that point you have a complete solution, including the "C" meme as well as the ".exe" meme. > > If the goal is > > to achieve a PR win, the important thing is to pick a meme that's capable > > of succeeding, and stay "on message" with it. > > Translating to a plain langauge: "PSF should spend money spreading >around a counter-PR". I am afraid PSF doesn't have enough money, and >even if it has - should we really run down that path? I'm not suggesting spending any money. Heck, I'm not suggesting *doing* anything. I just answered Guido's question about PR. Whether anybody wants to *do* anything about it is an entirely separate issue from discussing *what* would need to be done, if something was going to be done. From gmccaughan at synaptics-uk.com Thu Dec 9 17:39:21 2004 From: gmccaughan at synaptics-uk.com (Gareth McCaughan) Date: Thu Dec 9 17:39:53 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> References: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> Message-ID: <200412091639.21596.gmccaughan@synaptics-uk.com> On Wednesday 2004-12-08 22:39, Phillip J. Eby wrote: [Guido:] >> One thing that bugs me: the article says 3 or 4 times that Python is >> slow, each time with a refutation ("but it's so flexible", "but it's >> fast enough") but still, they sure seem to harp on the point. This is >> a PR issue that Python needs to fight -- any ideas? [Philip:] > The only thing that will fix the PR issue is to have a Python compiler > distributed as part of the language. It doesn't matter if it doesn't > support the full generality of Python, or even if it doesn't speed many > operations up much. The only real requirements are that it can be used to > produce "native" executables, and that it be an official part of the > language, not a separately-distributed tool like Psyco or Pyrex. Then, it > will perhaps be a sufficient "security blanket" to stop people FUDding > about it. Unfortunately, this may not be enough. Consider, by way of counterexample, Common Lisp, which - is compiled to native code - has optional type declarations - actually *does* run fast when compiled - has had all these properties for years and years but is still almost universally decried as "slow" by people who have never actually used it. It's true that it doesn't (as part of the standard, and in the free implementations I know of) have the ability to generate standalone executables with filenames ending in ".exe"; perhaps that's the key thing. The other thing that might work is to change the name of the language to "C" plus optional punctuation. -- g From janssen at parc.com Thu Dec 9 17:56:09 2004 From: janssen at parc.com (Bill Janssen) Date: Thu Dec 9 17:56:57 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: Your message of "Thu, 09 Dec 2004 08:39:21 PST." <200412091639.21596.gmccaughan@synaptics-uk.com> Message-ID: <04Dec9.085613pst."58617"@synergy1.parc.xerox.com> > The other thing that might work is to change the name of the language > to "C" plus optional punctuation. You mean "C@@" (pronounced C-pie-pie)? Bill From fredrik at pythonware.com Thu Dec 9 18:57:10 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu Dec 9 18:54:56 2004 Subject: [Python-Dev] Re: How to install tile (or any other tcl module) References: <20041209145540.GB13555@panix.com> Message-ID: >> I've been trying to get Tile to work with python. >> It can make your tkinter apps look like >> http://tktable.sourceforge.net/tile/screenshots/demo-alt-unix.png >> See http://tktable.sourceforge.net/tile/ > > Sorry, this is not a good place to get Python support; python-dev is for > people actively working on Python language development. Please use > comp.lang.python or one of the other resources. in this case, http://mail.python.org/mailman/listinfo/tkinter-discuss is the right forum (also see: http://tkinter.unpythonic.net/wiki/TkinterDiscuss ) From fumanchu at amor.org Thu Dec 9 19:21:00 2004 From: fumanchu at amor.org (Robert Brewer) Date: Thu Dec 9 19:23:07 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places Message-ID: <3A81C87DC164034AA4E2DDFE11D258E324538E@exchange.hqamor.amorhq.net> Anthony Baxter wrote: > FWIW, I was planning on writing a tutorial (working title: > "Making Python Code Not Suck") for some conference > or another... Perhaps, given your high profile in the Python developer community, you might reconsider the title? Little details like that are what PR is made of. Imagine Bush's next Executive Order being titled "Making American [Business|Military|People] Not Suck"... ;) Robert Brewer MIS Amor Ministries fumanchu@amor.org From mwh at python.net Thu Dec 9 19:28:01 2004 From: mwh at python.net (Michael Hudson) Date: Thu Dec 9 19:28:03 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: <3A81C87DC164034AA4E2DDFE11D258E324538E@exchange.hqamor.amorhq.net> (Robert Brewer's message of "Thu, 9 Dec 2004 10:21:00 -0800") References: <3A81C87DC164034AA4E2DDFE11D258E324538E@exchange.hqamor.amorhq.net> Message-ID: <2mk6rrpbri.fsf@starship.python.net> "Robert Brewer" writes: > Anthony Baxter wrote: >> FWIW, I was planning on writing a tutorial (working title: >> "Making Python Code Not Suck") for some conference >> or another... > > Perhaps, given your high profile in the Python developer community, you > might reconsider the title? Little details like that are what PR is made > of. Imagine Bush's next Executive Order being titled "Making American > [Business|Military|People] Not Suck"... ;) Anthony's Australian, people expect this sort of thing from him :) Cheers, mwh -- Also, remember to put the galaxy back when you've finished, or an angry mob of astronomers will come round and kneecap you with a small telescope for littering. -- Simon Tatham, ucam.chat, from Owen Dunn's review of the year From pje at telecommunity.com Thu Dec 9 19:31:15 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Dec 9 19:32:05 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <200412091639.21596.gmccaughan@synaptics-uk.com> References: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20041209131843.038f3530@mail.telecommunity.com> At 04:39 PM 12/9/04 +0000, Gareth McCaughan wrote: >On Wednesday 2004-12-08 22:39, Phillip J. Eby wrote: > > The only thing that will fix the PR issue is to have a Python compiler > > distributed as part of the language. It doesn't matter if it doesn't > > support the full generality of Python, or even if it doesn't speed many > > operations up much. The only real requirements are that it can be used to > > produce "native" executables, and that it be an official part of the > > language, not a separately-distributed tool like Psyco or Pyrex. Then, it > > will perhaps be a sufficient "security blanket" to stop people FUDding > > about it. > >Unfortunately, this may not be enough. Consider, by way of counterexample, >Common Lisp, which > - is compiled to native code > - has optional type declarations > - actually *does* run fast when compiled > - has had all these properties for years and years >but is still almost universally decried as "slow" by people who have >never actually used it. It's true that it doesn't (as part of the >standard, and in the free implementations I know of) have the ability >to generate standalone executables with filenames ending in ".exe"; >perhaps that's the key thing. Among the great unwashed masses of Windows programmers, it probably is. Look at the marketing for virtually any off-beat commercial language for Windows (that's still around to look at), and you will usually find a prominent bullet point about how it makes .exe's. This is an incredibly powerful meme, though I don't entirely understand why. I do agree that it still may not be enough; I'm just saying that it's not a coincidence that so many development tools for non-C languages on Windows all ended up deciding to feature that bullet point. My theory is that it's because it's a FAQ, and it's often found on the official FAQ's of open source languages as well. Granted, the scope of the question is much broader than just speed, since it also encompasses application distribution issues, and the social status of the developer. One of the subtexts of the question is, "Can I make a *real* program with this thing?", or put more honestly, "Will other programmers laugh at me when they see my program isn't an .exe?" >The other thing that might work is to change the name of the language >to "C" plus optional punctuation. Well, we could always just call it CPython, and bill it as a faster C implementation of that popular dynamic language for Java, Jython. That way, we can also leverage the meme that C is faster than Java. :) Isn't it funny, by the way, that people don't go around talking about how slow Jython is? At least, I haven't seen it to the extent that I've seen with CPython. From foom at fuhm.net Thu Dec 9 20:16:18 2004 From: foom at fuhm.net (James Y Knight) Date: Thu Dec 9 20:16:24 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <5.1.1.6.0.20041209131843.038f3530@mail.telecommunity.com> References: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041209131843.038f3530@mail.telecommunity.com> Message-ID: On Dec 9, 2004, at 1:31 PM, Phillip J. Eby wrote: > Isn't it funny, by the way, that people don't go around talking about > how slow Jython is? At least, I haven't seen it to the extent that > I've seen with CPython. People talk about how slow CPython is, is because they are writing code targeted for it that traditionally would have been written in C. Python *is* slower than C. You can try to deny it, but it is simple fact. However, python is a really nice language to program in. So people use it anyways. It is "fast enough" for many things. Nobody talks about how slow Jython is, because nobody(1) is writing entire programs in it. Jython is used as a way to script Java programs, and as such, it is pretty much completely unimportant how fast it runs. Java has also traditionally got a lot of bad press for its lack of speed, and it was usually of the "not fast enough" variety, which is much worse than what's said of Python. However, a huge amount of progress has been made and Java really is quite speedy these days. It was not PR that made it faster, but hard work on the part of Sun. There is currently a lot of work going on in the python community to make Python faster, which indicates both that Python is too slow for some people, and that it will be faster in the future. This is a good thing, and not something that should be shoved under the rug to try to pretend python is ultra mega super fast already. James 1) Of course someone probably is, but you get the idea. From tdelaney at avaya.com Thu Dec 9 20:25:54 2004 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Thu Dec 9 20:27:14 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DE4A915A@au3010avexu1.global.avaya.com> Michael Hudson wrote: > Anthony's Australian, people expect this sort of thing from him :) As another Australian, I think that "Making Python Not Suck" implies that if you don't do extra things, Python Sucks. This is not a good thing IMO. "Making Python Suck Less" would be even worse. How about "Python - You Can Have Your Cake And Eat It Too". Tim Delaney From jhylton at gmail.com Thu Dec 9 20:56:41 2004 From: jhylton at gmail.com (Jeremy Hylton) Date: Thu Dec 9 20:56:45 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: References: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041209131843.038f3530@mail.telecommunity.com> Message-ID: On Thu, 9 Dec 2004 14:16:18 -0500, James Y Knight wrote: > On Dec 9, 2004, at 1:31 PM, Phillip J. Eby wrote: > > Isn't it funny, by the way, that people don't go around talking about > > how slow Jython is? At least, I haven't seen it to the extent that > > I've seen with CPython. > > People talk about how slow CPython is, is because they are writing code > targeted for it that traditionally would have been written in C. Python > *is* slower than C. You can try to deny it, but it is simple fact. > However, python is a really nice language to program in. So people use > it anyways. It is "fast enough" for many things. I think you are exactly right. It's easy to get the impression that Python is slow when many programs written in Python *are* slowly than the same program written in C, C++, or Java. If you look are other "dynamic" languages, you'll find that IronPython is often faster than CPython and that Smalltalk and Common Lisp are often substantially faster than Python. Python isn't remarkably slow for a scripting language, but it is not fast. > There is currently a lot of work going on in the python community to > make Python faster, which indicates both that Python is too slow for > some people, and that it will be faster in the future. This is a good > thing, and not something that should be shoved under the rug to try to > pretend python is ultra mega super fast already. I agree, although it's not clear to me how much faster it will be in the future. Making a *fast* Python based on our own virtual execution environment (as opposed to piggybacking a JVM or CLR) is a big project. It's not clear than anyone has enough resources to make credible effort, so we're left to piecemeal improvements. The improvement I see from 2.2 to 2.4 on my box is about 55% (measured using pystone) over almost three years. I think the right thing to do with PR is frame the question differently. We need to say that people use Python because it lets them right clear, simple code or that it is easy to maintain or that dynamic languages are excellent for prototyping -- whatever the message is -- and simply avoid talking about speed. There are a lot of issues that affect the selection of a programming language, and speed is not always the most important one. It is easy for a journalist to write about and it is easy to report (possibly meaningless) measurements of speed. Jeremy From s.percivall at chello.se Thu Dec 9 21:10:02 2004 From: s.percivall at chello.se (Simon Percivall) Date: Thu Dec 9 21:10:07 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: References: <000b01c4dda2$fe7a87e0$e841fea9@oemcomputer> <20041209102250.GB12506@phd.pp.ru> Message-ID: <4FA61149-4A1E-11D9-9ADB-0003934AD54A@chello.se> On 2004-12-09, at 15.07, Scott David Daniels wrote: > Oleg Broytmann wrote: >> Raymond Hettinger wrote: >>> * Python's website has traditionally been self-centered, leaving >>> others >>> to have to make the case for their own products. Perhaps, it is >>> time to >>> change that. Those who really care about speed cannot make a >>> balanced >>> decision about Python without considering psyco, pyrex, numpy, and >>> the >>> like as part of the total development environment. >> That's overreaction, I think. People always say this and that about >> python - python is slow, and python is not like java, and python does >> not have static type checks... And what? Should the webmasters adapt >> the >> site to every complain? > > Perhaps a link on the main page to a "for even more speed" page where > we > can combine advice on how to improve application performance (measure, > abstract, choose good data structures) with references to these other > projects for particular applications. This is the place to say things > like "operating on every character of a string is seldom efficient." +1 Many examples of common idioms and tasks and how to do them right in Python would also be useful for such a page. //Simon From allison at sumeru.stanford.EDU Thu Dec 9 21:18:07 2004 From: allison at sumeru.stanford.EDU (Dennis Allison) Date: Thu Dec 9 21:18:59 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DE4A915A@au3010avexu1.global.avaya.com> Message-ID: The goal here is to make Python better known and to counter some of the prevalent myths. One way to accomplish this goal is to publish literate technical articles with real content including performance measurements and pointers to the code. Perhaps Guido could be a real-life N. Bourbaki and co-publish with developers. A significant issue is: where to publish. Certainly publishing in the technical society journals (ACM & IEEE CS) would make sense. Software Practice and Experience would also be good. Likewise DDJ and Game Developer Magazine. But I suspect that much of our target audience will be missed if we limit ourselves to these journals. We also need to get articles placed in the mass market computer mags. And we need to build an editorial relationship with the mass market journals so that when X says they think "Python is slow" there's some they know they can call for a truth-squad quote. And, parenthetically, I continue to be amazed at the number of projects that use Python, but do it in stealth-mode and view it as their silver-bullet and competative edge. I wish more people would publish their experience. From glyph at divmod.com Thu Dec 9 22:11:33 2004 From: glyph at divmod.com (Glyph Lefkowitz) Date: Thu Dec 9 22:10:35 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> References: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> Message-ID: <1102626693.16291.57.camel@localhost> On Wed, 2004-12-08 at 17:39 -0500, Phillip J. Eby wrote: > The only thing that will fix the PR issue is to have a Python compiler > distributed as part of the language. It doesn't matter if it doesn't > support the full generality of Python, or even if it doesn't speed many > operations up much. The only real requirements are that it can be used to > produce "native" executables, and that it be an official part of the > language, not a separately-distributed tool like Psyco or Pyrex. Then, it > will perhaps be a sufficient "security blanket" to stop people FUDding > about it. I am aware that Pyrex is still in flux, so perhaps it is too soon to propose this even for 2.5, but I think it's worth bringing up anyway: I would like Pyrex to be distributed with the Python core. I agree that it should be modified to produce full .exe files and not just .dlls on Windows, but many potential users *are* seriously concerned about efficiency and not just simplifying distribution. The main benefit would be imparting a Pythonesque flavor to optimized code. The thing which experienced Python-ers say is, technically, correct: "If you need speed, rewrite the hot-spots in C.". To someone already steeped in the Python way, this means, "write your application in Python, do your architecture in Python, profile it, and write the 300 lines or so of your 25000 line application that are *really* speed-critical in a faster language". There are a few, very very specialized areas where this approach is not the best one, but unfortunately C programmers do not hear this when you say it. What they hear is, "If speed is important to your application, write it in C, and only write the parts where you need flexibility in Python. They will all be deadly slow. If you notice that they are slow, you are going to have to rewrite them in C anyway." To wit, Python is only interesting to the people who are going to "script" your application. Although writing code in Pyrex ends up being a technically similar solution, politically it is radically different. If the Python-expert's answer were instead, "write the slow parts in Pyrex", then the C programmer's plan changes - instead of writing in C, because they know they are going to need to rewrite all the slow parts anyway, they realize that there is an explicitly Python way to rewrite the slow parts, that may leverage SOME of their existing C knowledge. They learn Python and appreciate Python secure in the knowledge that this effort will not be wasted. Finally, there are real benefits to Pyrex when making the transition between levels in your optimized code. All the common areas for mistakes when interfacing with Python - missing refcounts when getting, changing, or deleting attributes, calling Python callbacks, and using basic Python data structures - are all taken care of so that the dive into C is far less abrupt. The people this PR is geared to are those who would write in C because of efficiency concerns, so they would likely be looking at Pyrex before choosing to start a large project, and the simple code examples would be both familiar enough (hey, that's a 32 bit integer! I remember that!) but also different enough (wow, you can just call a script with "foo()"?) that they would be impressed. From barry at python.org Thu Dec 9 22:20:32 2004 From: barry at python.org (Barry Warsaw) Date: Thu Dec 9 22:20:37 2004 Subject: Freezing Python (was Re: [Python-Dev] 2.4 news reaches interesting places) In-Reply-To: <5.1.1.6.0.20041209102402.042c8580@mail.telecommunity.com> References: <5.1.1.6.0.20041208201940.0210b270@mail.telecommunity.com> <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041208201940.0210b270@mail.telecommunity.com> <5.1.1.6.0.20041209102402.042c8580@mail.telecommunity.com> Message-ID: <1102627232.16867.47.camel@geddy.wooz.org> On Thu, 2004-12-09 at 10:30, Phillip J. Eby wrote: > ISTR that for a long time, Visual Basic actually did the same thing. A few > magazines mentioned the fact, but nobody really cared. However, if this is > really a concern, bundle Pyrex as well. Both Pyrex and py2exe are > distutils-based, so at that point you have a complete solution, including > the "C" meme as well as the ".exe" meme. As an aside, I wonder if there's interest in collaborating on freeze tools among the various existing solutions here. I've recently needed to look at this and I chose cx_Freeze, primarily because it supported the platforms I needed. It's a very good tool. I've had some conversations with Anthony Tuininga on the cx_Freeze mailing list and there seems to be some interest in perhaps putting together a SIG or something. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041209/ce332aec/attachment.pgp From bob at redivi.com Thu Dec 9 22:30:30 2004 From: bob at redivi.com (Bob Ippolito) Date: Thu Dec 9 22:31:08 2004 Subject: Freezing Python (was Re: [Python-Dev] 2.4 news reaches interesting places) In-Reply-To: <1102627232.16867.47.camel@geddy.wooz.org> References: <5.1.1.6.0.20041208201940.0210b270@mail.telecommunity.com> <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041208201940.0210b270@mail.telecommunity.com> <5.1.1.6.0.20041209102402.042c8580@mail.telecommunity.com> <1102627232.16867.47.camel@geddy.wooz.org> Message-ID: <8CDE78F4-4A29-11D9-8CCD-000A9567635C@redivi.com> On Dec 9, 2004, at 4:20 PM, Barry Warsaw wrote: > On Thu, 2004-12-09 at 10:30, Phillip J. Eby wrote: > >> ISTR that for a long time, Visual Basic actually did the same thing. >> A few >> magazines mentioned the fact, but nobody really cared. However, if >> this is >> really a concern, bundle Pyrex as well. Both Pyrex and py2exe are >> distutils-based, so at that point you have a complete solution, >> including >> the "C" meme as well as the ".exe" meme. > > As an aside, I wonder if there's interest in collaborating on freeze > tools among the various existing solutions here. I've recently needed > to look at this and I chose cx_Freeze, primarily because it supported > the platforms I needed. It's a very good tool. > > I've had some conversations with Anthony Tuininga on the cx_Freeze > mailing list and there seems to be some interest in perhaps putting > together a SIG or something. That's certainly worth doing. Currently we have, in general use, at least cx_Freeze, py2exe, and py2app. I have been abstracting away a lot of the dependency finding issues in py2app into a cross-platform module called modulegraph . Modulegraph supercedes the standard library modulefinder. Because it uses a graph data structure, it gives you the flexibility say "pydoc doesn't *really* depend on Tkinter". It also has functionality that lets you specify implicit dependencies (via PyImport_.. or such), so that it plays nicely enough with C extensions. This code is basically at the point where py2exe and cx_Freeze could import it and use it instead of modulefinder (with a small amount of hacking, the API is different). -bob From aahz at pythoncraft.com Thu Dec 9 22:35:56 2004 From: aahz at pythoncraft.com (Aahz) Date: Thu Dec 9 22:35:58 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: References: <338366A6D2E2CA4C9DAEAE652E12A1DE4A915A@au3010avexu1.global.avaya.com> Message-ID: <20041209213556.GB4013@panix.com> On Thu, Dec 09, 2004, Dennis Allison wrote: > > And, parenthetically, I continue to be amazed at the number of projects > that use Python, but do it in stealth-mode and view it as their > silver-bullet and competative edge. I wish more people would publish > their experience. http://www.paulgraham.com/avg.html -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "19. A language that doesn't affect the way you think about programming, is not worth knowing." --Alan Perlis From pje at telecommunity.com Thu Dec 9 22:37:59 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Dec 9 22:38:58 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <1102626693.16291.57.camel@localhost> References: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20041209162547.06531700@mail.telecommunity.com> At 04:11 PM 12/9/04 -0500, Glyph Lefkowitz wrote: >On Wed, 2004-12-08 at 17:39 -0500, Phillip J. Eby wrote: > > > The only thing that will fix the PR issue is to have a Python compiler > > distributed as part of the language. It doesn't matter if it doesn't > > support the full generality of Python, or even if it doesn't speed many > > operations up much. The only real requirements are that it can be used to > > produce "native" executables, and that it be an official part of the > > language, not a separately-distributed tool like Psyco or Pyrex. Then, it > > will perhaps be a sufficient "security blanket" to stop people FUDding > > about it. > >I am aware that Pyrex is still in flux, so perhaps it is too soon to >propose this even for 2.5, but I think it's worth bringing up anyway: I >would like Pyrex to be distributed with the Python core. I agree that >it should be modified to produce full .exe files and not just .dlls on >Windows, but many potential users *are* seriously concerned about >efficiency and not just simplifying distribution. +1 on all the stuff you said, with one minor exception. Pyrex-the-language is often unpythonically ugly and verbose at present. If Python had an official syntax for optional static type declaration, Pyrex's syntax could be aligned with that, and that would at least eliminate most of the inline 'cdef' ugliness, leaving only C type declarations and Python property declarations as the main syntax issues to be resolved. (Maybe by using something like the 'ctypes' API, and having the compiler recognize that API, such that C is directly callable from Python anyway, so compiling or interpreting Python makes no difference to ability to access C... but I digress.) Of course, this would take some effort from the core developers, especially Guido, to consider the various syntax needs and formulate official solutions. But if it were done, the Python-vs.-Pyrex distinction could fade away altogether, replaced with the meme, "just add type declarations to slow parts, and tell Python you want the module compiled to C." IOW, if Pyrex is merely the name of a compiler, not a separate language, then our master plan for world domination is complete. :) From michael.walter at gmail.com Thu Dec 9 22:43:19 2004 From: michael.walter at gmail.com (Michael Walter) Date: Thu Dec 9 22:43:22 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <5.1.1.6.0.20041209162547.06531700@mail.telecommunity.com> References: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <1102626693.16291.57.camel@localhost> <5.1.1.6.0.20041209162547.06531700@mail.telecommunity.com> Message-ID: <877e9a17041209134356f9290e@mail.gmail.com> If I parse you correctly, this would be great. - Michael On Thu, 09 Dec 2004 16:37:59 -0500, Phillip J. Eby wrote: > At 04:11 PM 12/9/04 -0500, Glyph Lefkowitz wrote: > > > >On Wed, 2004-12-08 at 17:39 -0500, Phillip J. Eby wrote: > > > > > The only thing that will fix the PR issue is to have a Python compiler > > > distributed as part of the language. It doesn't matter if it doesn't > > > support the full generality of Python, or even if it doesn't speed many > > > operations up much. The only real requirements are that it can be used to > > > produce "native" executables, and that it be an official part of the > > > language, not a separately-distributed tool like Psyco or Pyrex. Then, it > > > will perhaps be a sufficient "security blanket" to stop people FUDding > > > about it. > > > >I am aware that Pyrex is still in flux, so perhaps it is too soon to > >propose this even for 2.5, but I think it's worth bringing up anyway: I > >would like Pyrex to be distributed with the Python core. I agree that > >it should be modified to produce full .exe files and not just .dlls on > >Windows, but many potential users *are* seriously concerned about > >efficiency and not just simplifying distribution. > > +1 on all the stuff you said, with one minor exception. Pyrex-the-language > is often unpythonically ugly and verbose at present. If Python had an > official syntax for optional static type declaration, Pyrex's syntax could > be aligned with that, and that would at least eliminate most of the inline > 'cdef' ugliness, leaving only C type declarations and Python property > declarations as the main syntax issues to be resolved. (Maybe by using > something like the 'ctypes' API, and having the compiler recognize that > API, such that C is directly callable from Python anyway, so compiling or > interpreting Python makes no difference to ability to access C... but I > digress.) > > Of course, this would take some effort from the core developers, especially > Guido, to consider the various syntax needs and formulate official > solutions. But if it were done, the Python-vs.-Pyrex distinction could > fade away altogether, replaced with the meme, "just add type declarations > to slow parts, and tell Python you want the module compiled to C." > > IOW, if Pyrex is merely the name of a compiler, not a separate language, > then our master plan for world domination is complete. :) > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/michael.walter%40gmail.com > From glyph at divmod.com Thu Dec 9 23:11:45 2004 From: glyph at divmod.com (Glyph Lefkowitz) Date: Thu Dec 9 23:10:47 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <5.1.1.6.0.20041209162547.06531700@mail.telecommunity.com> References: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041209162547.06531700@mail.telecommunity.com> Message-ID: <1102630305.16291.62.camel@localhost> On Thu, 2004-12-09 at 16:37 -0500, Phillip J. Eby wrote: > +1 on all the stuff you said, with one minor exception. Pyrex-the-language > is often unpythonically ugly and verbose at present. Personally I have no problem with Pyrex's current tradeoffs, but given the amount of effort that this is going to inevitably involve, I'm sure that at least some of the changes you suggested would be necessary. So... +0 on your suggestions, I suppose. From faassen at infrae.com Fri Dec 10 01:09:05 2004 From: faassen at infrae.com (Martijn Faassen) Date: Fri Dec 10 01:09:08 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: References: Message-ID: <41B8E921.2090903@infrae.com> Guido van Rossum wrote: [snip] > One thing that bugs me: the article says 3 or 4 times that Python is > slow, each time with a refutation ("but it's so flexible", "but it's > fast enough") but still, they sure seem to harp on the point. This is > a PR issue that Python needs to fight -- any ideas? One thing you could do is announce loudly that the European Union is actually funding a international consortium of scientists, industry and software engineers to speed up Python (among other things :) -- PyPy. I'm sure that a press release about this in the right channels could make quite a bit of positive noise. "Python, backed by the EU". Of course one worry is that the 'slow bureaucracy' meme would rub off on it. Of course any press release would need to be discussed with the PyPy people. Regards, Martijn From bob at redivi.com Fri Dec 10 02:17:21 2004 From: bob at redivi.com (Bob Ippolito) Date: Fri Dec 10 02:17:58 2004 Subject: [Python-Dev] Proposed Mac OS X change for 2.3.5 and 2.4.1 - MACOSX_DEPLOYMENT_TARGET vs. Python, round two, fight! Message-ID: <3DBFC242-4A49-11D9-8CCD-000A9567635C@redivi.com> I've included a lot of background information here, if you just want to know the details of the proposed change, skip to the bottom. As some of you may know, Python 2.4's configure script and distutils has some tricky behavior with regard to the ``MACOSX_DEPLOYMENT_TARGET`` environment variable on Mac OS X 10.3 and later. Unless otherwise specified, assume that I am talking about a build environment of Mac OS X 10.3 or later. This behavior is as follows: If ``MACOSX_DEPLOYMENT_TARGET`` is not set during configure: 1. distutils builds modules with ``-F/Python/Installation/Location -framework Python`` as per usual 2.3.x behavior if it is also not set 2. If ``MACOSX_DEPLOYMENT_TARGET`` is set during a later run of distutils, then distutils complains that "10.3" mismatches the configure time setting of "" This Python framework has the following misfeatures: 1. All your extensions are dependent on the installation location of this particular Python 2. This installation of Python 2.4 *may break the building of extensions* for any previous version of Python that uses the same distutils behavior. It will certainly break them if they are installed to the same framework directory, but if the version of Python 2.3 is early enough, such as the stock 2.3.0, it will break that too. Also, any future version of Python installed to the same frameworks directory *will break the building of extensions* for this Python installation! 3. The Python framework will not be compatible with versions of Mac OS X earlier than 10.3 anyway due to changes in libSystem! This is stupid, and it should NOT be default behavior! If ``MACOSX_DEPLOYMENT_TARGET`` is set to "10.3" or higher during configure: 1. distutils builds modules with ``-undefined dynamic_lookup`` 2. If ``MACOSX_DEPLOYMENT_TARGET`` is set to something other than "10.3", or unset, it will complain that the current setting mismatches the configure setting of "10.3" The features: 1. All your extensions are independent of the Python installation location, and are thus compatible with any other Python with the same major version. 2. This installation of Python 2.4 will still, unavoidably, break the building of extensions for any previous version of Python using the bad distutils behavior mentioned above. This installation is not susceptible to breakage, however. The misfeatures: 1. You need to have ``MACOSX_DEPLOYMENT_TARGET`` set during configure, most people don't know to do this. 2. You also need to have ``MACOSX_DEPLOYMENT_TARGET`` set when using distutils. Even less people know to do this. The solution to this is to have the following behavior instead: 1. If no ``MACOSX_DEPLOYMENT_TARGET`` is set during configure, and the build machine is Mac OS X 10.3 or later, then set it to "10.3". 2. If no ``MACOSX_DEPLOYMENT_TARGET`` is set during distutils, but it WAS set during configure, then set it to the configure time value. 3. Otherwise, if it is set to a LOWER value, then fail. Checking for an exact match is bad, because the user or extension author should be free to build a particular extension using 10.4 specific features against a Python that is 10.3+ compatible. For a build machine running Mac OS X 10.2 or earlier, it should ignore all of this behavior as per usual. Unless anyone has a reasonable objection to this proposed solution, then we should make sure it goes into Python 2.4.1 and Python 2.3.5. These threads might also be of interest: http://mail.python.org/pipermail/pythonmac-sig/2004-November/012192.html http://mail.python.org/pipermail/pythonmac-sig/2004-December/012237.html http://mail.python.org/pipermail/pythonmac-sig/2004-December/012275.html -bob From skip at pobox.com Thu Dec 9 13:12:31 2004 From: skip at pobox.com (Skip Montanaro) Date: Fri Dec 10 03:29:04 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <000b01c4dda2$fe7a87e0$e841fea9@oemcomputer> References: <000b01c4dda2$fe7a87e0$e841fea9@oemcomputer> Message-ID: <16824.16687.343664.151558@montanaro.dyndns.org> Raymond> * Any PR effort should also emphasize that no usability Raymond> trade-offs were made along the way. A number of features Raymond> make Py2.4 easier to use than 1.5.6: list comps, genexps, Raymond> generators, sets, nested scopes, int/long unification, Raymond> decorators, etc. Not to mention which, such beasts aren't commonly available for C. What about C++? I found it interesting that a guy at work wrote a string caching class for our C++ programmers to use. He told me he got the idea from Python's int caching. Skip From anthony at interlink.com.au Fri Dec 10 03:31:20 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Dec 10 03:31:37 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DE4A915A@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DE4A915A@au3010avexu1.global.avaya.com> Message-ID: <200412101331.21291.anthony@interlink.com.au> On Friday 10 December 2004 06:25, Delaney, Timothy C (Timothy) wrote: > Michael Hudson wrote: > > Anthony's Australian, people expect this sort of thing from him :) > > As another Australian, I think that "Making Python Not Suck" implies > that if you don't do extra things, Python Sucks. > > This is not a good thing IMO. > > "Making Python Suck Less" would be even worse. Don't worry - it will have a nice friendly title by the time I present it. I'm currently aiming for OSCON, which isn't until August. It's going to be aimed at people who _know_ Python, but aren't necessarily experts, and focus not just on performance, but maintainable and sane code (for instance, not many people seem to realise that __del__ methods are almost always not the right answer to a problem). > How about "Python - You Can Have Your Cake And Eat It Too". Nah - I have a bunch of ideas kicking around in my head, something will pop out. I doubt I'll ever be able to top the title of my pycon talk, though... Anthony From kdart at kdart.com Fri Dec 10 04:41:19 2004 From: kdart at kdart.com (Keith Dart) Date: Fri Dec 10 04:41:30 2004 Subject: [Python-Dev] PyOS_InputHook enhancement proposal In-Reply-To: <20041209131555.GB10465@panix.com> References: <41B7D8C5.9000008@ims.u-tokyo.ac.jp> <20041209131555.GB10465@panix.com> Message-ID: <41B91ADF.6040800@kdart.com> Aahz wrote: > On Thu, Dec 09, 2004, Michiel Jan Laurens de Hoon wrote: > >>My suggestion is therefore to replace PyOS_InputHook by two functions >>PyOS_AddInputHook and PyOS_RemoveInputHook, and let Python keep track of >>which hooks are installed. This way, an extension module can add a hook >>function without having to worry about other extension modules trying >>to use the same hook. >> >>Any comments? Would I need to submit a PEP for this proposal? > > > Because this is only for the C API, your best bet is to write a patch > and submit it to SF. If people whine or it gets rejected, then write a > PEP. I did modify the readline module that hooks this and can call back to a Python function. There are also methods for installing and removing the Python function. I did this for a different reason. I need Python signal handlers to run, and they don't run when the execution path is currently in some C code (such as readline). The hook forces the interpreter to run, and check for signals as a side effect. Using this I can be sitting in raw_input(), or interactive mode, and signal handlers are still run. If you want it, let me know. Actually, the modded readline module is part of pyNMS, on sourceforge. -- \/ \/ (O O) -- --------------------oOOo~(_)~oOOo---------------------------------------- Keith Dart vcard: public key: ID: F3D288E4 URL: ============================================================================ From arigo at tunes.org Fri Dec 10 10:41:28 2004 From: arigo at tunes.org (Armin Rigo) Date: Fri Dec 10 10:50:49 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <86F33E75-498A-11D9-BB31-000D93C603C0@indranet.co.nz> References: <86F33E75-498A-11D9-BB31-000D93C603C0@indranet.co.nz> Message-ID: <20041210094128.GA15865@vicky.ecs.soton.ac.uk> Hi Andrew, On Thu, Dec 09, 2004 at 03:32:10PM +1300, Andrew McGregor wrote: > The other thing we can do is finish the portable backend for psyco and > make it a standard module. Then Python won't be slow, it will be > compiled, and py2exe will be able to make a single-file executable. You probably mean that Psyco can dynamically compile Python bytecodes even if they have been hidden into an .exe file by py2exe. At first reading, it appears that you are propagating the misconception "compiler == speed == .exe" i.e. Psyco could "compile your program into an .exe file". As far as I see, this is definitely not possible. Of course, for PR purposes the difference is small -- an .exe file that runs at compiled speed, or at least machine-level VM speed -- but advertising should work around and not build on top of existing confusion IMHO. A bientot, Armin From arigo at tunes.org Fri Dec 10 11:06:59 2004 From: arigo at tunes.org (Armin Rigo) Date: Fri Dec 10 11:16:15 2004 Subject: [Python-Dev] The other Py2.4 issue In-Reply-To: <001401c4dda9$ad605680$e841fea9@oemcomputer> References: <001401c4dda9$ad605680$e841fea9@oemcomputer> Message-ID: <20041210100659.GB15865@vicky.ecs.soton.ac.uk> Hi, On Wed, Dec 08, 2004 at 11:43:49PM -0500, Raymond Hettinger wrote: > Acceptance for Py2.4 partially hinges on how quickly third party apps > have their binaries updated. > > I wonder if there is anything we can do to help. For people like myself, Linux programmers not developing on Windows every day, there is precious little information available about how to compile our extension modules for the new Windows distribution. I was actually very disappointed to have to google my way around until I found a page that explained to me that to use Microsoft's free compilers you need to manually patch the distutils included with 2.4. (I know this is being worked on right now but I'd have expected it to be present in the 2.4 release.) (The only page I could find still refers to a pre-final 2.4 so their distutils patch doesn't apply without hand-editing, though that should be fixed by now.) Moreover, the standard documentation's "Extending and Embedding" has apparently not been updated at all since 2.3. That's quite unexpected too... In other words, if you want 3rd parties to compile Windows binaries for 2.4, tell them how. (I even hoped that just doing "python setup.py build" would fail but refer me to a web page that explains me which free compilers I should get and install from Microsoft.) A bientot, Armin. From skip at pobox.com Fri Dec 10 11:49:30 2004 From: skip at pobox.com (Skip Montanaro) Date: Fri Dec 10 11:49:47 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <20041210094128.GA15865@vicky.ecs.soton.ac.uk> References: <86F33E75-498A-11D9-BB31-000D93C603C0@indranet.co.nz> <20041210094128.GA15865@vicky.ecs.soton.ac.uk> Message-ID: <16825.32570.898575.571826@montanaro.dyndns.org> >> The other thing we can do is finish the portable backend for psyco >> and make it a standard module. Then Python won't be slow, it will be >> compiled, and py2exe will be able to make a single-file executable. Armin> You probably mean that Psyco can dynamically compile Python Armin> bytecodes even if they have been hidden into an .exe file by Armin> py2exe. I didn't read it that way. My impression was that py2exe be modified to include and enable psyco if it's available when building an .exe. You would, in theory, get a single file distribution as well as dynamic compilation. Skip From theller at python.net Fri Dec 10 12:12:36 2004 From: theller at python.net (Thomas Heller) Date: Fri Dec 10 12:11:49 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <16825.32570.898575.571826@montanaro.dyndns.org> (Skip Montanaro's message of "Fri, 10 Dec 2004 04:49:30 -0600") References: <86F33E75-498A-11D9-BB31-000D93C603C0@indranet.co.nz> <20041210094128.GA15865@vicky.ecs.soton.ac.uk> <16825.32570.898575.571826@montanaro.dyndns.org> Message-ID: Skip Montanaro writes: > >> The other thing we can do is finish the portable backend for psyco > >> and make it a standard module. Then Python won't be slow, it will be > >> compiled, and py2exe will be able to make a single-file executable. > > Armin> You probably mean that Psyco can dynamically compile Python > Armin> bytecodes even if they have been hidden into an .exe file by > Armin> py2exe. > > I didn't read it that way. My impression was that py2exe be modified to > include and enable psyco if it's available when building an .exe. You > would, in theory, get a single file distribution as well as dynamic > compilation. I haven't tried it, but using psyco in a script and building an exe from it with py2exe should work right out of the box - but maybe this isn't what you had in mind? And single-file exes with py2exe are on my plan for a medium time range, although currently I'm not really sure which way to go. Thomas From mwh at python.net Fri Dec 10 12:56:38 2004 From: mwh at python.net (Michael Hudson) Date: Fri Dec 10 12:56:39 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: (Jeremy Hylton's message of "Thu, 9 Dec 2004 14:56:41 -0500") References: <5.1.1.6.0.20041208172852.03020bd0@mail.telecommunity.com> <5.1.1.6.0.20041209131843.038f3530@mail.telecommunity.com> Message-ID: <2mfz2epds9.fsf@starship.python.net> Jeremy Hylton writes: > I agree, although it's not clear to me how much faster it will be in > the future. Making a *fast* Python based on our own virtual execution > environment (as opposed to piggybacking a JVM or CLR) is a big > project. It's not clear than anyone has enough resources to make > credible effort I think the EU does! Cheers, mwh -- NUTRIMAT: That drink was individually tailored to meet your personal requirements for nutrition and pleasure. ARTHUR: Ah. So I'm a masochist on a diet am I? -- The Hitch-Hikers Guide to the Galaxy, Episode 9 From mwh at python.net Fri Dec 10 13:02:59 2004 From: mwh at python.net (Michael Hudson) Date: Fri Dec 10 13:03:01 2004 Subject: [Python-Dev] PyOS_InputHook enhancement proposal In-Reply-To: <41B91ADF.6040800@kdart.com> (Keith Dart's message of "Thu, 09 Dec 2004 19:41:19 -0800") References: <41B7D8C5.9000008@ims.u-tokyo.ac.jp> <20041209131555.GB10465@panix.com> <41B91ADF.6040800@kdart.com> Message-ID: <2mbrd2pdho.fsf@starship.python.net> Keith Dart writes: > I did modify the readline module that hooks this and can call back > to a Python function. There are also methods for installing and > removing the Python function. I did this for a different reason. I > need Python signal handlers to run, and they don't run when the > execution path is currently in some C code (such as readline). The > hook forces the interpreter to run, and check for signals as a side > effect. Using this I can be sitting in raw_input(), or interactive > mode, and signal handlers are still run. Have you seen what happened to the signal handling code in readline in Python 2.4? I don't think your modifications will be needed anymore, but you should check. Cheers, mwh -- I don't remember any dirty green trousers. -- Ian Jackson, ucam.chat From p.f.moore at gmail.com Fri Dec 10 13:06:01 2004 From: p.f.moore at gmail.com (Paul Moore) Date: Fri Dec 10 13:06:04 2004 Subject: [Python-Dev] The other Py2.4 issue In-Reply-To: <20041210100659.GB15865@vicky.ecs.soton.ac.uk> References: <001401c4dda9$ad605680$e841fea9@oemcomputer> <20041210100659.GB15865@vicky.ecs.soton.ac.uk> Message-ID: <79990c6b041210040671584abc@mail.gmail.com> On Fri, 10 Dec 2004 10:06:59 +0000, Armin Rigo wrote: > For people like myself, Linux programmers not developing on Windows every day, > there is precious little information available about how to compile our > extension modules for the new Windows distribution. I was actually very > disappointed to have to google my way around until I found a page that > explained to me that to use Microsoft's free compilers you need to manually > patch the distutils included with 2.4. (I know this is being worked on right > now but I'd have expected it to be present in the 2.4 release.) (The only > page I could find still refers to a pre-final 2.4 so their distutils patch > doesn't apply without hand-editing, though that should be fixed by now.) [...] > In other words, if you want 3rd parties to compile Windows binaries for 2.4, > tell them how. I think that the details about the Microsoft free compilers is a bit misleading. Sure, it's great that it exists, but at the present time, it is not the best free option (too many downloads required, too many rough edges). For most C extensions, the best free option is mingw. This is fully supported by distutils, and has been for a while. It is documented in the manuals, but basically all you need is to do the following: 1. Obtain a reasonably recent mingw (3.3.3 definitely works, you need msvcr71 support). 2. Install it, and make sure it is in your PATH. 3. Build libpython24.a as documented in the Python manuals (recent versions of mingw can read MS import libraries, so this may no longer be needed, but I haven't tested this yet). 4. python setup.py build --compiler=mingw32. The only difficulties are: 1. Obtaining support libraries like jpeg, sdl, etc. It *may* be a nuisance getting mingw-compatible build, but as I say above, MS-format libraries may now be fine, and there are good sources of mingw-compatible libraries like gnuwin32.sourceforge.net. 2. Microsoft-specific code. (As I'm replying to Armin, maybe the fact that inline assembler formats are different is relevant :-) :-)) To be honest, I'd only ever use the MS free compilers if I had a critical need to build an extension which had some highly MS-specific code or build processes. And even then, I'd probably give up and wait for someone with MSVC7 to produce a binary... If anyone has any particular modules (of their own, or third party) they have problems with, I'd be happy to have a go at building, and report back. Maybe a page on the Python Wiki about building modules using mingw would be worth adding. Hmm, might do that tonight... Hope this helps, Paul. From skip at pobox.com Fri Dec 10 16:10:41 2004 From: skip at pobox.com (Skip Montanaro) Date: Fri Dec 10 16:10:52 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: References: <86F33E75-498A-11D9-BB31-000D93C603C0@indranet.co.nz> <20041210094128.GA15865@vicky.ecs.soton.ac.uk> <16825.32570.898575.571826@montanaro.dyndns.org> Message-ID: <16825.48241.997055.384959@montanaro.dyndns.org> Thomas> I haven't tried it, but using psyco in a script and building an Thomas> exe from it with py2exe should work right out of the box - but Thomas> maybe this isn't what you had in mind? I was thinking of implicitly mixing in psyco, even if the script didn't use it. Maybe I have too much faith in psyco. ;-) Skip From pje at telecommunity.com Fri Dec 10 16:17:42 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri Dec 10 16:19:19 2004 Subject: [Python-Dev] The other Py2.4 issue In-Reply-To: <20041210100659.GB15865@vicky.ecs.soton.ac.uk> References: <001401c4dda9$ad605680$e841fea9@oemcomputer> <001401c4dda9$ad605680$e841fea9@oemcomputer> Message-ID: <5.1.1.6.0.20041210101609.020ed6e0@mail.telecommunity.com> At 10:06 AM 12/10/04 +0000, Armin Rigo wrote: >Hi, > >On Wed, Dec 08, 2004 at 11:43:49PM -0500, Raymond Hettinger wrote: > > Acceptance for Py2.4 partially hinges on how quickly third party apps > > have their binaries updated. > > > > I wonder if there is anything we can do to help. > >For people like myself, Linux programmers not developing on Windows every day, >there is precious little information available about how to compile our >extension modules for the new Windows distribution. I personally find MinGW (by way of Cygwin) to be the easiest way to do it. I posted here previously with the procedure and even a script to prepare the necessary libpython24.a file. From arigo at tunes.org Fri Dec 10 18:07:50 2004 From: arigo at tunes.org (Armin Rigo) Date: Fri Dec 10 18:16:55 2004 Subject: [Python-Dev] 2.4 news reaches interesting places In-Reply-To: <16825.32570.898575.571826@montanaro.dyndns.org> References: <86F33E75-498A-11D9-BB31-000D93C603C0@indranet.co.nz> <20041210094128.GA15865@vicky.ecs.soton.ac.uk> <16825.32570.898575.571826@montanaro.dyndns.org> Message-ID: <20041210170750.GA12032@vicky.ecs.soton.ac.uk> Hi Skip, On Fri, Dec 10, 2004 at 04:49:30AM -0600, Skip Montanaro wrote: > > >> The other thing we can do is finish the portable backend for psyco > >> and make it a standard module. Then Python won't be slow, it will be > >> compiled, and py2exe will be able to make a single-file executable. > > Armin> You probably mean that Psyco can dynamically compile Python > Armin> bytecodes even if they have been hidden into an .exe file by > Armin> py2exe. > > I didn't read it that way. My impression was that py2exe be modified to > include and enable psyco if it's available when building an .exe. You > would, in theory, get a single file distribution as well as dynamic > compilation. Yes, I agree with this. What I meant is that when I first read the original paragraph (the 1st one quoted above), I thought it meant that in the future py2exe and Psyco could be combined in such a way that we'd essentially have a compiler from Python producing a "classical" compiled binary. A lot of people could read it that way. The question is if we should advertise a Psyco+py2exe combination using a similar wording, such that it superficially sounds like we are doing a "classical" compilation from .py to .exe, whereas it actually means that we are hiding Psyco with the bytecodes in a .exe. After all, from a user's (or journalist's) point of view the result is similar, performancewise. Whether Psyco is reliable enough for this is yet another issue... I'd classify Psyco as "mostly reliable" only... A bientot, Armin. From arigo at tunes.org Fri Dec 10 18:19:21 2004 From: arigo at tunes.org (Armin Rigo) Date: Fri Dec 10 18:28:22 2004 Subject: [Python-Dev] The other Py2.4 issue In-Reply-To: <79990c6b041210040671584abc@mail.gmail.com> References: <001401c4dda9$ad605680$e841fea9@oemcomputer> <20041210100659.GB15865@vicky.ecs.soton.ac.uk> <79990c6b041210040671584abc@mail.gmail.com> Message-ID: <20041210171921.GB12032@vicky.ecs.soton.ac.uk> Hi, On Fri, Dec 10, 2004 at 12:06:01PM +0000, Paul Moore wrote: > For most C extensions, the best free option is mingw. Sorry, I was not aware that mingw supports the new VC7.1-type of runtime that is needed for the extension module to load with the official Python 2.4 distribution. Note that compiling with Microsoft's free compiler (after the huge downloads and the manual patching) worked just fine, so I think it is worth a note somewhere. Another note: can you report on whether building libpython24.a can be skipped for mingw? I'm thinking about the specific situation where we want on-site compilation of extension modules with a minimal number of things to install first. E.g. if we need to compile libpython24.a it means we need to fetch the Python sources themselves first. (A use case is for prorgams that dynamically produce and compile extension modules, as "weave" or the PyPy test suite do.) A bientot, Armin. From pje at telecommunity.com Fri Dec 10 19:05:17 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri Dec 10 19:07:03 2004 Subject: [Python-Dev] The other Py2.4 issue In-Reply-To: <20041210171921.GB12032@vicky.ecs.soton.ac.uk> References: <79990c6b041210040671584abc@mail.gmail.com> <001401c4dda9$ad605680$e841fea9@oemcomputer> <20041210100659.GB15865@vicky.ecs.soton.ac.uk> <79990c6b041210040671584abc@mail.gmail.com> Message-ID: <5.1.1.6.0.20041210125911.0283dec0@mail.telecommunity.com> At 05:19 PM 12/10/04 +0000, Armin Rigo wrote: >Another note: can you report on whether building libpython24.a can be skipped >for mingw? I'm thinking about the specific situation where we want on-site >compilation of extension modules with a minimal number of things to install >first. E.g. if we need to compile libpython24.a it means we need to fetch the >Python sources themselves first. Actually, no, you don't need the sources. See: http://mail.python.org/pipermail/python-dev/2004-January/041676.html for a script that builds libpython24.a from the python24.lib distributed with Python for Windows. From bob at redivi.com Fri Dec 10 19:12:04 2004 From: bob at redivi.com (Bob Ippolito) Date: Fri Dec 10 19:12:42 2004 Subject: [Python-Dev] The other Py2.4 issue In-Reply-To: <5.1.1.6.0.20041210125911.0283dec0@mail.telecommunity.com> References: <79990c6b041210040671584abc@mail.gmail.com> <001401c4dda9$ad605680$e841fea9@oemcomputer> <20041210100659.GB15865@vicky.ecs.soton.ac.uk> <79990c6b041210040671584abc@mail.gmail.com> <5.1.1.6.0.20041210125911.0283dec0@mail.telecommunity.com> Message-ID: On Dec 10, 2004, at 1:05 PM, Phillip J. Eby wrote: > At 05:19 PM 12/10/04 +0000, Armin Rigo wrote: > >> Another note: can you report on whether building libpython24.a can be >> skipped >> for mingw? I'm thinking about the specific situation where we want >> on-site >> compilation of extension modules with a minimal number of things to >> install >> first. E.g. if we need to compile libpython24.a it means we need to >> fetch the >> Python sources themselves first. > > Actually, no, you don't need the sources. See: > > http://mail.python.org/pipermail/python-dev/2004-January/041676.html > > for a script that builds libpython24.a from the python24.lib > distributed with Python for Windows. Shouldn't this be distributed with binary distributions of Python, to save people the trouble? Or, if it can be skipped, the procedure for doing a mingw build with the .lib should be documented and that'd be the end of it. -bob From pje at telecommunity.com Fri Dec 10 19:19:10 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri Dec 10 19:20:58 2004 Subject: [Python-Dev] The other Py2.4 issue In-Reply-To: References: <5.1.1.6.0.20041210125911.0283dec0@mail.telecommunity.com> <79990c6b041210040671584abc@mail.gmail.com> <001401c4dda9$ad605680$e841fea9@oemcomputer> <20041210100659.GB15865@vicky.ecs.soton.ac.uk> <79990c6b041210040671584abc@mail.gmail.com> <5.1.1.6.0.20041210125911.0283dec0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20041210131425.030fd360@mail.telecommunity.com> At 01:12 PM 12/10/04 -0500, Bob Ippolito wrote: >On Dec 10, 2004, at 1:05 PM, Phillip J. Eby wrote: >>At 05:19 PM 12/10/04 +0000, Armin Rigo wrote: >> >>>Another note: can you report on whether building libpython24.a can be >>>skipped >>>for mingw? I'm thinking about the specific situation where we want on-site >>>compilation of extension modules with a minimal number of things to install >>>first. E.g. if we need to compile libpython24.a it means we need to >>>fetch the >>>Python sources themselves first. >> >>Actually, no, you don't need the sources. See: >> >>http://mail.python.org/pipermail/python-dev/2004-January/041676.html >> >>for a script that builds libpython24.a from the python24.lib distributed >>with Python for Windows. > >Shouldn't this be distributed with binary distributions of Python, to save >people the trouble? The Python developers who produce the Windows binaries don't use mingw/cygwin, so this would put a maintenance burden on them. >Or, if it can be skipped, the procedure for doing a mingw build with the >.lib should be documented and that'd be the end of it. It's actually documented now, in the "Installing Python Modules" manual. See: http://docs.python.org/inst/tweak-flags.html#SECTION000622000000000000000 It just would need to have the libpython.a instructions removed in that case. From amk at amk.ca Fri Dec 10 20:01:55 2004 From: amk at amk.ca (A.M. Kuchling) Date: Fri Dec 10 20:02:27 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: <20041209141104.GA18348@phd.pp.ru> References: <000b01c4dda2$fe7a87e0$e841fea9@oemcomputer> <20041209102250.GB12506@phd.pp.ru> <20041209141104.GA18348@phd.pp.ru> Message-ID: <20041210190155.GA11443@rogue.amk.ca> On Thu, Dec 09, 2004 at 05:11:04PM +0300, Oleg Broytmann wrote: > some popular areas. Let's add another topic, "Making things fast". Let's > even make it the first topic, though I personnaly dont see a need for > this. The topic guides are migrating into the Wiki, and there's already a Wiki page about this: http://www.python.org/moin/PythonSpeed --amk From phd at mail2.phd.pp.ru Fri Dec 10 20:07:58 2004 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Fri Dec 10 20:08:00 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: <20041210190155.GA11443@rogue.amk.ca> References: <000b01c4dda2$fe7a87e0$e841fea9@oemcomputer> <20041209102250.GB12506@phd.pp.ru> <20041209141104.GA18348@phd.pp.ru> <20041210190155.GA11443@rogue.amk.ca> Message-ID: <20041210190758.GC16678@phd.pp.ru> On Fri, Dec 10, 2004 at 02:01:55PM -0500, A.M. Kuchling wrote: > On Thu, Dec 09, 2004 at 05:11:04PM +0300, Oleg Broytmann wrote: > > some popular areas. Let's add another topic, "Making things fast". Let's > > even make it the first topic, though I personnaly dont see a need for > > this. > > The topic guides are migrating into the Wiki, and there's already a Wiki page > about this: http://www.python.org/moin/PythonSpeed Thank you. Actually, I've thought about wiki when I was writing that email, but I was afarid that people who (in my very humble opinion) are overreacting - will not tolerate putting such an important topic into a wiki. My personnal opinion is that a good topic about the speed of Python in the wiki is enough. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From jhylton at gmail.com Fri Dec 10 20:28:13 2004 From: jhylton at gmail.com (Jeremy Hylton) Date: Fri Dec 10 20:28:16 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: <20041210190155.GA11443@rogue.amk.ca> References: <000b01c4dda2$fe7a87e0$e841fea9@oemcomputer> <20041209102250.GB12506@phd.pp.ru> <20041209141104.GA18348@phd.pp.ru> <20041210190155.GA11443@rogue.amk.ca> Message-ID: On Fri, 10 Dec 2004 14:01:55 -0500, A.M. Kuchling wrote: > On Thu, Dec 09, 2004 at 05:11:04PM +0300, Oleg Broytmann wrote: > > some popular areas. Let's add another topic, "Making things fast". Let's > > even make it the first topic, though I personnaly dont see a need for > > this. > > The topic guides are migrating into the Wiki, and there's already a Wiki page > about this: http://www.python.org/moin/PythonSpeed The Wiki entry seems to reinforce the impression that bugged Guido to begin with. It provides a bunch of "but ..." explanations about why Python's speed isn't that important. Python is slow, but "speed of development is far more important." If you want to avoid people saying "Python is slow, but ..." you need to give them a different message. "Writing code quickly is crucial in today's marketplace." (whatever, that's not much of a message.) David's dynamic languages whitepaper has more of message, for example. Jeremy From p.f.moore at gmail.com Fri Dec 10 21:40:10 2004 From: p.f.moore at gmail.com (Paul Moore) Date: Fri Dec 10 21:40:13 2004 Subject: [Python-Dev] The other Py2.4 issue In-Reply-To: <20041210171921.GB12032@vicky.ecs.soton.ac.uk> References: <001401c4dda9$ad605680$e841fea9@oemcomputer> <20041210100659.GB15865@vicky.ecs.soton.ac.uk> <79990c6b041210040671584abc@mail.gmail.com> <20041210171921.GB12032@vicky.ecs.soton.ac.uk> Message-ID: <79990c6b041210124039803de9@mail.gmail.com> On Fri, 10 Dec 2004 17:19:21 +0000, Armin Rigo wrote: > Another note: can you report on whether building libpython24.a can be skipped > for mingw? A first attempt seems to almost work. There is a problem with structures, however - I get an error about unresolved references to _imp___Py_NoneStruct. My guess is that there's some issue with resolving references to data (as opposed to functions) from DLLs. I'll investigate further and then report back. But this is specifically if you *don't* build libpytho24.a. If you build that, there's no problem at all. Paul. From p.f.moore at gmail.com Fri Dec 10 23:20:29 2004 From: p.f.moore at gmail.com (Paul Moore) Date: Fri Dec 10 23:20:32 2004 Subject: [Python-Dev] The other Py2.4 issue In-Reply-To: <79990c6b041210124039803de9@mail.gmail.com> References: <001401c4dda9$ad605680$e841fea9@oemcomputer> <20041210100659.GB15865@vicky.ecs.soton.ac.uk> <79990c6b041210040671584abc@mail.gmail.com> <20041210171921.GB12032@vicky.ecs.soton.ac.uk> <79990c6b041210124039803de9@mail.gmail.com> Message-ID: <79990c6b0412101420522d59f3@mail.gmail.com> On Fri, 10 Dec 2004 20:40:10 +0000, Paul Moore wrote: > On Fri, 10 Dec 2004 17:19:21 +0000, Armin Rigo wrote: > > Another note: can you report on whether building libpython24.a can be skipped > > for mingw? > > A first attempt seems to almost work. There is a problem with > structures, however - I get an error about unresolved references to > _imp___Py_NoneStruct. My guess is that there's some issue with > resolving references to data (as opposed to functions) from DLLs. I'll > investigate further and then report back. OK. After some more extensive investigation: 1. You cannot use python24.lib (ie, skip building libpython24.a). If you do, data values exported from python24.dll (specifically, Py_NoneStruct, which is quite important :-)) do not get exported correctly. I don't know exactly why - I guess it's something to do with how the MS LIB format works. 2. You can, however, link direct to python24.dll, which works fine. Unfortunately, there are 2 problems with this - first, distutils doesn't support it as things stand, and second, it isn't possible to get the full path of the Python DLL without using Mark Hammond's pywin32, or custom C code :-( (You need to get from sys.dllhandle to the filename, and the API to do this isn't exposed). 3. In some cases, an extension built with mingw still contains references to msvcrt (I think Martin has noted this). This seems *not* to be an issue. What is happening, is that any symbols referenced from user code are satisfied from msvcr71, and so match the Python DLL. However, if the mingw startup code uses symbols which have not been satisfied by the time all references from user code are resolved, then these references are satisfied from msvcrt. But I can't see any way that these references are going to do any harm, as they are startup code, which by its nature will have been written to work cleanly in this situation (linking like this is explicitly supported by mingw). Martin, as the CRT compatibility expert, does this sound reasonable? The mingw people certainly seem to think that the situation is OK... You could probably incorporate Philip Eby's script, posted here a while ago, which he mentioned earlier in this thread, into a setup.py so that if libpython24.a didn't already exist, the setup script would build it. This would give you a reasonably clean way of making sure it was present. Philip's script doesn't need anything but mingw installed, AIUI. Longer term, I'll see if I can put together a distutils patch to make it reference pythonXX.dll directly, so removing the need for libpythonXX.a in the future. Obviously, this is only going to be relevant for Python 2.5 and later, though. Sorry for the long mail, but I thought it would be useful to get the details in the archives... Paul. From python at rcn.com Fri Dec 10 23:42:29 2004 From: python at rcn.com (Raymond Hettinger) Date: Fri Dec 10 23:45:36 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: Message-ID: <001d01c4df09$87e5caa0$e841fea9@oemcomputer> > The Wiki entry seems to reinforce the impression that bugged Guido to > begin with. It provides a bunch of "but ..." explanations about why > Python's speed isn't that important. Python is slow, but "speed of > development is far more important." I felt the same way when reading it. Also, it seemed to embody the political outlook that optimization is sinful. The document could be much more positive, fact based, and informative. Also, the wording seems somewhat outdated. A draft for a new entry is included below. Review and feedback are requested. Raymond Techniques for Improving Performance and Scalability ==================================================== Here are some coding guidelines for applications that demand peek performance (in terms of memory utilization, speed, or scalability). Use the best algorithms and fastest tools ----------------------------------------- . Membership testing with sets and dictionaries is much faster, O(1), than searching sequences, O(n). When testing "a in b", b should be a set or dictionary instead of a list or tuple. . String concatenation is best done with ''.join(seq) which is an O(n) process. In contrast, using the '+' or '+=' operators can result in an O(n**2) process because new strings may be built for each intermediate step. Py2.4 mitigates this issue somewhat; however, ''.join(seq) remains the best practice. . Many tools come in both list form and iterator form (map and itertools.imap, list comprehension and generator expressions, dict.items and dict.iteritems). In general, the iterator forms are more memory friendly and more scalable. They are preferred whenever a real list is not required. . Many core building blocks are coded in optimized C. Applications that take advantage of them can make substantial performance gains. The building blocks include all of the builtin datatypes (lists, tuples, sets, and dictionaries) and extension modules like array, itertools, and collections.deque. . Likewise, the builtin functions run faster than hand-built equivalents. For example, map(operator.add, v1, v2) is faster than map(lambda x,y: x+y, v1, v2). . List comprehensions run a bit faster than equivalent for-loops. . Lists perform well as fixed length arrays and as stacks. For queue applications using pop(0) or insert(0,v)), collections.deque() offers superior O(1) performance because it avoids rebuilding a full list for each insertion or deletion, an O(n) process. . Custom sort ordering is best performed with Py2.4's key= option or with the traditional decorate-sort-undecorate technique. Both approaches call the key function just once per element. In contrast, sort's cmp= option is called many times per element during a sort. For example, sort(key=str.lower) is faster than sort(cmp=lambda a,b: cmp(a.lower(), b.lower())). Take advantage of interpreter optimizations ------------------------------------------- . In functions, local variables are accessed more quickly than global variables, builtins, and attribute lookups. So, it is sometimes worth localizing variable access in inner-loops. For example, the code for random.shuffle() localizes access with the line, random=self.random. That saves the shuffling loop from having to repeatedly lookup self.random. Outside of loops, the gain is minimal and rarely worth it. . The previous recommendation is a generalization of the rule to factor constant expressions out of loops. Also, constant folding should be done manually. Inside loops, write "x=3" instead of "x=1+2". . Function call overhead is large compared to other instructions. Accordingly, it is sometimes worth in-lining code in the middle of a few time-critical loops. . Starting with Py2.3, the interpreter optimizes "while 1" to just a single jump. In contrast "while True" takes several more steps. While the latter is preferred for clarity, time critical code should use the first form. . Multiple assignment is slower than individual assignment. For example "x,y=a,b" is slower than "x=a; y=b". However, multiple assignment is faster for variable swaps. For example, "x,y=y,x" is faster than "t=x; x=y; y=t". . Chained comparisons are faster than using the "and" operator. Write "x < y < z" instead of "x < y and y < z". . A few fast approaches should be considered hacks and reserved for only the most demanding applications. For example, "x and True or False" is faster than "bool(x)". Take advantage of diagnostic tools ---------------------------------- . The hotshot and profile modules help identify performance bottlenecks. . The timeit module offers immediate performance comparisons between alternative approaches to writing individual lines of code. Performance can dictate overall strategy ---------------------------------------- . SAX is typically faster and more memory efficient than DOM approaches to XML. . Threading can improve the response time in applications that would otherwise waste time waiting for I/O. . Select can help minimize the overhead for polling multiple sockets. From martin at v.loewis.de Sat Dec 11 00:45:01 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Dec 11 00:45:06 2004 Subject: [Python-Dev] The other Py2.4 issue In-Reply-To: <20041210100659.GB15865@vicky.ecs.soton.ac.uk> References: <001401c4dda9$ad605680$e841fea9@oemcomputer> <20041210100659.GB15865@vicky.ecs.soton.ac.uk> Message-ID: <41BA34FD.6050306@v.loewis.de> Armin Rigo wrote: > In other words, if you want 3rd parties to compile Windows binaries for 2.4, > tell them how. The straight-forward answer is: Get VC7.1 (aka VS.NET 2003), and invoke python setup.py bdist_wininst That's not too hard to do, I think. Regards, Martin From martin at v.loewis.de Sat Dec 11 00:52:52 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Dec 11 00:52:55 2004 Subject: [Python-Dev] The other Py2.4 issue In-Reply-To: <5.1.1.6.0.20041210131425.030fd360@mail.telecommunity.com> References: <5.1.1.6.0.20041210125911.0283dec0@mail.telecommunity.com> <79990c6b041210040671584abc@mail.gmail.com> <001401c4dda9$ad605680$e841fea9@oemcomputer> <20041210100659.GB15865@vicky.ecs.soton.ac.uk> <79990c6b041210040671584abc@mail.gmail.com> <5.1.1.6.0.20041210125911.0283dec0@mail.telecommunity.com> <5.1.1.6.0.20041210131425.030fd360@mail.telecommunity.com> Message-ID: <41BA36D4.2080903@v.loewis.de> Phillip J. Eby wrote: > The Python developers who produce the Windows binaries don't use > mingw/cygwin, so this would put a maintenance burden on them. If this were completely automatic (e.g. part of msi.py), I'd happily add all utilities needed to integrated this into the build process. For this to happen: 1. Somebody else needs to do the actual work. I.e. write a patch to msi.py that invokes all the necessary magic, and test that patch. 2. The patch should avoid manual configuration of the "use cygwin" case. I.e. it should try to find the cygwin on the path, or in their standard location, or use any other mechanism available to locate cygwin. It should produce an error message if cygwin tools cannot be found, and then proceed without out creating libpython24.a Regards, Martin From chui.tey at advdata.com.au Fri Dec 10 12:53:16 2004 From: chui.tey at advdata.com.au (Chui G. Tey) Date: Sat Dec 11 04:24:58 2004 Subject: [Python-Dev] Supporting Third Party Modules (was The other Py2.4 issue) Message-ID: >Raymond: > >Acceptance for Py2.4 partially hinges on how quickly third party apps >have their binaries updated. > >I wonder if there is anything we can do to help. One good way of helping out is to provide an dynamic loading function that third party modules could access the basic python functions such as PyArgParseTuple, PyString_AsString etc regardless of which python the user is running. This would be similar to the COM approach. You can load all the function pointers into a struct and then call them. Third party modules would link against this DLL independent of which python is being used. Chui From gvwilson at cs.toronto.edu Tue Dec 7 21:32:58 2004 From: gvwilson at cs.toronto.edu (Greg Wilson) Date: Sat Dec 11 04:25:52 2004 Subject: [Python-Dev] decorators article for DDJ? Message-ID: Hi folks. Is anyone interested in doing an article on Python decorators for "Doctor Dobb's Journal"? If so, please drop me a line... Thanks, Greg From neal at metaslash.com Thu Dec 9 03:33:50 2004 From: neal at metaslash.com (Neal Norwitz) Date: Sat Dec 11 04:26:49 2004 Subject: [Python-Dev] removing aclocal.m4 Message-ID: <20041209023350.GB7217@janus.swcomplete.com> Can we remove aclocal.m4? The last log message states: fix for bug #811160 - autoconf vs. hp/ux system header files. also applied to release23-maint. Note that aclocal.m4 can go away when autoconf 2.58 is out. And configure says: # Generated by GNU Autoconf 2.59 for python 2.5. Neal From ncoghlan at iinet.net.au Sat Dec 11 05:41:24 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Dec 11 05:41:31 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: <001d01c4df09$87e5caa0$e841fea9@oemcomputer> References: <001d01c4df09$87e5caa0$e841fea9@oemcomputer> Message-ID: <41BA7A74.1010302@iinet.net.au> Raymond Hettinger wrote: > guidelines for applications that demand peek performance (in terms of memory Peak performance, perhaps? :) Anyway, it looks pretty good to me, but I have a few additional ideas. Add a section of Caveats (we know they exist - might as well be upfront about it): Caveats -------- . This document is based primarily on the CPython interpreter from www.python.org. However, most of the recommendations should apply to any Python interpreter that supports the mentioned feature. . For small data sets, some of the suggestions will perform more slowly than the approaches they are given as alternatives too. This is due to the fact that those suggestions introduce a little more fixed overhead in order to avoid overhead in processing each data item. The crossover point (where the more scaleable approach begins to show a net performance gain) is generally application specific. Use the diagnostic tools mentioned later to make an informed decision for your application. > Use the best algorithms and fastest tools > ----------------------------------------- > . Membership testing with sets and dictionaries is much faster, > O(1), than searching sequences, O(n). When testing "a in b", b should > be a set or dictionary instead of a list or tuple. Should the bisect module be mentioned? If you have a sorted list already, using bisect may be faster than constructing an intermediate set. Also, you can use bisect if an item's *position* in the list is important. list.index(x) uses a linear search (since it can't assume a sorted list), whereas bisect expects the list to be pre-sorted. > intermediate step. Py2.4 mitigates this issue somewhat; however, > ''.join(seq) remains the best practice. I'd say 'The CPython 2.4 interpreter', rather than just Py2.4. > . List comprehensions run a bit faster than equivalent for-loops. I'd move this to the next section (the algorithms are the same, but the interpreter can speed up one more than the other) > . Custom sort ordering is best performed with Py2.4's key= option or with the > traditional decorate-sort-undecorate technique. Both approaches call the key > function just once per element. In contrast, sort's cmp= option is called > many times per element during a sort. For example, sort(key=str.lower) is > faster than sort(cmp=lambda a,b: cmp(a.lower(), b.lower())). If sorting is going to be performed repeatedly (e.g. maintaining a list in sorted order), it may be feasible to store the list of keys, rather than regenerating them every time. This also plays nicely with using the bisect module to update the list (as the list of keys is available to determine the correct index for insertion). I can't find a 'sorted list' recipe in the Cookbook, so I might have to create one. > Take advantage of interpreter optimizations > ------------------------------------------- [snip] . Prefer iteration (using for loops, list comprehensions, or generator expressions) over iterables to explicit invocation of .next() > Take advantage of diagnostic tools > ---------------------------------- > > . The hotshot and profile modules help identify performance bottlenecks. Currently profile is a fair bit more useful than hotshot, mainly due to the fact that it isolates time spent in C functions from the Python code that calls those functions. Perhaps another section for External Libraries? If you're doing serious number crunching in Python, using NumPy is practically a requirement. Cheers, Nick. -- Nick Coghlan | ncoghlan@email.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net From mdehoon at ims.u-tokyo.ac.jp Sat Dec 11 07:13:39 2004 From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon) Date: Sat Dec 11 07:10:24 2004 Subject: [Python-Dev] The other Py2.4 issue In-Reply-To: <5.1.1.6.0.20041210125911.0283dec0@mail.telecommunity.com> References: <79990c6b041210040671584abc@mail.gmail.com> <001401c4dda9$ad605680$e841fea9@oemcomputer> <20041210100659.GB15865@vicky.ecs.soton.ac.uk> <79990c6b041210040671584abc@mail.gmail.com> <5.1.1.6.0.20041210125911.0283dec0@mail.telecommunity.com> Message-ID: <41BA9013.50101@ims.u-tokyo.ac.jp> >> E.g. if we need to compile libpython24.a it means we need to >> fetch the Python sources themselves first. > > Actually, no, you don't need the sources. See: > > http://mail.python.org/pipermail/python-dev/2004-January/041676.html > > for a script that builds libpython24.a from the python24.lib distributed > with Python for Windows. > Previously I built libpython24.a; it can be downloaded from my website. See http://bonsai.ims.u-tokyo.ac.jp/~mdehoon/software/python/cygwin.html --Michiel. -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon From martin at v.loewis.de Sat Dec 11 09:39:28 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Dec 11 09:39:31 2004 Subject: [Python-Dev] Supporting Third Party Modules (was The other Py2.4 issue) In-Reply-To: References: Message-ID: <41BAB240.6060101@v.loewis.de> Chui G. Tey wrote: > One good way of helping out is to provide an dynamic loading function > that third party modules could access the basic python functions such as > PyArgParseTuple, PyString_AsString etc regardless of which python the > user is running. This would be similar to the COM approach. You can load > all the function pointers into a struct and then call them. > > Third party modules would link against this DLL independent of which > python is being used. I believe this is not implementable: How can the DLL know which Python DLL to use? Regards, Martin From martin at v.loewis.de Sat Dec 11 09:53:29 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Dec 11 09:53:32 2004 Subject: [Python-Dev] removing aclocal.m4 In-Reply-To: <20041209023350.GB7217@janus.swcomplete.com> References: <20041209023350.GB7217@janus.swcomplete.com> Message-ID: <41BAB589.4030303@v.loewis.de> Neal Norwitz wrote: > Can we remove aclocal.m4? The last log message states: > > fix for bug #811160 - autoconf vs. hp/ux system header files. > also applied to release23-maint. > > Note that aclocal.m4 can go away when autoconf 2.58 is out. It appears to me that 2.59 indeed fixes the HP-UX problem; a diff with and without aclocal.m4 shows chunks like cat confdefs.h >>conftest.$ac_ext cat >>conftest.$ac_ext <<_ACEOF /* end confdefs.h. */ +/* Define $ac_func to an innocuous variant, in case declares $ac_func. + For example, HP-UX 11i declares gettimeofday. */ +#define $ac_func innocuous_$ac_func + /* System header to define __stub macros and hopefully few prototypes, which can conflict with char $ac_func (); below. Prefer to if __STDC__ is defined, since - exists even on freestanding compilers. Under hpux, - including includes and causes problems - checking for functions defined therein. */ -#if defined (__STDC__) && !defined (_HPUX_SOURCE) + exists even on freestanding compilers. */ + +#ifdef __STDC__ # include #else # include #endif + +#undef $ac_func + So they manage to get limits.h define, say, innoucuous_clock(), instead of clock(), whereas we currently avoid including limits.h on HP-UX. IOW, it seems like it should work, but somebody should test it on HP-UX to be sure. Regards, Martin From bob at redivi.com Sat Dec 11 10:51:30 2004 From: bob at redivi.com (Bob Ippolito) Date: Sat Dec 11 10:52:07 2004 Subject: [Python-Dev] Supporting Third Party Modules (was The other Py2.4 issue) In-Reply-To: <41BAB240.6060101@v.loewis.de> References: <41BAB240.6060101@v.loewis.de> Message-ID: <3BBC89A1-4B5A-11D9-80F2-000A9567635C@redivi.com> On Dec 11, 2004, at 3:39 AM, Martin v. L?wis wrote: > Chui G. Tey wrote: >> One good way of helping out is to provide an dynamic loading function >> that third party modules could access the basic python functions such >> as >> PyArgParseTuple, PyString_AsString etc regardless of which python the >> user is running. This would be similar to the COM approach. You can >> load >> all the function pointers into a struct and then call them. Third >> party modules would link against this DLL independent of which >> python is being used. > > I believe this is not implementable: How can the DLL know which Python > DLL to use? Well for py2app on Mac OS X, I wrote an executable stub that chooses a Python runtime from an XML file, looks up and binds a few symbols from it dynamically, and then starts doing stuff. Works for 2.3 (as long as it's not compiled debug or trace refs) and 2.4 (universally, because it has exported functions for incref and decref). For extension modules, this makes much less sense, but I wanted to be able to bootstrap Python programs with a precompiled executable that I didn't have to maintain across several versions. -bob From fredrik at pythonware.com Sat Dec 11 11:14:38 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat Dec 11 11:12:23 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places References: Message-ID: > One thing that bugs me: the article says 3 or 4 times that Python is > slow, each time with a refutation ("but it's so flexible", "but it's > fast enough") but still, they sure seem to harp on the point. fwiw, IDG's Computer Sweden, "sweden's leading IT-newspaper" has a surprisingly big Python article in their most recent issue: PYTHON FEELS WELL Better performance biggest news in 2.4 and briefly interviews swedish zope-developer Johan Carlsson and Python- Ware co-founder Håkan Karlsson. PS. the heading is a play on the swedish phrase "må pyton", which means "feel like crap"... From fredrik at pythonware.com Sat Dec 11 15:06:05 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat Dec 11 15:03:51 2004 Subject: [Python-Dev] Re: Supporting Third Party Modules (was The other Py2.4issue) References: <41BAB240.6060101@v.loewis.de> Message-ID: Martin v. Löwis wrote: >> Third party modules would link against this DLL independent of which >> python is being used. > > I believe this is not implementable: How can the DLL know which Python > DLL to use? the Python interpreter could initialize this DLL, using some suitable mechanism. alternatively, it could figure out what EXE you used to start the program in the first place, and figure out what Python DLL that EXE uses. my ExeMaker tool can do that, for example: http://effbot.org/zone/exemaker.htm (see the #!python.exe entry in the table) From faassen at infrae.com Sat Dec 11 15:07:27 2004 From: faassen at infrae.com (Martijn Faassen) Date: Sat Dec 11 15:07:30 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: References: Message-ID: <41BAFF1F.3060200@infrae.com> Fredrik Lundh wrote: [snip] > fwiw, IDG's Computer Sweden, "sweden's leading IT-newspaper" has a > surprisingly big Python article in their most recent issue: > > PYTHON FEELS WELL > Better performance biggest news in 2.4 [snip] Perhaps the message getting out is actually that Python's performance is getting better, so nothing to worry about. The original article that Guido cited had a headline along the same lines: "Faster Python grabs programmers". So perhaps there really is nothing to worry about in this department: the headlines are saying "Python now faster". That to me sounds like an excellent way to combat "Python is slow" memes. :) Regards, Martijn From gvanrossum at gmail.com Sat Dec 11 17:22:17 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat Dec 11 17:22:20 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: References: Message-ID: > fwiw, IDG's Computer Sweden, "sweden's leading IT-newspaper" has a > surprisingly big Python article in their most recent issue: > > PYTHON FEELS WELL > Better performance biggest news in 2.4 > > and briefly interviews swedish zope-developer Johan Carlsson and Python- > Ware co-founder H?kan Karlsson. Maybe we've done this to ourselves then. I'm sure the interviewer didn't know what was the biggest news until after Johan and H?kan told him. And this despite all the functional improvements that are much more interesting for most Python programmers: generator expressions, decorators, a ton of new modules... Ah, but several of those modules are rewrites in C of modules that were previously written in Python. Maybe we really *should* stop emphasizing speed so much to ourselves, and focus more on fixing bugs and adding features instead of looking for speedup opportunities. I think Python's message ought to be more something like "programming for the rest of us -- a child can learn Python, but Python's no toy." BTW I strongly disagree that making easy .EXE binaries available will address this issue; while not bundled, there are plenty of solutions for maning .EXEs for those who need them, and this is not something that typically worries managers. But the perception of Python as "slow" does worry managers. (Wish I could add more to this thread but my family is calling...) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From phd at mail2.phd.pp.ru Sat Dec 11 17:27:14 2004 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Sat Dec 11 17:27:16 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: References: Message-ID: <20041211162714.GA32590@phd.pp.ru> On Sat, Dec 11, 2004 at 08:22:17AM -0800, Guido van Rossum wrote: > But the perception of Python as > "slow" does worry managers. Much more those managers are worried that Python isn't backed by a large corporation. For Java there is Sun, for Visual Basic there is the biggest and most powerful corporation. But who is for Python? Who is responsible? Whom to blame? Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From arigo at tunes.org Sat Dec 11 17:55:09 2004 From: arigo at tunes.org (Armin Rigo) Date: Sat Dec 11 18:04:11 2004 Subject: [Python-Dev] The other Py2.4 issue In-Reply-To: <41BA34FD.6050306@v.loewis.de> References: <001401c4dda9$ad605680$e841fea9@oemcomputer> <20041210100659.GB15865@vicky.ecs.soton.ac.uk> <41BA34FD.6050306@v.loewis.de> Message-ID: <20041211165508.GA10252@vicky.ecs.soton.ac.uk> Hi Martin, On Sat, Dec 11, 2004 at 12:45:01AM +0100, "Martin v. L?wis" wrote: > The straight-forward answer is: Get VC7.1 (aka VS.NET 2003), and invoke > > python setup.py bdist_wininst > > That's not too hard to do, I think. Hum, this is getting into a Linux-vs-Windows argument. I don't want to invest time and money on Windows tools just to compile my extension module for Windows users... Armin From arigo at tunes.org Sat Dec 11 17:55:16 2004 From: arigo at tunes.org (Armin Rigo) Date: Sat Dec 11 18:04:22 2004 Subject: [Python-Dev] The other Py2.4 issue In-Reply-To: <5.1.1.6.0.20041210131425.030fd360@mail.telecommunity.com> References: <5.1.1.6.0.20041210125911.0283dec0@mail.telecommunity.com> <79990c6b041210040671584abc@mail.gmail.com> <001401c4dda9$ad605680$e841fea9@oemcomputer> <20041210100659.GB15865@vicky.ecs.soton.ac.uk> <79990c6b041210040671584abc@mail.gmail.com> <5.1.1.6.0.20041210125911.0283dec0@mail.telecommunity.com> <5.1.1.6.0.20041210131425.030fd360@mail.telecommunity.com> Message-ID: <20041211165516.GB10252@vicky.ecs.soton.ac.uk> Hi, On Fri, Dec 10, 2004 at 01:19:10PM -0500, Phillip J. Eby wrote: > >>http://mail.python.org/pipermail/python-dev/2004-January/041676.html > > > >Shouldn't this be distributed with binary distributions of Python, to save > >people the trouble? > > The Python developers who produce the Windows binaries don't use > mingw/cygwin, so this would put a maintenance burden on them. This thread started as "how to convince people to build their extension modules quickly for Python 2.4". I was just suggesting that taking a bit of that burden away from us non-Windows-users would be helpful :-) > >Or, if it can be skipped, the procedure for doing a mingw build with the > >.lib should be documented and that'd be the end of it. > > It's actually documented now, in the "Installing Python Modules" manual. Oh right... Well, I didn't notice because, oddly, there is another set of pages on the same subject (but with different details and up-to-date-ness) in "Extending and Embedding the Python Interpreter". I guess that one of them should be removed and point to the other... A bientot, Armin. From pje at telecommunity.com Sat Dec 11 18:07:16 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat Dec 11 18:05:43 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: References: Message-ID: <5.1.1.6.0.20041211120241.03c12b50@mail.telecommunity.com> At 08:22 AM 12/11/04 -0800, Guido van Rossum wrote: >BTW I strongly disagree that making easy .EXE binaries available will >address this issue; while not bundled, there are plenty of solutions >for maning .EXEs for those who need them, and this is not something >that typically worries managers. But the perception of Python as >"slow" does worry managers. The relevant memes are that "compiled == fast", and that ".exe == compiled". It's not about .exe as a distribution format, but rather that producing an .exe is supporting evidence of having a "real" compiler. IOW, if there exists a compiler, but it doesn't produce .exe's, it won't be considered a "real" compiler by many programmers. From fredrik at pythonware.com Sat Dec 11 19:06:19 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat Dec 11 19:04:06 2004 Subject: [Python-Dev] Re: Re: 2.4 news reaches interesting places References: Message-ID: Guido van Rossum wrote: > > fwiw, IDG's Computer Sweden, "sweden's leading IT-newspaper" has a > > surprisingly big Python article in their most recent issue: > > > > PYTHON FEELS WELL > > Better performance biggest news in 2.4 > > (hmm. I seem to have accidentally deleted a line here...) > > and briefly interviews swedish zope-developer Johan Carlsson and Python- > > Ware co-founder Håkan Karlsson. > > Maybe we've done this to ourselves then. I'm sure the interviewer > didn't know what was the biggest news until after Johan and Håkan told > him. And this despite all the functional improvements that are much > more interesting for most Python programmers: generator expressions, > decorators, a ton of new modules... I don't know Johan, but I'm pretty sure Håkan didn't talk about performance; he's more interested in *development* speed... (and I don't think he's been following 2.4 that closely). Looking again, the article says: "Among the news in 2.4 is better performance, especially for new functions that were added in 2.3." and then mentions integer/long integer unification and the decimal type as notable additions, and continues "and so it continues, with a long list of improved details". The article also points to www.python.org/2.4, where the first link is: http://www.python.org/2.4/highlights.html which says: Here are the (subjective) highlights of what's new in Python 2.4. * Faster A number of modules that were added in Python 2.3 (such as sets and heapq) have been recoded in C. In addition, there's been a number of other speedups to the interpreter. (See section 8.1, Optimizations, of the "What's New" document for more [this lists a couple of thousand tweaks by Raymond, and Armin's string concatenation hack]). * New language features /.../ unification of integers and long integers /.../ decimal - a new numeric type that allows for the accurate representation of floating point numbers /.../ so I don't think you can blame Johan or Håkan... the writer simply read the python.org material, and picked a couple of things that he found interesting (decorators and generator expressions may be a big thing for an experienced pythoneer, but they are probably a bit too obscure for a general audience...) > Ah, but several of those modules are rewrites in C of modules that > were previously written in Python. Maybe we really *should* stop > emphasizing speed so much to ourselves, and focus more on fixing bugs > and adding features instead of looking for speedup opportunities. yes please. > (Wish I could add more to this thread but my family is calling...) same here. more later. From tismer at stackless.com Sat Dec 11 19:57:55 2004 From: tismer at stackless.com (Christian Tismer) Date: Sat Dec 11 19:57:57 2004 Subject: [Python-Dev] The other Py2.4 issue In-Reply-To: <20041211165508.GA10252@vicky.ecs.soton.ac.uk> References: <001401c4dda9$ad605680$e841fea9@oemcomputer> <20041210100659.GB15865@vicky.ecs.soton.ac.uk> <41BA34FD.6050306@v.loewis.de> <20041211165508.GA10252@vicky.ecs.soton.ac.uk> Message-ID: <41BB4333.2070003@stackless.com> Armin Rigo wrote: > Hi Martin, > > On Sat, Dec 11, 2004 at 12:45:01AM +0100, "Martin v. L?wis" wrote: > >>The straight-forward answer is: Get VC7.1 (aka VS.NET 2003), and invoke >> >>python setup.py bdist_wininst >> >>That's not too hard to do, I think. > > > Hum, this is getting into a Linux-vs-Windows argument. I don't want to invest > time and money on Windows tools just to compile my extension module for > Windows users... Maybe we can set this up as a service? I have the compiler and could do something like doing a checkout, build, and upload new binary for a number of projects. What is needed is a simple but secure notification method. Probably I need one windows machine which is always online. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From ncoghlan at iinet.net.au Sun Dec 12 01:39:13 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sun Dec 12 01:39:17 2004 Subject: [Python-Dev] Re: 2.4 news reaches interesting places In-Reply-To: <000201c4df6b$5c9cef60$e841fea9@oemcomputer> References: <000201c4df6b$5c9cef60$e841fea9@oemcomputer> Message-ID: <41BB9331.2060408@iinet.net.au> Raymond Hettinger wrote: Makes sense to me - I look forward to seeing the next version. Cheers, Nick. -- Nick Coghlan | ncoghlan@email.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net From ncoghlan at iinet.net.au Sun Dec 12 03:20:01 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sun Dec 12 03:20:07 2004 Subject: [Python-Dev] PEP 338: Executing modules inside packages with '-m' Message-ID: <41BBAAD1.4090208@iinet.net.au> Python 2.4's -m command line switch only works for modules directly on sys.path. Trying to use it with modules inside packages will fail with a "Module not found" error. This PEP aims to fix that for Python 2.5. Previously, posting of a draft version of the PEP to python-dev and python-list didn't actually generate any responses. I'm not sure if that's an indication that people don't see the restriction to top-level modules as a problem (and hence think the PEP is unecessary), or think the extension to handle packages is obvious (and hence see no need to comment). For c.l.p, it could just be a sign that Python 2.4 hasn't been out long enough for anyone to care what I'm yabbering on about :) Anyway, all comments are appreciated (even a simple "Sounds good to me"). Cheers, Nick. *********************************************************************** PEP: 338 Title: Executing modules inside packages with '-m' Version: $Revision: 1.2 $ Last-Modified: $Date: 2004/12/11 20:31:10 $ Author: Nick Coghlan Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 16-Oct-2004 Python-Version: 2.5 Post-History: 8-Nov-2004 Abstract ======== This PEP defines semantics for executing modules inside packages as scripts with the ``-m`` command line switch. The proposed semantics are that the containing package be imported prior to execution of the script. Rationale ========= Python 2.4 adds the command line switch ``-m`` to allow modules to be located using the Python module namespace for execution as scripts. The motivating examples were standard library modules such as ``pdb`` and ``profile``. A number of users and developers have requested extension of the feature to also support running modules located inside packages. One example provided is pychecker's ``pychecker.checker`` module. This capability was left out of the Python 2.4 implementation because the appropriate semantics were not entirely clear. The opinion on python-dev was that it was better to postpone the extension to Python 2.5, and go through the PEP process to help make sure we got it right. Scope of this proposal ========================== In Python 2.4, a module located using ``-m`` is executed just as if its filename had been provided on the command line. The goal of this PEP is to get as close as possible to making that statement also hold true for modules inside packages. Prior discussions suggest it should be noted that this PEP is **not** about any of the following: - changing the idiom for making Python modules also useful as scripts (see PEP 299 [1]_). - lifting the restriction of ``-m`` to modules of type PY_SOURCE or PY_COMPILED (i.e. ``.py``, ``.pyc``, ``.pyo``, ``.pyw``). - addressing the problem of ``-m`` not understanding zip imports or Python's sys.metapath. The issues listed above are considered orthogonal to the specific feature addressed by this PEP. Current Behaviour ================= Before describing the new semantics, it's worth covering the existing semantics for Python 2.4 (as they are currently defined only by the source code). When ``-m`` is used on the command line, it immediately terminates the option list (like ``-c``). The argument is interpreted as the name of a top-level Python module (i.e. one which can be found on ``sys.path``). If the module is found, and is of type ``PY_SOURCE`` or ``PY_COMPILED``, then the command line is effectively reinterpreted from ``python -m `` to ``python ``. This includes setting ``sys.argv[0]`` correctly (some scripts rely on this - Python's own ``regrtest.py`` is one example). If the module is not found, or is not of the correct type, an error is printed. Proposed Semantics ================== The semantics proposed are fairly simple: if ``-m`` is used to execute a module inside a package as a script, then the containing package is imported before executing the module in accordance with the semantics for a top-level module. This is necessary due to the way Python's import machinery locates modules inside packages. A package may modify its own __path__ variable during initialisation. In addition, paths may affected by ``*.pth`` files. Accordingly, the only way for Python to reliably locate the module is by importing the containing package and inspecting its __path__ variable. Note that the package is *not* imported into the ``__main__`` module's namespace. The effects of these semantics that will be visible to the executed module are: - the containing package will be in sys.modules - any external effects of the package initialisation (e.g. installed import hooks, loggers, atexit handlers, etc.) Reference Implementation ======================== A reference implementation is available on SourceForge [2]_. In this implementation, if the ``-m`` switch fails to locate the requested module at the top level, it effectively reinterprets the command from ``python -m