From martin@v.loewis.de Sat Dec 1 01:07:19 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Sat, 1 Dec 2001 02:07:19 +0100 Subject: [Python-Dev] -DINET6 in Makefile In-Reply-To: <3C07F016.AEC20158@lemburg.com> (mal@lemburg.com) References: <3C050CAF.2A5CEDC5@lemburg.com> <200111302022.fAUKMO002089@mira.informatik.hu-berlin.de> <3C07F016.AEC20158@lemburg.com> Message-ID: <200112010107.fB117JO03160@mira.informatik.hu-berlin.de> > > > Shouldn't the define be placed into the pyconfig.h file or > > > only in those extensions which need it ? > > > > Wouldn't it cause the same trouble there? > > No, because the pyconfig.h import is under extension control > (e.g. you can first include the system or lib header files > and only then import pyconfig.h). Of course, doing so would be really stupid. Python.h *must* be the first include, or things may break. Regards, Martin From martin@v.loewis.de Sat Dec 1 01:11:34 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Sat, 1 Dec 2001 02:11:34 +0100 Subject: [Python-Dev] gc.garbage In-Reply-To: References: Message-ID: <200112010111.fB11BYZ03163@mira.informatik.hu-berlin.de> > [Martin v. Loewis] > > I wish the C API hadn't been changed for 2.2, rendering useless all > > code that had been created to support GC in 2.0 and 2.1. > > Would we really need more than one hand to count all that code <0.9 wink>? Mine was among it (in pyexpat), and I really hate looking at the code now. Not only need I support two versions of Unicode (i.e. having or not having it), but also three versions of GC, all in a single file. Regards, Martin From mark.favas@csiro.au Sat Dec 1 01:55:49 2001 From: mark.favas@csiro.au (Mark Favas) Date: Sat, 01 Dec 2001 09:55:49 +0800 Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms Message-ID: <3C0838A5.7B35D64A@csiro.au> Anyone else getting test_builtin failures with current CVS, or does it only show on 64-bit platforms? Changes in the past week seem to have caused the failure. Isolated to following (will post bug on SF): ***** older CVS works ***** 201 mark@gonzo.per.dem.csiro.au python Python 2.2b2+ (#539, Nov 26 2001, 09:52:25) [C] on osf1V4 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> s=-1-sys.maxint >>> d=`s` >>> d '-9223372036854775808' >>> s -9223372036854775808 >>> ^D ***** current CVS fails ***** 202 mark@gonzo.per.dem.csiro.au cd dist/src 203 mark@gonzo.per.dem.csiro.au ./python Python 2.2b2+ (#541, Dec 1 2001, 08:04:58) [C] on osf1V4 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> s=-1-sys.maxint >>> d=`s` >>> d '8t\x10@\x01' >>> s -9223372036854775808 >>> ^D -- Mark Favas - m.favas@per.dem.csiro.au CSIRO, Private Bag No 5, Wembley, Western Australia 6913, AUSTRALIA From tim.one@home.com Sat Dec 1 02:41:56 2001 From: tim.one@home.com (Tim Peters) Date: Fri, 30 Nov 2001 21:41:56 -0500 Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms In-Reply-To: <3C0838A5.7B35D64A@csiro.au> Message-ID: [Mark Favas] > Anyone else getting test_builtin failures with current CVS, No. > or does it only show on 64-bit platforms? Changes in the past week > seem to have caused the failure. Isolated to following (will post bug > on SF): Thanks! > ***** older CVS works ***** > 201 mark@gonzo.per.dem.csiro.au python > Python 2.2b2+ (#539, Nov 26 2001, 09:52:25) [C] on osf1V4 > Type "help", "copyright", "credits" or "license" for more information. > >>> import sys > >>> s=-1-sys.maxint > >>> d=`s` > >>> d > '-9223372036854775808' > >>> s > -9223372036854775808 > >>> ^D > > ***** current CVS fails ***** > 202 mark@gonzo.per.dem.csiro.au cd dist/src > 203 mark@gonzo.per.dem.csiro.au ./python > Python 2.2b2+ (#541, Dec 1 2001, 08:04:58) [C] on osf1V4 > Type "help", "copyright", "credits" or "license" for more information. > >>> import sys > >>> s=-1-sys.maxint > >>> d=`s` > >>> d > '8t\x10@\x01' > >>> s > -9223372036854775808 > >>> ^D It doesn't make sense, but it *smells* like a sprintf -> PyOS_snprintf screwup ... OK, our int repr code has always been wrong(!): static PyObject * int_repr(PyIntObject *v) { char buf[20]; PyOS_snprintf(buf, sizeof(buf), "%ld", v->ob_ival); return PyString_FromString(buf); } 20 bytes isn't enough to hold the result on a 64-bit box (insert rant about the idiot practice of trying to make stack buffers as small as possible). You have 20 characters in your result, but need 21 to hold the trailing 0 byte too. I don't know what snprintf does when there's not enough room, but I think you just showed us what it does on Tru64 . Doing repr impllicitly instead works because that goes thru int_print instead of int_repr. Doing repr explicitly "worked" before by accident (the trailing null overwrite some random byte on the stack). Please change 20 to 200(*) (or 64 -- your choice) and see whether that fixes it? From guido@python.org Sat Dec 1 03:01:24 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 30 Nov 2001 22:01:24 -0500 Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms In-Reply-To: Your message of "Fri, 30 Nov 2001 21:41:56 EST." References: Message-ID: <200112010301.WAA06035@cj20424-a.reston1.va.home.com> > I don't know what snprintf does when there's not enough room, but > I think you just showed us what it does on Tru64 . Hm, snprintf is *supposed* to truncate the result, but it seems that here it refused to do anything. Maybe PyOS_snprintf should be a wrapper that checks the return value of snprintf? I noticed that *none* of the recently checked-in PyOS_snprintf calls have their return value checked, and I think it's better to leave it that way (since in many cases we really don't know what to do instead) -- maybe PyOS_snprintf should even return void (or does it already?). --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@home.com Sat Dec 1 03:42:58 2001 From: tim.one@home.com (Tim Peters) Date: Fri, 30 Nov 2001 22:42:58 -0500 Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms In-Reply-To: <200112010301.WAA06035@cj20424-a.reston1.va.home.com> Message-ID: > Hm, snprintf is *supposed* to truncate the result, According to C99, yes, but MS has its own pre-C99 snprintf semantics, and glibc has had at least two different versions. > but it seems that here it refused to do anything. Maybe PyOS_snprintf > should be a wrapper that checks the return value of snprintf? Check it for what? We can't (at least not yet) count on uniform behavior across platform snprintf implementations. > I noticed that *none* of the recently checked-in PyOS_snprintf calls > have their return value checked, They were all derived (and most mindlessly) from existing sprintf calls that didn't check either. A goal of this transformation was to change as little as possible. > and I think it's better to leave it that way (since in many cases we > really don't know what to do instead) -- maybe PyOS_snprintf should > even return void (or does it already?). The comments suggest it wants to return the C99-defined value (an int, < 0 for an encoding error, else the number of characters (excluding \0) written if the size is big enough, else the number of characters that would have been written (excluding \0) had size been big enough. So the output was converted in full iff the return value is non-negative and strictly less than the size passed in. That's fine by me, and I'll note that MS snprintf meets that much too (if size isn't big enough, it returns a negative result). So perhaps in a debug build PyOS_snprintf could assert that the returned value is non-negative and less than the passed-in size ... From barry@zope.com Sat Dec 1 04:17:31 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 30 Nov 2001 23:17:31 -0500 Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms References: <3C0838A5.7B35D64A@csiro.au> Message-ID: <15368.23003.392434.202067@anthem.wooz.org> >>>>> "TP" == Tim Peters writes: TP> It doesn't make sense, but it *smells* like a sprintf -> TP> PyOS_snprintf screwup ... OK, our int repr code has always TP> been wrong(!): | static PyObject * | int_repr(PyIntObject *v) | { | char buf[20]; | PyOS_snprintf(buf, sizeof(buf), "%ld", v->ob_ival); | return PyString_FromString(buf); | } TP> 20 bytes isn't enough to hold the result on a 64-bit box TP> (insert rant about the idiot practice of trying to make stack TP> buffers as small as possible). You have 20 characters in your TP> result, but need 21 to hold the trailing 0 byte too. I don't TP> know what snprintf does when there's not enough room, but I TP> think you just showed us what it does on Tru64 . Heh, I was going to suggest that this might be a good place to substitute a call to PyString_FromFormat*() but then I read this little nugget: case 'd': case 'i': case 'x': (void) va_arg(count, int); /* 20 bytes should be enough to hold a 64-bit integer */ n += 20; break; love-ly y'rs, -Barry From tim.one@home.com Sat Dec 1 04:31:51 2001 From: tim.one@home.com (Tim Peters) Date: Fri, 30 Nov 2001 23:31:51 -0500 Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms In-Reply-To: <15368.23003.392434.202067@anthem.wooz.org> Message-ID: [Barry] > ... > Heh, I was going to suggest that this might be a good place to > substitute a call to PyString_FromFormat*() but then I read this > little nugget: > > case 'd': case 'i': case 'x': > (void) va_arg(count, int); > /* 20 bytes should be enough to > hold a 64-bit > integer */ > n += 20; > break; This use of 20 is fine (I checked all occurrences of "20" in the codebase, BTW); int_repr's use of 20 was wrong because it failed to allow for the trailing \0 byte too; the loop in PyString_FromFormatV doesn't have to worry about that (PyString_FromStringAndSize() later adds 1 of its own for the trailing \0 -- of course it doesn't actually add anything, but it adds 1 "in effect" ). From guido@python.org Sat Dec 1 08:06:01 2001 From: guido@python.org (Guido van Rossum) Date: Sat, 01 Dec 2001 03:06:01 -0500 Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms In-Reply-To: Your message of "Fri, 30 Nov 2001 22:42:58 EST." References: Message-ID: <200112010806.DAA06935@cj20424-a.reston1.va.home.com> > > Hm, snprintf is *supposed* to truncate the result, > > According to C99, yes, but MS has its own pre-C99 snprintf semantics, and > glibc has had at least two different versions. > > > but it seems that here it refused to do anything. Maybe PyOS_snprintf > > should be a wrapper that checks the return value of snprintf? > > Check it for what? We can't (at least not yet) count on uniform behavior > across platform snprintf implementations. See below. On my system, the snprintf man page documents both behaviors, with the gcc version where it switches. > > I noticed that *none* of the recently checked-in PyOS_snprintf calls > > have their return value checked, > > They were all derived (and most mindlessly) from existing sprintf calls that > didn't check either. A goal of this transformation was to change as little > as possible. > > > and I think it's better to leave it that way (since in many cases we > > really don't know what to do instead) -- maybe PyOS_snprintf should > > even return void (or does it already?). > > The comments suggest it wants to return the C99-defined value (an int, < 0 > for an encoding error, else the number of characters (excluding \0) written > if the size is big enough, else the number of characters that would have > been written (excluding \0) had size been big enough. So the output was > converted in full iff the return value is non-negative and strictly less > than the size passed in. That's fine by me, and I'll note that MS snprintf > meets that much too (if size isn't big enough, it returns a negative > result). So perhaps in a debug build PyOS_snprintf could assert that the > returned value is non-negative and less than the passed-in size ... Should work for gcc too. Note that the Tru64 output suggests that the behavior (== effect on the buffer) may differ too: a truncated version of the desired output may or may not be written to the buffer. This can be addressed by pre-filling the buffer with a useful pattern in PyOS_snprintf before calling the system's snprintf -- e.g. "*-*-*-*" (to give a patriotic example :-). --Guido van Rossum (home page: http://www.python.org/~guido/) PS: I feel like I'm writing this on borrowed time -- Comcast and Excite@Home haven't disconnected me from the net yet. :-) From mark.favas@csiro.au Sat Dec 1 10:25:50 2001 From: mark.favas@csiro.au (Mark Favas) Date: Sat, 01 Dec 2001 18:25:50 +0800 Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms References: <200112010806.DAA06935@cj20424-a.reston1.va.home.com> Message-ID: <3C08B02E.EC833A0@csiro.au> The increase in buf size from 20 to 64 fixes the problem, as Tim expected. Note that Tru64 (at least V4, unsure about V5) has no native snprintf implementation. Guido van Rossum wrote: > > > > Hm, snprintf is *supposed* to truncate the result, > > > > According to C99, yes, but MS has its own pre-C99 snprintf semantics, and > > glibc has had at least two different versions. > > > > > but it seems that here it refused to do anything. Maybe PyOS_snprintf > > > should be a wrapper that checks the return value of snprintf? > > -- Mark Favas - mark.favas@csiro.au From fredrik@pythonware.com Sat Dec 1 11:35:16 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Sat, 1 Dec 2001 12:35:16 +0100 Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms References: <3C0838A5.7B35D64A@csiro.au> <15368.23003.392434.202067@anthem.wooz.org> Message-ID: <010a01c17a5c$45c1c030$ced241d5@hagrid> Barry A. Warsaw wrote: > TP> 20 bytes isn't enough to hold the result on a 64-bit box > TP> (insert rant about the idiot practice of trying to make stack > TP> buffers as small as possible). You have 20 characters in your > TP> result, but need 21 to hold the trailing 0 byte too. I don't > TP> know what snprintf does when there's not enough room, but I > TP> think you just showed us what it does on Tru64 . > > Heh, I was going to suggest that this might be a good place to > substitute a call to PyString_FromFormat*() but then I read this > little nugget: > > case 'd': case 'i': case 'x': > (void) va_arg(count, int); > /* 20 bytes should be enough to hold a 64-bit > integer */ > n += 20; > break; The size calculated by PyString_FromFormat is used to allocate a Python string, not a C string. The Python string allocator always adds an extra 0 byte at the end. (if it hadn't, I would have used 21 instead) From mal@lemburg.com Sat Dec 1 11:27:20 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 01 Dec 2001 12:27:20 +0100 Subject: [Python-Dev] Subclassing varying length types (What's a PyStructSequence ?) References: Message-ID: <3C08BE98.79946952@lemburg.com> Tim Peters wrote: > > [M.-A. Lemburg] > > ... > > Hmm, this makes me wonder: perhaps we should start thinking > > about phasing out varying length PyObjects in the interpreter... > > No chance, IMO: the memory savings is too great. > > > esp. the inability to subclass strings looks like a bummer for > > future extensions of this particular type. Unicode doesn't have > > this problem, BTW. > > OTOH, I know someone at Zope Corp who could testify with force about the > memory burden of switching to Unicode strings -- if you've got gobs of 'em, > it's much worse than a factor of 2 blowup. Moving to obmalloc.c should help > that a lot (two malloc overheads per Unicode string, and obmalloc overheads > are much lower). Perhaps we should start thinking about optimizing at least one of the Unicode malloc away in Python 2.3: the Unicode object itself can well be kept on a free list with the Py_UNICODE buffer freed and set to NULL. Doesn't save any memory but would improve the performance. BTW, is the memory burden really such a big argument these days ? I can imagine this being an argument on resource restrained platforms such as Palms (thanks to Martin, the Plam guys can now switch off Unicode completely), but hardly on gigabyte machines with access 100s of GBs swap-space :-) > > Or we need to come up with a fairly nice way of making > > subclassing varying length types a lot easier, e.g. by > > adding a special pointer ob_ext to PyObject_VAR_HEAD > > which then allows declaring type extensions in an malloced > > buffer. > > > > Thoughts ? > > Not a one . Any idea how we could make subclassing these types less hackish, then ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Sat Dec 1 13:06:37 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 01 Dec 2001 14:06:37 +0100 Subject: [Python-Dev] -DINET6 in Makefile References: <3C050CAF.2A5CEDC5@lemburg.com> <200111302022.fAUKMO002089@mira.informatik.hu-berlin.de> <3C07F016.AEC20158@lemburg.com> <200112010107.fB117JO03160@mira.informatik.hu-berlin.de> Message-ID: <3C08D5DD.1F3E8B55@lemburg.com> "Martin v. Loewis" wrote: > > > > > Shouldn't the define be placed into the pyconfig.h file or > > > > only in those extensions which need it ? > > > > > > Wouldn't it cause the same trouble there? > > > > No, because the pyconfig.h import is under extension control > > (e.g. you can first include the system or lib header files > > and only then import pyconfig.h). > > Of course, doing so would be really stupid. Python.h *must* be the > first include, or things may break. Why should "things break" ? I've doing this for years in lots of Python extensions... Back on topic: how are we going to get -DINET6 out of the Makefile ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From martin@v.loewis.de Sat Dec 1 13:53:36 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Sat, 1 Dec 2001 14:53:36 +0100 Subject: [Python-Dev] Subclassing varying length types (What's a PyStructSequence ?) In-Reply-To: <3C08BE98.79946952@lemburg.com> (mal@lemburg.com) References: <3C08BE98.79946952@lemburg.com> Message-ID: <200112011353.fB1Drab17745@mira.informatik.hu-berlin.de> > BTW, is the memory burden really such a big argument these days ? You should consider that malloc overhead is often 16 bytes per object. Given that PyUnicodeObject is 24 bytes in 2.2, system malloc will allocate 48 bytes per Unicode object on modern architectures. I would think 100% overhead *is* a big argument. If you relate this to the actual data, it gets worse: A Unicode string of length 1 would still require 32 bytes on an allocator that aligns to 16. Therefore, to store 2 bytes of real data, you need 80 bytes of memory. I don't know how much overhead pymalloc adds, though; I believe it is significantly less expensive. Regards, Martin From mal@lemburg.com Sat Dec 1 14:49:49 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 01 Dec 2001 15:49:49 +0100 Subject: [Python-Dev] Subclassing varying length types (What's a PyStructSequence ?) References: <3C08BE98.79946952@lemburg.com> <200112011353.fB1Drab17745@mira.informatik.hu-berlin.de> Message-ID: <3C08EE0D.DF82C41F@lemburg.com> "Martin v. Loewis" wrote: > > > BTW, is the memory burden really such a big argument these days ? > > You should consider that malloc overhead is often 16 bytes per > object. Given that PyUnicodeObject is 24 bytes in 2.2, system malloc > will allocate 48 bytes per Unicode object on modern architectures. I > would think 100% overhead *is* a big argument. > > If you relate this to the actual data, it gets worse: A Unicode string > of length 1 would still require 32 bytes on an allocator that aligns > to 16. Therefore, to store 2 bytes of real data, you need 80 bytes of > memory. > > I don't know how much overhead pymalloc adds, though; I believe it is > significantly less expensive. Oh, I wasn't arguing against using pymalloc :-) I only think that nowadays, the trade-off "more flexibility vs. memory consumption" leans more towards the former than the latter. Not only because memory is cheap, but also because more flexibility tends to result in use of better algorithms and these can lead to better overall performance and lower total memory use. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From guido@python.org Sat Dec 1 15:46:20 2001 From: guido@python.org (Guido van Rossum) Date: Sat, 01 Dec 2001 10:46:20 -0500 Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms In-Reply-To: Your message of "Sat, 01 Dec 2001 18:25:50 +0800." <3C08B02E.EC833A0@csiro.au> References: <200112010806.DAA06935@cj20424-a.reston1.va.home.com> <3C08B02E.EC833A0@csiro.au> Message-ID: <200112011546.KAA08107@cj20424-a.reston1.va.home.com> > The increase in buf size from 20 to 64 fixes the problem, as Tim > expected. Note that Tru64 (at least V4, unsure about V5) has no native > snprintf implementation. Oops, you're right. It seems that mysnprintf.c doesn't copy any bytes to the buffer when there's an overflow. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sat Dec 1 15:59:18 2001 From: guido@python.org (Guido van Rossum) Date: Sat, 01 Dec 2001 10:59:18 -0500 Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms In-Reply-To: Your message of "Sat, 01 Dec 2001 10:46:20 EST." <200112011546.KAA08107@cj20424-a.reston1.va.home.com> References: <200112010806.DAA06935@cj20424-a.reston1.va.home.com> <3C08B02E.EC833A0@csiro.au> <200112011546.KAA08107@cj20424-a.reston1.va.home.com> Message-ID: <200112011559.KAA08652@cj20424-a.reston1.va.home.com> > Oops, you're right. It seems that mysnprintf.c doesn't copy any bytes > to the buffer when there's an overflow. Confirmed, and fixed in CVS. --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy@zope.com Sat Dec 1 17:02:54 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Sat, 1 Dec 2001 12:02:54 -0500 (EST) Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms In-Reply-To: References: <200112010301.WAA06035@cj20424-a.reston1.va.home.com> Message-ID: <15369.3390.364861.781161@slothrop.digicool.com> The uses of PyOS_snprintf() in sysmodule.c actually checks the return value. It always checked the sprintf() returned value and triggered a Py_FatalError() if the return value was too big. I changed it to print a warning message that the output was truncated. Jeremy From jim@zope.com Sat Dec 1 17:22:34 2001 From: jim@zope.com (Jim Fulton) Date: Sat, 01 Dec 2001 12:22:34 -0500 Subject: [Python-Dev] ExtensionClass plans (was Re: [ZODB-Dev] Another ExtensionClass incompatibility: __pow__()) References: <20011130115328.A6028@mems-exchange.org> Message-ID: <3C0911DA.752DD61E@zope.com> Greg Ward wrote: > > Sigh. In addition to ignoring __r{add,sub,mul,div}__() methods, > ExtensionClass also calls __pow__() differently. (snip) > So what is the future of ExtensionClass, anyways? I intend that Python 2.2 (or maybe 2.3) will make it irrelevent. > Are the little > incompatibility bugs between classic Python classes and ExtensionClass > ever going to be fixed? Probably not by us. Patches are gratefully accepted. We are going to keep supporting it for a while at it's current level to support projects other than ZODB that require it. > Or has ExtensionClass been abandoned in favour > of a future ZODB based on new-style Python 2.2 classes? This is my plan. :) > If so, when > will this future ZODB become reality? I'm glad you asked. As part of our Zope 3 prototyping, we've begun moving toward a next generation of ZODB. We currently have an experimental ZODB that: - Doesn't require ExtensionClass - Does depend on new-style Python classes - Separates the Persistence and Transaction frameworks - Uses weak references in the cache - Uses a pure-python Persistent implementation (hence no BTrees). We will be creaing a C Persistent base class. All the fun is currently happening on the Zope-3x-branch branch of the Zope3 checkout. You can see the relevent ZODB pieces at: http://cvs.zope.org/Zope3/lib/python/Persistence/?only_with_tag=Zope-3x-branch http://cvs.zope.org/Zope3/lib/python/Transaction/?only_with_tag=Zope-3x-branch http://cvs.zope.org/Zope3/lib/python/ZODB/?only_with_tag=Zope-3x-branch Of course all of this is highly speculative at this point. Stay tuned. ;) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jack@oratrix.nl Sat Dec 1 23:26:00 2001 From: jack@oratrix.nl (Jack Jansen) Date: Sun, 02 Dec 2001 00:26:00 +0100 Subject: [Python-Dev] Differentiating $(EXT) between build and install Message-ID: <20011201232600.45BC4E2677@oratrix.oratrix.nl> Folks, there's so many bug reports and questions about needing the --with-suffix on Mac OS X when you're building on an HFS+ filesystem that I'd like to fix this. The warning in configure doesn't cut it because it gets lost in all the other messages. What I'd like to do is have a separate Makefile variable, BUILDEXT, that gives the extension to use for python while it lives in the build directory. It would default to EXT, except if on HFS+ when EXT is empty. But this might change a couple of other things as well, so my question is: is this an acceptable change so late in the beta stage? -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | ++++ see http://www.xs4all.nl/~tank/ ++++ From MarkH@ActiveState.com Sun Dec 2 00:52:47 2001 From: MarkH@ActiveState.com (Mark Hammond) Date: Sun, 2 Dec 2001 11:52:47 +1100 Subject: [Python-Dev] Subclassing varying length types (What's a PyStructSequence ?) In-Reply-To: <3C08EE0D.DF82C41F@lemburg.com> Message-ID: [MAL] > I only think that nowadays, the trade-off "more flexibility vs. > memory consumption" leans more towards the former than the latter. > Not only because memory is cheap, but also because more flexibility > tends to result in use of better algorithms and these can lead to > better overall performance and lower total memory use. This tends to sound great - until you have a large app that is consuming too many MBs :) Mozilla's use of strings is fascinating. They have a very complex C++ string API - all aimed at being as space-optimized as possible. They go to huge lengths to avoid string copies and extra allocation at the expense of an API nobody understands :) The point is that for fundamental types (including Unicode), medium size apps may use millions of these objects. Everything we can do to optimize their speed and size is beneficial. Mark. From adam@isdn.net.il Sun Dec 2 07:36:43 2001 From: adam@isdn.net.il (adam) Date: Sun, 2 Dec 2001 09:36:43 +0200 Subject: [Python-Dev] REMOVE !! References: Message-ID: <002301c17b04$17cd4a20$0101c80a@LocalHost> ----- Original Message ----- From: To: Sent: Saturday, December 01, 2001 6:33 AM Subject: Python-Dev digest, Vol 1 #1748 - 15 msgs > Send Python-Dev mailing list submissions to > python-dev@python.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.python.org/mailman/listinfo/python-dev > or, via email, send a message with subject or body 'help' to > python-dev-request@python.org > > You can reach the person managing the list at > python-dev-admin@python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-Dev digest..." > > > Today's Topics: > > 1. ArtifactFile: I can't change my patch files (James C. Ahlstrom) > 2. Re: problems with DBM nonuniformity (Jason R. Mastaler) > 3. Re: -DINET6 in Makefile (Martin v. Loewis) > 4. Re: -DINET6 in Makefile (M.-A. Lemburg) > 5. RE: gc.garbage (Tim Peters) > 6. RE: gc.garbage (Tim Peters) > 7. RE: Subclassing varying length types (What's a PyStructSequence ?) (Tim Peters) > 8. Re: -DINET6 in Makefile (Martin v. Loewis) > 9. Re: gc.garbage (Martin v. Loewis) > 10. test_builtin failing? or just 64-bit platforms (Mark Favas) > 11. RE: test_builtin failing? or just 64-bit platforms (Tim Peters) > 12. Re: test_builtin failing? or just 64-bit platforms (Guido van Rossum) > 13. RE: test_builtin failing? or just 64-bit platforms (Tim Peters) > 14. RE: test_builtin failing? or just 64-bit platforms (Barry A. Warsaw) > 15. RE: test_builtin failing? or just 64-bit platforms (Tim Peters) > > --__--__-- > > Message: 1 > Date: Fri, 30 Nov 2001 12:07:03 -0500 > From: "James C. Ahlstrom" > To: python-dev@python.org > Subject: [Python-Dev] ArtifactFile: I can't change my patch files > > I am trying to replace files in my patch #483466, but > I am getting > > File delete: ArtifactFile: Permission Denied > > on SourceForge. If I need to be a member of the Python > project to do this, could someone please add me? Or > tell me what's wrong? > > The new files implement the changes to sys.path[0] > that we have been discussing. I still need to make > further changes to import.c so that "" and relative > paths are handled. So don't install this patch yet, > but look it over for problems if you want. > > Jim > > > --__--__-- > > Message: 2 > To: python-list@python.org, python-dev@python.org, > Date: Fri, 30 Nov 2001 10:17:24 -0700 > From: "Jason R. Mastaler" > Subject: [Python-Dev] Re: problems with DBM nonuniformity > > Skip Montanaro writes: > > > Seems to me the natural thing to do would be to add > > "get_data_filename" and "get_index_filename" methods (or something > > similar) to the underlying modules (dbhash, bsddb, dbm, etc) and > > expose them through anydbm. > > I see. So you agree that with the current implementation, there isn't > a reliable way to do what I'm trying to do with DBM? > > > It's too late for 2.2, but I suspect if you implemented something > > and method name(s) could be settled on it would make it into CVS > > early in the 2.3 cycle. This seems like a small enough change that > > you just file a bug report on SourceForge with the proposal and add > > an implementation when you have something workable. > > I'm not sure when I'll be able to get to this, but I'll put it on my > TODO list. In the meantime, I think I'll just support the auto > regeneration feature I mentioned with CDB[1] instead of DBM since its > file interface is consistent across platforms. > > Footnotes: > 1. python-cdb extension module (http://pilcrow.madison.wi.us/) > > -- > (TMDA - http://tmda.sourceforge.net) > (Python-based SPAM reduction system) > > > --__--__-- > > Message: 3 > Date: Fri, 30 Nov 2001 21:22:24 +0100 > From: "Martin v. Loewis" > To: mal@lemburg.com > CC: python-dev@python.org > Subject: Re: [Python-Dev] -DINET6 in Makefile > > > What's the reasoning behind putting -DINET6 into the default > > compiler options of the generic Makefile ? > > I believe the sole reason is that the author of the patch didn't know > how to get it into pyconfig.h. itojun recently confirmed that all uses > of the INET6 can be replaced with ENABLE_IPV6, and that the define may > go away. > > I hesitate to change that, though, since some of the IPv6 > implementations *may* require that INET6 is defined when processing > the "system" headers (not all IPv6 implementations we support actually > come with the operating system). > > > I'm just asking because such a define will be inherited by > > all extensions being compiled with distutils and the Makefile.pre.in > > setup process... sounds like trouble if you ask me. > > What kind of trouble? > > > Shouldn't the define be placed into the pyconfig.h file or > > only in those extensions which need it ? > > Wouldn't it cause the same trouble there? > > Regards, > Martin > > > --__--__-- > > Message: 4 > Date: Fri, 30 Nov 2001 21:46:14 +0100 > From: "M.-A. Lemburg" > Organization: eGenix.com Software GmbH > To: "Martin v. Loewis" > CC: python-dev@python.org > Subject: Re: [Python-Dev] -DINET6 in Makefile > > "Martin v. Loewis" wrote: > > > > > What's the reasoning behind putting -DINET6 into the default > > > compiler options of the generic Makefile ? > > > > I believe the sole reason is that the author of the patch didn't know > > how to get it into pyconfig.h. itojun recently confirmed that all uses > > of the INET6 can be replaced with ENABLE_IPV6, and that the define may > > go away. > > > > I hesitate to change that, though, since some of the IPv6 > > implementations *may* require that INET6 is defined when processing > > the "system" headers (not all IPv6 implementations we support actually > > come with the operating system). > > For Python's own use, it should suffice defining the symbol in > pyconfig.h. > > > > I'm just asking because such a define will be inherited by > > > all extensions being compiled with distutils and the Makefile.pre.in > > > setup process... sounds like trouble if you ask me. > > > > What kind of trouble? > > The symbol could enable some logic which may not be desired > by the application, e.g. cause system includes to change, > socket semantics of wrapped libs could also be affected etc. > > > > Shouldn't the define be placed into the pyconfig.h file or > > > only in those extensions which need it ? > > > > Wouldn't it cause the same trouble there? > > No, because the pyconfig.h import is under extension control > (e.g. you can first include the system or lib header files > and only then import pyconfig.h). > > -- > Marc-Andre Lemburg > CEO eGenix.com Software GmbH > ______________________________________________________________________ > Consulting & Company: http://www.egenix.com/ > Python Software: http://www.lemburg.com/python/ > > > --__--__-- > > Message: 5 > From: "Tim Peters" > To: "Neil Schemenauer" , > "PythonDev" > Subject: RE: [Python-Dev] gc.garbage > Date: Fri, 30 Nov 2001 18:30:43 -0500 > > [Neil Schemenauer] > > Is there some way to prevent people from assigning to certain module > > variables? > > [Tim] > >> Not that I know of. If you're terribly concerned, gc could look up > >> "garbage" in its dict on each access. That's what, e.g., > >> PRINT_ITEM does with sys.stdout. ... > > [Neil] > > What would happen if it's not a list? PRINT_ITEM raises RuntimeError. > > I suppose the collector could do the same. > > Sure, that would be fine. > > >> But I'd be keener to see new words spelling out which parts of the gc > >> interface are and aren't intended "to work" across releases ... > > > All of them? :-) Seriously, there could come a time when GC can no > > longer be disabled. The debugging and threshold stuff is fairly > > implementation dependent. get_referrers() and get_objects() are highly > > implementation dependent. I suppose gc.collect() should always be > > available. Anything else is fair game, IMHO. > > I meant "new words" in the docs, not on Python-Dev . > > > Incidentally, I can't say I'm happy with GC as it stands. > > Well, you're young and hopeful -- you'll get over both. I have, and am > indeed happy with GC as it stands. > > > It uses too much memory now that so many objects are tracked. > > There I disagree, but subtly: it always used "too much" memory. The > marginal memory cost in adding a gazillion new tracked types was minor, as > very few programs have a gazillion frame objects or traceback objects or > generator-iterator objects (etc) sitting around. The vast bulk of the > damage was done the instant lists, tuples, dicts and instances got tracked. > So it goes. > > > I had worked on the idea of a separate heap for GC objects for a while > > but couldn't figure out how to make generational collection to work. > > Generational gimmicks are rare in non-copying collectors for this very > reason, right? > > > As Don Beaudry's sig says: "so much code, so little time". :-) > > Time for Don to change his sig -- his young and hopeful days should be long > gone by now too . > > > > --__--__-- > > Message: 6 > From: "Tim Peters" > To: > Subject: RE: [Python-Dev] gc.garbage > Date: Fri, 30 Nov 2001 18:30:44 -0500 > > [Martin v. Loewis] > > I wish the C API hadn't been changed for 2.2, rendering useless all > > code that had been created to support GC in 2.0 and 2.1. > > Would we really need more than one hand to count all that code <0.9 wink>? > > not-aware-of-any-myself-ly y'rs - tim > > > --__--__-- > > Message: 7 > From: "Tim Peters" > To: > Subject: RE: [Python-Dev] Subclassing varying length types (What's a PyStructSequence ?) > Date: Fri, 30 Nov 2001 18:35:23 -0500 > > [M.-A. Lemburg] > > ... > > Hmm, this makes me wonder: perhaps we should start thinking > > about phasing out varying length PyObjects in the interpreter... > > No chance, IMO: the memory savings is too great. > > > esp. the inability to subclass strings looks like a bummer for > > future extensions of this particular type. Unicode doesn't have > > this problem, BTW. > > OTOH, I know someone at Zope Corp who could testify with force about the > memory burden of switching to Unicode strings -- if you've got gobs of 'em, > it's much worse than a factor of 2 blowup. Moving to obmalloc.c should help > that a lot (two malloc overheads per Unicode string, and obmalloc overheads > are much lower). > > > Or we need to come up with a fairly nice way of making > > subclassing varying length types a lot easier, e.g. by > > adding a special pointer ob_ext to PyObject_VAR_HEAD > > which then allows declaring type extensions in an malloced > > buffer. > > > > Thoughts ? > > Not a one . > > > > --__--__-- > > Message: 8 > Date: Sat, 1 Dec 2001 02:07:19 +0100 > From: "Martin v. Loewis" > To: mal@lemburg.com > CC: python-dev@python.org > Subject: Re: [Python-Dev] -DINET6 in Makefile > > > > > Shouldn't the define be placed into the pyconfig.h file or > > > > only in those extensions which need it ? > > > > > > Wouldn't it cause the same trouble there? > > > > No, because the pyconfig.h import is under extension control > > (e.g. you can first include the system or lib header files > > and only then import pyconfig.h). > > Of course, doing so would be really stupid. Python.h *must* be the > first include, or things may break. > > Regards, > Martin > > > > --__--__-- > > Message: 9 > Date: Sat, 1 Dec 2001 02:11:34 +0100 > From: "Martin v. Loewis" > To: tim.one@home.com > CC: python-dev@python.org > Subject: Re: [Python-Dev] gc.garbage > > > [Martin v. Loewis] > > > I wish the C API hadn't been changed for 2.2, rendering useless all > > > code that had been created to support GC in 2.0 and 2.1. > > > > Would we really need more than one hand to count all that code <0.9 wink>? > > Mine was among it (in pyexpat), and I really hate looking at the code > now. Not only need I support two versions of Unicode (i.e. having or > not having it), but also three versions of GC, all in a single file. > > Regards, > Martin > > > --__--__-- > > Message: 10 > Date: Sat, 01 Dec 2001 09:55:49 +0800 > From: Mark Favas > Organization: CSIRO > To: python-dev@python.org > Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms > > Anyone else getting test_builtin failures with current CVS, or does it > only show on 64-bit platforms? Changes in the past week seem to have > caused the failure. Isolated to following (will post bug on SF): > > ***** older CVS works ***** > 201 mark@gonzo.per.dem.csiro.au python > Python 2.2b2+ (#539, Nov 26 2001, 09:52:25) [C] on osf1V4 > Type "help", "copyright", "credits" or "license" for more information. > >>> import sys > >>> s=-1-sys.maxint > >>> d=`s` > >>> d > '-9223372036854775808' > >>> s > -9223372036854775808 > >>> ^D > > ***** current CVS fails ***** > 202 mark@gonzo.per.dem.csiro.au cd dist/src > 203 mark@gonzo.per.dem.csiro.au ./python > Python 2.2b2+ (#541, Dec 1 2001, 08:04:58) [C] on osf1V4 > Type "help", "copyright", "credits" or "license" for more information. > >>> import sys > >>> s=-1-sys.maxint > >>> d=`s` > >>> d > '8t\x10@\x01' > >>> s > -9223372036854775808 > >>> ^D > > > -- > Mark Favas - m.favas@per.dem.csiro.au > CSIRO, Private Bag No 5, Wembley, Western Australia 6913, AUSTRALIA > > > --__--__-- > > Message: 11 > From: "Tim Peters" > To: "Mark Favas" , > > Subject: RE: [Python-Dev] test_builtin failing? or just 64-bit platforms > Date: Fri, 30 Nov 2001 21:41:56 -0500 > > [Mark Favas] > > Anyone else getting test_builtin failures with current CVS, > > No. > > > or does it only show on 64-bit platforms? Changes in the past week > > seem to have caused the failure. Isolated to following (will post bug > > on SF): > > Thanks! > > > ***** older CVS works ***** > > 201 mark@gonzo.per.dem.csiro.au python > > Python 2.2b2+ (#539, Nov 26 2001, 09:52:25) [C] on osf1V4 > > Type "help", "copyright", "credits" or "license" for more information. > > >>> import sys > > >>> s=-1-sys.maxint > > >>> d=`s` > > >>> d > > '-9223372036854775808' > > >>> s > > -9223372036854775808 > > >>> ^D > > > > ***** current CVS fails ***** > > 202 mark@gonzo.per.dem.csiro.au cd dist/src > > 203 mark@gonzo.per.dem.csiro.au ./python > > Python 2.2b2+ (#541, Dec 1 2001, 08:04:58) [C] on osf1V4 > > Type "help", "copyright", "credits" or "license" for more information. > > >>> import sys > > >>> s=-1-sys.maxint > > >>> d=`s` > > >>> d > > '8t\x10@\x01' > > >>> s > > -9223372036854775808 > > >>> ^D > > It doesn't make sense, but it *smells* like a sprintf -> PyOS_snprintf > screwup ... OK, our int repr code has always been wrong(!): > > static PyObject * > int_repr(PyIntObject *v) > { > char buf[20]; > PyOS_snprintf(buf, sizeof(buf), "%ld", v->ob_ival); > return PyString_FromString(buf); > } > > 20 bytes isn't enough to hold the result on a 64-bit box (insert rant about > the idiot practice of trying to make stack buffers as small as possible). > You have 20 characters in your result, but need 21 to hold the trailing 0 > byte too. I don't know what snprintf does when there's not enough room, but > I think you just showed us what it does on Tru64 . > > Doing repr impllicitly instead works because that goes thru int_print > instead of int_repr. Doing repr explicitly "worked" before by accident (the > trailing null overwrite some random byte on the stack). Please change 20 to > 200(*) (or 64 -- your choice) and see whether that fixes it? > > > > --__--__-- > > Message: 12 > To: "Tim Peters" > cc: "Mark Favas" , python-dev@python.org > Subject: Re: [Python-Dev] test_builtin failing? or just 64-bit platforms > From: Guido van Rossum > Date: Fri, 30 Nov 2001 22:01:24 -0500 > > > I don't know what snprintf does when there's not enough room, but > > I think you just showed us what it does on Tru64 . > > Hm, snprintf is *supposed* to truncate the result, but it seems that > here it refused to do anything. Maybe PyOS_snprintf should be a > wrapper that checks the return value of snprintf? I noticed that > *none* of the recently checked-in PyOS_snprintf calls have their > return value checked, and I think it's better to leave it that way > (since in many cases we really don't know what to do instead) -- maybe > PyOS_snprintf should even return void (or does it already?). > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > > --__--__-- > > Message: 13 > From: "Tim Peters" > To: "Guido van Rossum" > Cc: "Mark Favas" , > > Subject: RE: [Python-Dev] test_builtin failing? or just 64-bit platforms > Date: Fri, 30 Nov 2001 22:42:58 -0500 > > > Hm, snprintf is *supposed* to truncate the result, > > According to C99, yes, but MS has its own pre-C99 snprintf semantics, and > glibc has had at least two different versions. > > > but it seems that here it refused to do anything. Maybe PyOS_snprintf > > should be a wrapper that checks the return value of snprintf? > > Check it for what? We can't (at least not yet) count on uniform behavior > across platform snprintf implementations. > > > I noticed that *none* of the recently checked-in PyOS_snprintf calls > > have their return value checked, > > They were all derived (and most mindlessly) from existing sprintf calls that > didn't check either. A goal of this transformation was to change as little > as possible. > > > and I think it's better to leave it that way (since in many cases we > > really don't know what to do instead) -- maybe PyOS_snprintf should > > even return void (or does it already?). > > The comments suggest it wants to return the C99-defined value (an int, < 0 > for an encoding error, else the number of characters (excluding \0) written > if the size is big enough, else the number of characters that would have > been written (excluding \0) had size been big enough. So the output was > converted in full iff the return value is non-negative and strictly less > than the size passed in. That's fine by me, and I'll note that MS snprintf > meets that much too (if size isn't big enough, it returns a negative > result). So perhaps in a debug build PyOS_snprintf could assert that the > returned value is non-negative and less than the passed-in size ... > > > > --__--__-- > > Message: 14 > Date: Fri, 30 Nov 2001 23:17:31 -0500 > To: "Tim Peters" > Cc: "Mark Favas" , > Subject: RE: [Python-Dev] test_builtin failing? or just 64-bit platforms > From: barry@zope.com (Barry A. Warsaw) > > > >>>>> "TP" == Tim Peters writes: > > TP> It doesn't make sense, but it *smells* like a sprintf -> > TP> PyOS_snprintf screwup ... OK, our int repr code has always > TP> been wrong(!): > > | static PyObject * > | int_repr(PyIntObject *v) > | { > | char buf[20]; > | PyOS_snprintf(buf, sizeof(buf), "%ld", v->ob_ival); > | return PyString_FromString(buf); > | } > > TP> 20 bytes isn't enough to hold the result on a 64-bit box > TP> (insert rant about the idiot practice of trying to make stack > TP> buffers as small as possible). You have 20 characters in your > TP> result, but need 21 to hold the trailing 0 byte too. I don't > TP> know what snprintf does when there's not enough room, but I > TP> think you just showed us what it does on Tru64 . > > Heh, I was going to suggest that this might be a good place to > substitute a call to PyString_FromFormat*() but then I read this > little nugget: > > case 'd': case 'i': case 'x': > (void) va_arg(count, int); > /* 20 bytes should be enough to hold a 64-bit > integer */ > n += 20; > break; > > love-ly y'rs, > -Barry > > > --__--__-- > > Message: 15 > From: "Tim Peters" > To: "Barry A. Warsaw" > Cc: "Mark Favas" , > > Subject: RE: [Python-Dev] test_builtin failing? or just 64-bit platforms > Date: Fri, 30 Nov 2001 23:31:51 -0500 > > [Barry] > > ... > > Heh, I was going to suggest that this might be a good place to > > substitute a call to PyString_FromFormat*() but then I read this > > little nugget: > > > > case 'd': case 'i': case 'x': > > (void) va_arg(count, int); > > /* 20 bytes should be enough to > > hold a 64-bit > > integer */ > > n += 20; > > break; > > This use of 20 is fine (I checked all occurrences of "20" in the codebase, > BTW); int_repr's use of 20 was wrong because it failed to allow for the > trailing \0 byte too; the loop in PyString_FromFormatV doesn't have to worry > about that (PyString_FromStringAndSize() later adds 1 of its own for the > trailing \0 -- of course it doesn't actually add anything, but it adds 1 "in > effect" ). > > > > > --__--__-- > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > > > End of Python-Dev Digest From tim.one@home.com Sun Dec 2 08:41:28 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 2 Dec 2001 03:41:28 -0500 Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms In-Reply-To: <200112010806.DAA06935@cj20424-a.reston1.va.home.com> Message-ID: [Guido] > ... > Note that the Tru64 output suggests that the behavior (== effect on > the buffer) may differ too: a truncated version of the desired output > may or may not be written to the buffer. Which later turned out to be our fault. However, the MS native _snprintf is worse than that: if the buffer isn't big enough, it fills it to the end *without* a trailing \0. A subsequent (e.g.) PyString_FromString() would then pick up an arbitrarily long stretch of garbage bytes beyond the buffer's end. You can see the effect by, e.g., making int_repr's buffer 4 bytes, then doing repr(sys.maxint), on Windows. > This can be addressed by pre-filling the buffer with a useful pattern in > PyOS_snprintf before calling the system's snprintf -- e.g. "*-*-*-*" (to > give a patriotic example :-). I don't know why you'd want to burn time doing this -- IMO, if the buffer isn't big enough, it should almost always be due to an internal Python bug. The point of PyOS_snprintf isn't to let us be lazy about buffer decls or careless with formats, it's to prevent exploits even when we screw up. > PS: I feel like I'm writing this on borrowed time -- Comcast and > Excite@Home haven't disconnected me from the net yet. :-) It's quite the soap opera! AT&T pulled out of the talks, and AT&T's @Home customers were then cut off. The other partners are still negotiating. Moral of the story: if you ever start a moderately successful ISP, think twice before spending 6 billion dollars to acquire a portal. If you can't resist, at least avoid selling your services below cost to companies much bigger than you, counting on portal advertising dollars to make up the difference. if-it-cost-6-billion-it-must-be-worth-at-least-12-ly y'rs - tim From tim.one@home.com Sun Dec 2 08:44:28 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 2 Dec 2001 03:44:28 -0500 Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms In-Reply-To: <15369.3390.364861.781161@slothrop.digicool.com> Message-ID: [Jeremy Hylton] > The uses of PyOS_snprintf() in sysmodule.c actually checks the return > value. It always checked the sprintf() returned value and triggered a > Py_FatalError() if the return value was too big. I changed it to > print a warning message that the output was truncated. The test appeared off on both ends to me, so I fiddled this code. Please check the new version against your vision of sanity (my vision: there's nothing special about -1: any return value < 0 is a problem; and there's also a problem if the return value equals the buffer size, since the return value is exclusive of the \0 byte). From martin@v.loewis.de Sun Dec 2 10:21:30 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Sun, 2 Dec 2001 11:21:30 +0100 Subject: [Python-Dev] -DINET6 in Makefile In-Reply-To: <3C08D5DD.1F3E8B55@lemburg.com> (mal@lemburg.com) References: <3C050CAF.2A5CEDC5@lemburg.com> <200111302022.fAUKMO002089@mira.informatik.hu-berlin.de> <3C07F016.AEC20158@lemburg.com> <200112010107.fB117JO03160@mira.informatik.hu-berlin.de> <3C08D5DD.1F3E8B55@lemburg.com> Message-ID: <200112021021.fB2ALUl01747@mira.informatik.hu-berlin.de> > > Of course, doing so would be really stupid. Python.h *must* be the > > first include, or things may break. > > Why should "things break" ? I've doing this for years in lots > of Python extensions... Python.h configures the C library, e.g. for multi-threading (by defining _REENTRANT) or LFS (by defining _FILE_OFFSET_BITS). If you include system headers before Python.h, you may find that different headers are differently configured. In turn, compilation may fail or produce bogus code. If the module itself compiles correctly, you may still find that it is inconsistent with the python executable that is going to load it. > Back on topic: how are we going to get -DINET6 out of the > Makefile ? I just committed a change. Regards, Martin From mal@lemburg.com Sun Dec 2 10:26:41 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 02 Dec 2001 11:26:41 +0100 Subject: [Python-Dev] Differentiating $(EXT) between build and install References: <20011201232600.45BC4E2677@oratrix.oratrix.nl> Message-ID: <3C0A01E1.779B38DA@lemburg.com> Jack Jansen wrote: > > Folks, > there's so many bug reports and questions about needing the > --with-suffix on Mac OS X when you're building on an HFS+ filesystem > that I'd like to fix this. The warning in configure doesn't cut it > because it gets lost in all the other messages. > > What I'd like to do is have a separate Makefile variable, BUILDEXT, > that gives the extension to use for python while it lives in the build > directory. It would default to EXT, except if on HFS+ when EXT is > empty. > > But this might change a couple of other things as well, so my question > is: is this an acceptable change so late in the beta stage? Wouldn't tweaking the setup.py to introduce a special case for MacOS be much simpler ? (adding BUILDEXT would need patching of the Makefile.pre.in process as well as distutils) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From tim.one@home.com Sun Dec 2 10:36:24 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 2 Dec 2001 05:36:24 -0500 Subject: [Python-Dev] Subclassing varying length types (What's a PyStructSequence ?) In-Reply-To: <200112011353.fB1Drab17745@mira.informatik.hu-berlin.de> Message-ID: [Martin v. Loewis] > You should consider that malloc overhead is often 16 bytes per > object. Given that PyUnicodeObject is 24 bytes in 2.2, system malloc > will allocate 48 bytes per Unicode object on modern architectures. I > would think 100% overhead *is* a big argument. > > If you relate this to the actual data, it gets worse: A Unicode string > of length 1 would still require 32 bytes on an allocator that aligns > to 16. I think that's unusual -- 8-byte alignment is most common even on 64-bit boxes. KSR had to align to 128-byte boundaries, but there's a reason KSR died . > Therefore, to store 2 bytes of real data, you need 80 bytes of > memory. > > I don't know how much overhead pymalloc adds, though; I believe it is > significantly less expensive. Yes, much less. On a 32-bit box, using the current #define's, and ignoring "arena" overhead(*), pymalloc uses 32 bytes per 4096 for bookkeeping. The remaining 4064 bytes can all be user data, but subject to 8-byte alignment, and to how many whole chunks of a given size can fit in 4064 bytes. For the PyUnicodeObject example, 8-byte alignment is without cost, and for the rest >>> divmod(4096 - 32, 24) (169, 8) >>> That is, pymalloc can get 169 PyUnicodeObjects out of a 4KB "page", with 32 bytes for bookkeeping, and 8 bytes left over (unused) -- total overhead is about 1%. (*) pymalloc gets "arenas" from the system malloc, where an arena is currently 256KB. Up to (worst case) 4KB of that is lost to align the start address to a 4KB boundary, and there's also the comparatively trivial (compared to 4KB!) overhead from the system malloc. From loewis@informatik.hu-berlin.de Sun Dec 2 13:53:33 2001 From: loewis@informatik.hu-berlin.de (Martin von Loewis) Date: Sun, 2 Dec 2001 14:53:33 +0100 (MET) Subject: [Python-Dev] Release criteria Message-ID: <200112021353.fB2DrXZ14135@paros.informatik.hu-berlin.de> Now that there are only ten days left for the release candidate, I wonder whether there are any release criteria beyond "builds on Windows". In particular, I think that the release candidate should not be produced until "critical" SF bugs have been resolved. In #420851, Barry indicates that bugs with a priority higher than six will block a release. This sounds reasonable. Looking at the remaining bugs, we see Priority 8: two reports (assigned to loewis and nobody) Priority 7: 11 reports (6x akuchling, 3x gvanrossum, 1x jackjansen, 1x fdrake) Given that the oldest of these reports is from 2001-07-30, I think it is unlikely that they will be all resolved within ten days. That seems to suggest one of the following alternatives: 1. Move the deadline for the release candidate, 2. Lower the priority for some of the reports, 3. Move the threshold for release-critical bugs (to, say, 8), 4. Give up the notion of release-critical bugs (which I would regret, since I hope "my" prio-8-bug should be fixed; it'll just take some more days). Regards, Martin From jeremy@zope.com Sun Dec 2 17:47:28 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Sun, 2 Dec 2001 12:47:28 -0500 (EST) Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms In-Reply-To: References: <15369.3390.364861.781161@slothrop.digicool.com> Message-ID: <15370.26928.239833.870381@slothrop.digicool.com> >>>>> "TP" == Tim Peters writes: TP> [Jeremy Hylton] >> The uses of PyOS_snprintf() in sysmodule.c actually checks the >> return value. It always checked the sprintf() returned value and >> triggered a Py_FatalError() if the return value was too big. I >> changed it to print a warning message that the output was >> truncated. TP> The test appeared off on both ends to me, so I fiddled this TP> code. Please check the new version against your vision of TP> sanity (my vision: there's nothing special about -1: any return TP> value < 0 is a problem; and there's also a problem if the return TP> value equals the buffer size, since the return value is TP> exclusive of the \0 byte). Definitely correct on the latter. Sensibly conservative on the former. The Linux man pages are silent on negative return values other than -1. Thanks for the fixes. Jeremy From fdrake@acm.org Sun Dec 2 20:45:52 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Sun, 2 Dec 2001 15:45:52 -0500 Subject: [Python-Dev] Release criteria In-Reply-To: <200112021353.fB2DrXZ14135@paros.informatik.hu-berlin.de> References: <200112021353.fB2DrXZ14135@paros.informatik.hu-berlin.de> Message-ID: <15370.37632.689000.981891@grendel.zope.com> Martin von Loewis writes: > In particular, I think that the release candidate should not be > produced until "critical" SF bugs have been resolved. In #420851, > Barry indicates that bugs with a priority higher than six will block a > release. This sounds reasonable. > > Looking at the remaining bugs, we see > > Priority 8: two reports (assigned to loewis and nobody) > > Priority 7: 11 reports (6x akuchling, 3x gvanrossum, > 1x jackjansen, 1x fdrake) The one assigned to me really must be fixed before release. There are a number of people who use the indexes, and they can be the only effective way to use the affected documents much of the time. > Given that the oldest of these reports is from 2001-07-30, I think it > is unlikely that they will be all resolved within ten days. That seems > to suggest one of the following alternatives: > > 1. Move the deadline for the release candidate, > 2. Lower the priority for some of the reports, > 3. Move the threshold for release-critical bugs (to, say, 8), The one assigned to me is at priority 7 because that would block the release. Shift the threshold up and I shift the priority of that bug to match. > 4. Give up the notion of release-critical bugs (which I would regret, > since I hope "my" prio-8-bug should be fixed; it'll just take > some more days). I agree we should keep the concept. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tim.one@home.com Sun Dec 2 20:01:03 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 2 Dec 2001 15:01:03 -0500 Subject: [Python-Dev] Subclassing varying length types (What's a PyStructSequence ?) In-Reply-To: <3C08BE98.79946952@lemburg.com> Message-ID: [MAL] > Perhaps we should start thinking about optimizing at least > one of the Unicode malloc away in Python 2.3: the Unicode > object itself can well be kept on a free list with the > Py_UNICODE buffer freed and set to NULL. Doesn't save any memory > but would improve the performance. pymalloc would improve both, so I'd much rather pursue that in 2.3 than yet another type-specific free list. > BTW, is the memory burden really such a big argument these days ? > I can imagine this being an argument on resource restrained > platforms such as Palms (thanks to Martin, the Plam guys can now > switch off Unicode completely), but hardly on gigabyte machines > with access 100s of GBs swap-space :-) Most of us have machines between those extremes, and the difference between 100MB and 300MB can be make-or-break. I don't see that any "flexibility" is gained merely by wasting memory . > ... > Any idea how we could make subclassing these types less > hackish, then ? Subclassing seems easy enough to me from the Python level; I don't have time to revisit C-level subclasssing here (and I don't know that it's hackish there either, but do think it's in need of docs). From tim.one@home.com Sun Dec 2 20:06:06 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 2 Dec 2001 15:06:06 -0500 Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms In-Reply-To: <15370.26928.239833.870381@slothrop.digicool.com> Message-ID: [Jeremy] > Definitely correct on the latter. Sensibly conservative on the > former. The Linux man pages are silent on negative return values > other than -1. My draft copy of C99 sez: The snprintf function returns the number of characters that would have been written had n been sufficiently large, not counting the terminating null character, or a negative value if an encoding error occurred. -1 isn't mentioned at all. What does the final C99 say? Regardless, MS's pre-C99 implementation says: _snprintf returns the number of bytes stored in buffer, not counting the terminating null character. If the number of bytes required to store the data exceeds count, then count bytes of data are stored in buffer and a negative value is returned. So -1 sounds Linux-specific. > Thanks for the fixes. Thank *you* for the opportunity to be thorough . From tim.one@home.com Sun Dec 2 20:39:23 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 2 Dec 2001 15:39:23 -0500 Subject: [Python-Dev] Release criteria In-Reply-To: <200112021353.fB2DrXZ14135@paros.informatik.hu-berlin.de> Message-ID: [Martin von Loewis] > Now that there are only ten days left for the release candidate, I > wonder whether there are any release criteria beyond "builds on > Windows". Sure. > In particular, I think that the release candidate should not be > produced until "critical" SF bugs have been resolved. In #420851, > Barry indicates that bugs with a priority higher than six will block a > release. This sounds reasonable. Yes, that's part of The Plan. > Looking at the remaining bugs, we see > > Priority 8: two reports (assigned to loewis and nobody) The "nobody" bug is sprintf -> PyOS_snprintf conversion; it's not assigned to anyone in particular because 3 people in PythonLabs have been working on it. > Priority 7: 11 reports (6x akuchling, 3x gvanrossum, > 1x jackjansen, 1x fdrake) > > Given that the oldest of these reports is from 2001-07-30, I think it > is unlikely that they will be all resolved within ten days. Me too, although it's the Distutil (akuchling) bugs that seem most at risk. Note that I just boosted one of the regexp bugs to priority 7 (/F, are you there?). Guido returns from paternity leave tomorrow, so PythonLabs will triple its available manpower then <0.9 wink>. > That seems to suggest one of the following alternatives: > > 1. Move the deadline for the release candidate, Possible, but at this time I think it's unlikely. > 2. Lower the priority for some of the reports, Most likely, for the Distutils and Mac-specific bugs (no, we're not going to hold up the release just because the "copy" menu command on a Mac-specific console window isn't enabled). > 3. Move the threshold for release-critical bugs (to, say, 8), No. > 4. Give up the notion of release-critical bugs No. From fredrik@pythonware.com Sun Dec 2 21:04:54 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Sun, 2 Dec 2001 22:04:54 +0100 Subject: [Python-Dev] Release criteria References: Message-ID: <019f01c17b75$02aee150$ced241d5@hagrid> Tim Peters wrote: > Note that I just boosted one of the regexp bugs to priority 7 (/F, are you > there?). hmm. that bug appears to be fixed in the PythonWare repository, but for some reason, I never checked it in... I'll investigate. From tim.one@home.com Sun Dec 2 20:59:00 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 2 Dec 2001 15:59:00 -0500 Subject: [Python-Dev] Release criteria In-Reply-To: <019f01c17b75$02aee150$ced241d5@hagrid> Message-ID: [Tim] > Note that I just boosted one of the regexp bugs to priority 7 > (/F, are you there?). [/F] > hmm. that bug appears to be fixed in the PythonWare repository, > but for some reason, I never checked it in... > > I'll investigate. Thanks! If you can resolve this before 2.2c1 Eve, I'm sure Guido's new son will sleep much better . From jack@oratrix.nl Sun Dec 2 21:03:54 2001 From: jack@oratrix.nl (Jack Jansen) Date: Sun, 02 Dec 2001 22:03:54 +0100 Subject: [Python-Dev] Differentiating $(EXT) between build and install In-Reply-To: Message by "M.-A. Lemburg" , Sun, 02 Dec 2001 11:26:41 +0100 , <3C0A01E1.779B38DA@lemburg.com> Message-ID: <20011202210359.E1281E2677@oratrix.oratrix.nl> Recently, "M.-A. Lemburg" said: > > What I'd like to do is have a separate Makefile variable, BUILDEXT, > > that gives the extension to use for python while it lives in the build > > directory. It would default to EXT, except if on HFS+ when EXT is > > empty. > > Wouldn't tweaking the setup.py to introduce a special case > for MacOS be much simpler ? (adding BUILDEXT would need patching > of the Makefile.pre.in process as well as distutils) The problem is only with building the interpreter. Because HFS+ is case-insensitive creating an executable "python" when there's already a subdirectory "Python" isn't going to work. So the idea is to use the BUILDEXT to create "python.exe" in stead, but install that as the normal "python" in the eventual install location. I would assume that the extra variable BUILDEXT in Makefile.pre.in woulnd't bother distutils or anything else. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From guido@python.org Sun Dec 2 23:57:41 2001 From: guido@python.org (Guido van Rossum) Date: Sun, 02 Dec 2001 18:57:41 -0500 Subject: [Python-Dev] test_builtin failing? or just 64-bit platforms In-Reply-To: Your message of "Sun, 02 Dec 2001 15:06:06 EST." References: Message-ID: <200112022357.SAA16201@cj20424-a.reston1.va.home.com> > -1 isn't mentioned at all. What does the final C99 say? Regardless, MS's > pre-C99 implementation says: [...] > So -1 sounds Linux-specific. Being a Unix old-timer, I cringe every time I see someone test for errors using "== -1". Except for anomalies like getpid() (which in the olden days could return any negative value except -1 as a legitimate pid), for the caller *any* negative value should be interpreted as an error, and tested for with "< 0". Never mind that most functions return -1. -1000 is just as much an error. Unhelpfully, --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sun Dec 2 23:59:40 2001 From: guido@python.org (Guido van Rossum) Date: Sun, 02 Dec 2001 18:59:40 -0500 Subject: [Python-Dev] Release criteria In-Reply-To: Your message of "Sun, 02 Dec 2001 14:53:33 +0100." <200112021353.fB2DrXZ14135@paros.informatik.hu-berlin.de> References: <200112021353.fB2DrXZ14135@paros.informatik.hu-berlin.de> Message-ID: <200112022359.SAA16213@cj20424-a.reston1.va.home.com> > Now that there are only ten days left for the release candidate, I > wonder whether there are any release criteria beyond "builds on > Windows". And on Linux. :-) > In particular, I think that the release candidate should not be > produced until "critical" SF bugs have been resolved. In #420851, > Barry indicates that bugs with a priority higher than six will block a > release. This sounds reasonable. I still abide by this rule. > Looking at the remaining bugs, we see > > Priority 8: two reports (assigned to loewis and nobody) Yours is for BSDI, a tiny minority platform. You set the priority yourself, meaning you want to fix it before the release candidate. But if you don't fix it, I'd rather lower the priority to 6 than hold up the release. The other is the sprintf thing -- I believe that's been addressed, at least to a sufficient degree to warrant lowering the priority. > Priority 7: 11 reports (6x akuchling, 3x gvanrossum, > 1x jackjansen, 1x fdrake) I plan to address mine (I'm coming back to work Monday). Fred already promised to address his. Andrew has a whole bunch of distutils and setup.py bugs assigned. I'm not sure that these should be allowed to hold up the release -- there just doesn't seem to be anyone at PythonLabs who has the expertise to fix them, and frankly, I don't see these bugs as release showstoppers. So maybe their priority should be lowered -- although I'd prefer to keep them at the current priority, in the hope that when Andrew (or someone else) finds some time, they know which bugs to address first. That is, assuming that all the priorities have been assigned rationally, which I'm not so sure of. I think Jack desperately wants to fix his bug in time -- that's why he raised the priority. > Given that the oldest of these reports is from 2001-07-30, I think it > is unlikely that they will be all resolved within ten days. That seems > to suggest one of the following alternatives: > > 1. Move the deadline for the release candidate, Unlikely, unless we find a real showstopper (none of the above count). > 2. Lower the priority for some of the reports, Might do. > 3. Move the threshold for release-critical bugs (to, say, 8), No way. > 4. Give up the notion of release-critical bugs (which I would regret, > since I hope "my" prio-8-bug should be fixed; it'll just take > some more days). Nah. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Mon Dec 3 03:59:01 2001 From: barry@zope.com (Barry A. Warsaw) Date: Sun, 2 Dec 2001 22:59:01 -0500 Subject: [Python-Dev] Release criteria References: <200112021353.fB2DrXZ14135@paros.informatik.hu-berlin.de> <200112022359.SAA16213@cj20424-a.reston1.va.home.com> Message-ID: <15370.63621.213117.237336@anthem.wooz.org> Don't forget the patches too. Although I've only just skimmed them, it looks like all the patches are at priority 5, most likely because they haven't been triaged. I'll try to do a quick scan through the unassigned patches. -Barry From mal@lemburg.com Mon Dec 3 09:09:38 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 03 Dec 2001 10:09:38 +0100 Subject: [Python-Dev] Subclassing varying length types (What's a PyStructSequence ?) References: Message-ID: <3C0B4152.3739F4EA@lemburg.com> Tim Peters wrote: > > [MAL] > > Perhaps we should start thinking about optimizing at least > > one of the Unicode malloc away in Python 2.3: the Unicode > > object itself can well be kept on a free list with the > > Py_UNICODE buffer freed and set to NULL. Doesn't save any memory > > but would improve the performance. > > pymalloc would improve both, so I'd much rather pursue that in 2.3 than yet > another type-specific free list. Have you tried disabling all free list and using pymalloc instead ? If this pays off, I agree, we should get rid off all of them. > > BTW, is the memory burden really such a big argument these days ? > > I can imagine this being an argument on resource restrained > > platforms such as Palms (thanks to Martin, the Plam guys can now > > switch off Unicode completely), but hardly on gigabyte machines > > with access 100s of GBs swap-space :-) > > Most of us have machines between those extremes, and the difference between > 100MB and 300MB can be make-or-break. I don't see that any "flexibility" is > gained merely by wasting memory . I would consider moving from 8-bit strings to Unicode an improvement in flexibility. It also results in better algroithms (== simpler, less error-prone, etc. in this case). As I said, it's a tradeoff flexibility vs. memory consumption. Whether it pays off depends on your application environment. It certainly does for companies like Micron and pays off stock-wise for a lot of people... uhm, getting off-topic here :-) > > ... > > Any idea how we could make subclassing these types less > > hackish, then ? > > Subclassing seems easy enough to me from the Python level; I don't have time > to revisit C-level subclasssing here (and I don't know that it's hackish > there either, but do think it's in need of docs). It is beautifully easy for non-varying-length types. Unfortunately, it happens that some of the basic types which would be attractive for subclassing are varying length types (such as string and tuples). In my case, I'm looking for away to subclass strings, but I haven't yet found an elegant solution to the problem of adding extra data to the instances. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Mon Dec 3 09:25:47 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 03 Dec 2001 10:25:47 +0100 Subject: [Python-Dev] -DINET6 in Makefile References: <3C050CAF.2A5CEDC5@lemburg.com> <200111302022.fAUKMO002089@mira.informatik.hu-berlin.de> <3C07F016.AEC20158@lemburg.com> <200112010107.fB117JO03160@mira.informatik.hu-berlin.de> <3C08D5DD.1F3E8B55@lemburg.com> <200112021021.fB2ALUl01747@mira.informatik.hu-berlin.de> Message-ID: <3C0B451B.18D38186@lemburg.com> "Martin v. Loewis" wrote: > > > > Of course, doing so would be really stupid. Python.h *must* be the > > > first include, or things may break. > > > > Why should "things break" ? I've doing this for years in lots > > of Python extensions... > > Python.h configures the C library, e.g. for multi-threading (by > defining _REENTRANT) or LFS (by defining _FILE_OFFSET_BITS). > > If you include system headers before Python.h, you may find that > different headers are differently configured. In turn, compilation may > fail or produce bogus code. > > If the module itself compiles correctly, you may still find that it is > inconsistent with the python executable that is going to load it. Sure, but the wrapped lib's headers files will expect the same logic (they want to configure the C lib too; and for pretty much the same reasons Python does). In the end I think that you can't really say whether things break or not due to the order of the #includes (lib header or Python.h first) because both can make assumptions which may fail on way or another. I'd close this with "Works for me" :-) > > Back on topic: how are we going to get -DINET6 out of the > > Makefile ? > > I just committed a change. Saw that checkin. Thanks for fixing this. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Mon Dec 3 09:30:04 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 03 Dec 2001 10:30:04 +0100 Subject: [Python-Dev] Differentiating $(EXT) between build and install References: <20011202210359.E1281E2677@oratrix.oratrix.nl> Message-ID: <3C0B461C.FE51188F@lemburg.com> Jack Jansen wrote: > > Recently, "M.-A. Lemburg" said: > > > What I'd like to do is have a separate Makefile variable, BUILDEXT, > > > that gives the extension to use for python while it lives in the build > > > directory. It would default to EXT, except if on HFS+ when EXT is > > > empty. > > > > Wouldn't tweaking the setup.py to introduce a special case > > for MacOS be much simpler ? (adding BUILDEXT would need patching > > of the Makefile.pre.in process as well as distutils) > > The problem is only with building the interpreter. Because HFS+ is > case-insensitive creating an executable "python" when there's already > a subdirectory "Python" isn't going to work. So the idea is to use the > BUILDEXT to create "python.exe" in stead, but install that as the > normal "python" in the eventual install location. > > I would assume that the extra variable BUILDEXT in Makefile.pre.in > woulnd't bother distutils or anything else. Ah, now I understand (I thought you were talking about compiling the extensions). No objections then for the short term... however, for the long term I guess we'll need some better solution like e.g. renaming Python/ to Interpreter/. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From martin@v.loewis.de Mon Dec 3 18:02:51 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Mon, 3 Dec 2001 19:02:51 +0100 Subject: [Python-Dev] Differentiating $(EXT) between build and install In-Reply-To: <3C0B461C.FE51188F@lemburg.com> (mal@lemburg.com) References: <20011202210359.E1281E2677@oratrix.oratrix.nl> <3C0B461C.FE51188F@lemburg.com> Message-ID: <200112031802.fB3I2po01345@mira.informatik.hu-berlin.de> > Ah, now I understand (I thought you were talking about compiling > the extensions). No objections then for the short term... however, > for the long term I guess we'll need some better solution like > e.g. renaming Python/ to Interpreter/. Or rename "python" to "parrot"? Regards, Martin From fdrake@acm.org Mon Dec 3 21:25:48 2001 From: fdrake@acm.org (Fred L. Drake) Date: Mon, 3 Dec 2001 16:25:48 -0500 (EST) Subject: [Python-Dev] [development doc updates] Message-ID: <20011203212548.5590C28698@cj42289-a.reston1.va.home.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Added documentation for the ScrolledText module. From skip@pobox.com (Skip Montanaro) Tue Dec 4 00:44:03 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 3 Dec 2001 18:44:03 -0600 Subject: [Python-Dev] Going off-list for awhile Message-ID: <15372.7251.365887.715297@beluga.mojam.com> Folks, I've set my various python-related subscriptions to nomail for the time being. Both of our web servers (www.musi-cal.com and www.mojam.com) were recently broken into and then used as launching points for attacks on other machines. They are both offline (probably for good). I am scrambling to try and get a new server set up and secured. Hopefully I'll be back in business in a few days. -- Skip Montanaro (skip@pobox.com - http://www.mojam.com/) From neal@metaslash.com Tue Dec 4 02:18:06 2001 From: neal@metaslash.com (Neal Norwitz) Date: Mon, 03 Dec 2001 21:18:06 -0500 Subject: [Python-Dev] pychecker & python 2.2 Message-ID: <3C0C325E.BD592F56@metaslash.com> Guido had mentioned awhile ago about possibly adding pychecker to Tools. If people want it in, what needs to be done? There are 2 differences between the last released version of pychecker and the current CVS. 1 is a bug fix for inconsistent return types and the other is a new warning for local variables which shadow globals. I will run the current pychecker on the std library soon and report any problems. I have been running purify and all of the "major" problems have bugs reports. These "major" problems are memory leaks, the biggest is about 5k from test_parser which leaks when doing recursion in build_node_children() parsermodule.c:728. Neal From neal@metaslash.com Tue Dec 4 02:42:14 2001 From: neal@metaslash.com (Neal Norwitz) Date: Mon, 03 Dec 2001 21:42:14 -0500 Subject: [Python-Dev] pychecker on 2.2 current Message-ID: <3C0C3806.B7325138@metaslash.com> Only found one potential problem: mimetypes.py:405: No global (sys) found This is inside usage() which presumably should only be used internally. sys is imported if __name__ == '__main__'. So the problem can only be triggered if usage() is called by external source. Possible fixes are: sys could be locally imported in usage() sys could be imported at the module scope usage could be made _usage() to denote it's private Neal From ping@lfw.org Tue Dec 4 12:54:25 2001 From: ping@lfw.org (Ka-Ping Yee) Date: Tue, 4 Dec 2001 04:54:25 -0800 (PST) Subject: [Python-Dev] __getattribute__ should require a string? Message-ID: In the current Python, __getattribute__ can be called directly with a non-string, causing much weirdness: >>> print ''.__getattribute__(1) Traceback (most recent call last): File "", line 1, in ? AttributeError: 'str' object has no attribute ' =E4=B7*' >>>=20 To avoid this, i've added a simple check to Objects/object.c: diff -c -r2.161 object.c *** object.c 2001/11/04 07:29:31 2.161 --- object.c 2001/12/04 12:46:13 *************** *** 1210,1215 **** --- 1210,1221 ---- descrgetfunc f; PyObject **dictptr; =20 + if (!PyString_Check(name)) { + PyErr_SetString(PyExc_TypeError, + "attribute name must be string"); + return NULL; + } +=20 if (tp->tp_dict =3D=3D NULL) { if (PyType_Ready(tp) < 0) return NULL; This produces the better behaviour: >>> print ''.__getattribute__(1) Traceback (most recent call last): File "", line 1, in ? TypeError: attribute name must be string >>>=20 I was about to check in this simple fix -- but then i discovered that it is currently possible to assign attributes with non-string keys: >>> def f(): pass ...=20 >>> f.__getattribute__(1) Traceback (most recent call last): File "", line 1, in ? AttributeError: 'function' object has no attribute ' =E4=B7*' >>> f.__setattr__(1, 1) >>> f.__getattribute__(1) 1 >>>=20 Adding the above check prevents this usage. Is there any reason why people should be allowed to assign and retrieve attributes with non-string names? -- ?!ng From guido@python.org Tue Dec 4 13:31:32 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 04 Dec 2001 08:31:32 -0500 Subject: [Python-Dev] __getattribute__ should require a string? In-Reply-To: Your message of "Tue, 04 Dec 2001 04:54:25 PST." References: Message-ID: <200112041331.IAA06968@cj20424-a.reston1.va.home.com> > Is there any reason why people should be allowed to assign and > retrieve attributes with non-string names? Not really, but why should we actively try to prevent it when it's easy to do self.__dict__[1] = 'a'? This means you can't rely on all attribute names being strings, and I'm not going to prevent *that*. The proper fix IMO for the bug you found is to be more careful when formatting the error message. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Dec 4 15:32:27 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 04 Dec 2001 10:32:27 -0500 Subject: [Python-Dev] __getattribute__ should require a string? In-Reply-To: Your message of "Tue, 04 Dec 2001 08:31:32 EST." <200112041331.IAA06968@cj20424-a.reston1.va.home.com> References: <200112041331.IAA06968@cj20424-a.reston1.va.home.com> Message-ID: <200112041532.fB4FWR011720@odiug.zope.com> > > Is there any reason why people should be allowed to assign and > > retrieve attributes with non-string names? > > Not really, but why should we actively try to prevent it when it's > easy to do self.__dict__[1] = 'a'? This means you can't rely on all > attribute names being strings, and I'm not going to prevent *that*. > > The proper fix IMO for the bug you found is to be more careful when > formatting the error message. I take it back. All the other functions in object.c require the getattr/setattr argument to be a string or Unicode object. I'll fix object.__getattribute__ and object.__setattr__ (which are really PyObject_Generic{Get,Set}Attr) to do the same. --Guido van Rossum (home page: http://www.python.org/~guido/) From arigo@ulb.ac.be Tue Dec 4 16:26:34 2001 From: arigo@ulb.ac.be (Armin Rigo) Date: Tue, 4 Dec 2001 17:26:34 +0100 (MET) Subject: [Python-Dev] Psyco to SourceForge Message-ID: Hello everybody, I have opened a "Psyco" project at SourceForge: http://sourceforge.net/projects/psyco/ I suggest that people interested in Psyco post a welcome and start discussion in the mailing list that I am currently opening there. This will allow me to stop (mis)using python-dev@python.org for announces. The source code I have uploaded at SourceForge (and in a .tgz archive at the Psyco homepage) is version 0.3.4, which is an important rewrite over the previous version. I have tried to address the two 'bad design' issues I mentionned in ISSUES.txt: * pycompiler.c-and-friends are much clearer now, thanks to a clean interface in vcompiler.h and processor.h to all the gory details; * the structure of the Python-dependent part Psyco now very closely follows that of the Python interpreter. Anyone used to the Objects/xxx.c files of Python should feel at home with Psyco's objects/pxxx.c files. In fact, they are just a rewrite of the Python interpreter into some "meta-C" code useable by the specializer of Psyco. If you gave up while trying to read the code of 0.3.3 please have another look now :-) Thanks, Armin Rigo. From fdrake@acm.org Tue Dec 4 17:03:32 2001 From: fdrake@acm.org (Fred L. Drake) Date: Tue, 4 Dec 2001 12:03:32 -0500 (EST) Subject: [Python-Dev] [development doc updates] Message-ID: <20011204170332.0B76A28698@beowolf.digicool.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ The indexes are now working for the Python/C API manual and the Reference Manual. From thomas.heller@ion-tof.com Tue Dec 4 19:35:19 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 4 Dec 2001 20:35:19 +0100 Subject: [Python-Dev] nested scopes confusion Message-ID: <083a01c17cfa$e13f6120$e000a8c0@thomasnotebook> I would have expected the following code to print 1, 2, ..., 9. Instead it prints 25, 25, 25, .. 25. def functions(): result = [] for i in range(10): def mth(*args): return i result.append(mth) i = 25 return result for mth in functions(): print mth() Reading PEP227, I can (barely) understand why it behaves this way. How do I achieve the desired effect? Note that the default argument trick (def mth(i=i): ...) does not work because *args is present. Thomas From thomas.heller@ion-tof.com Tue Dec 4 19:47:59 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 4 Dec 2001 20:47:59 +0100 Subject: [Python-Dev] nested scopes confusion References: <083a01c17cfa$e13f6120$e000a8c0@thomasnotebook> Message-ID: <08dc01c17cfc$a5ef22c0$e000a8c0@thomasnotebook> From: "Thomas Heller" > How do I achieve the desired effect? Note that the default argument > trick (def mth(i=i): ...) does not work because *args is present. Answering my own question (sorry), this works: def functions(): result = [] for i in range(10): def make_function(index): def mth(): return index return mth mth = make_function(i) result.append(mth) i = 25 return result for mth in functions(): print mth() But how will I understand this 3 months from now? Thomas From guido@python.org Tue Dec 4 19:47:23 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 04 Dec 2001 14:47:23 -0500 Subject: [Python-Dev] nested scopes confusion In-Reply-To: Your message of "Tue, 04 Dec 2001 20:35:19 +0100." <083a01c17cfa$e13f6120$e000a8c0@thomasnotebook> References: <083a01c17cfa$e13f6120$e000a8c0@thomasnotebook> Message-ID: <200112041947.fB4JlWR24198@odiug.zope.com> > I would have expected the following code to print > 1, 2, ..., 9. Instead it prints 25, 25, 25, .. 25. > > def functions(): > result = [] > for i in range(10): > def mth(*args): return i > result.append(mth) > i = 25 > return result > > for mth in functions(): > print mth() > > Reading PEP227, I can (barely) understand why it behaves this way. Yes, you're out of luck. Like in all decent languages with nested scope, Python binds to the variable, not to its value at the time the inner function is defined. > How do I achieve the desired effect? Note that the default argument > trick (def mth(i=i): ...) does not work because *args is present. Use a class with a __call__ method: class mth: def __init__(self, i): self.i = i def __call__(self, *args): return self.i and use: result.append(mth(i)) --Guido van Rossum (home page: http://www.python.org/~guido/) From thomas.heller@ion-tof.com Tue Dec 4 19:58:59 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 4 Dec 2001 20:58:59 +0100 Subject: [Python-Dev] nested scopes confusion References: <083a01c17cfa$e13f6120$e000a8c0@thomasnotebook> <200112041947.fB4JlWR24198@odiug.zope.com> Message-ID: <090401c17cfe$2ef12ea0$e000a8c0@thomasnotebook> > Yes, you're out of luck. Like in all decent languages with nested > scope, Python binds to the variable, not to its value at the time the > inner function is defined. So nested scopes are not really an alternative (at least in this case) to the 'lambda self, i=i: i' trick. > > > How do I achieve the desired effect? Note that the default argument > > trick (def mth(i=i): ...) does not work because *args is present. > > Use a class with a __call__ method: That's what I was trying to avoid, because it is too slow. Thanks, Thomas From jeremy@zope.com Tue Dec 4 19:57:23 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Tue, 4 Dec 2001 14:57:23 -0500 (EST) Subject: [Python-Dev] nested scopes confusion In-Reply-To: <083a01c17cfa$e13f6120$e000a8c0@thomasnotebook> References: <083a01c17cfa$e13f6120$e000a8c0@thomasnotebook> Message-ID: <15373.10915.984130.453517@slothrop.digicool.com> > def functions(): > result = [] > for i in range(10): > def mth(*args): return i > result.append(mth) > i = 25 > return result > > for mth in functions(): > print mth() >>>>> "TH" == Thomas Heller writes: TH> Reading PEP227, I can (barely) understand why it behaves this TH> way. It behaves this way because the reference to i in mth() uses the binding for i introduced in functions(). The important point here is that the binding is used, but i can be bound to different values at different times. The function mth() sees the value that i is bound to when it is called. In your example, mth() isn't called until after the loop finishes executing. TH> How do I achieve the desired effect? Note that the default TH> argument trick (def mth(i=i): ...) does not work because *args TH> is present. All sorts of ways, I imagine. You could use a class: class MthClass: def __init__(self, val): self.val = val def mth(self, *args): return self.val This version is probably immediately obvious to the reader. You could add an extra layer of functions, but this looks like a pretty obscure use of nested scopes to me. >>> def functions(): ... result = [] ... for i in range(10): ... def wrap(i): ... def mth(*args): ... return i ... return mth ... result.append(wrap(i)) ... i = 25 ... return result ... >>> l = functions() >>> l[0]() 0 >>> l[1]() 1 Jeremy From guido@python.org Tue Dec 4 19:59:39 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 04 Dec 2001 14:59:39 -0500 Subject: [Python-Dev] nested scopes confusion In-Reply-To: Your message of "Tue, 04 Dec 2001 20:47:59 +0100." <08dc01c17cfc$a5ef22c0$e000a8c0@thomasnotebook> References: <083a01c17cfa$e13f6120$e000a8c0@thomasnotebook> <08dc01c17cfc$a5ef22c0$e000a8c0@thomasnotebook> Message-ID: <200112041959.fB4Jxdn24268@odiug.zope.com> > Answering my own question (sorry), this works: > > > def functions(): > result = [] > for i in range(10): > def make_function(index): > def mth(): > return index > return mth > mth = make_function(i) > result.append(mth) > i = 25 > return result > > for mth in functions(): > print mth() > > But how will I understand this 3 months from now? What's the problem with understanding this code? It looks totally clear to me. You could add a few comments or use more fancyful function names. --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy@zope.com Tue Dec 4 20:06:53 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Tue, 4 Dec 2001 15:06:53 -0500 (EST) Subject: [Python-Dev] nested scopes confusion In-Reply-To: <090401c17cfe$2ef12ea0$e000a8c0@thomasnotebook> References: <083a01c17cfa$e13f6120$e000a8c0@thomasnotebook> <200112041947.fB4JlWR24198@odiug.zope.com> <090401c17cfe$2ef12ea0$e000a8c0@thomasnotebook> Message-ID: <15373.11485.440768.572641@slothrop.digicool.com> >>>>> "TH" == Thomas Heller writes: >> > How do I achieve the desired effect? Note that the default >> > argument trick (def mth(i=i): ...) does not work because *args >> > is present. >> >> Use a class with a __call__ method: TH> That's what I was trying to avoid, because it is too slow. Calling a regular method on an instance is about 5x faster than calling an instance that defines an __call__(). But the performance characteristics of class vs. function is really the bottleneck of your application? Are you sure that nested fuctions are really cheaper? Jeremy From thomas.heller@ion-tof.com Tue Dec 4 20:28:17 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 4 Dec 2001 21:28:17 +0100 Subject: [Python-Dev] nested scopes confusion References: <083a01c17cfa$e13f6120$e000a8c0@thomasnotebook><200112041947.fB4JlWR24198@odiug.zope.com><090401c17cfe$2ef12ea0$e000a8c0@thomasnotebook> <15373.11485.440768.572641@slothrop.digicool.com> Message-ID: <097901c17d02$46f6a3a0$e000a8c0@thomasnotebook> > But the performance characteristics of class vs. function is really > the bottleneck of your application? Are you sure that nested fuctions > are really cheaper? Is seems do. Here's a simple test: def Method(index): return lambda self, *args: index class A: def test(self, *args): return 5 class C: test = Method(42) a = A() c = C() Calling a.test() takes 3.3 microseconds, and c.test() takes 3.16 microseconds. Thomas From thomas.heller@ion-tof.com Tue Dec 4 20:37:41 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 4 Dec 2001 21:37:41 +0100 Subject: [Python-Dev] nested scopes confusion References: <083a01c17cfa$e13f6120$e000a8c0@thomasnotebook><200112041947.fB4JlWR24198@odiug.zope.com><090401c17cfe$2ef12ea0$e000a8c0@thomasnotebook> <15373.11485.440768.572641@slothrop.digicool.com> <097901c17d02$46f6a3a0$e000a8c0@thomasnotebook> Message-ID: <099f01c17d03$97d83f30$e000a8c0@thomasnotebook> From: "Thomas Heller" > > But the performance characteristics of class vs. function is really > > the bottleneck of your application? Are you sure that nested fuctions > > are really cheaper? [...] What were I talking about ;-) ? These are functions using nested scopes, not nested functions. Sorry, Thomas From jeremy@zope.com Tue Dec 4 20:41:29 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Tue, 4 Dec 2001 15:41:29 -0500 (EST) Subject: [Python-Dev] nested scopes confusion In-Reply-To: <097901c17d02$46f6a3a0$e000a8c0@thomasnotebook> References: <083a01c17cfa$e13f6120$e000a8c0@thomasnotebook> <200112041947.fB4JlWR24198@odiug.zope.com> <090401c17cfe$2ef12ea0$e000a8c0@thomasnotebook> <15373.11485.440768.572641@slothrop.digicool.com> <097901c17d02$46f6a3a0$e000a8c0@thomasnotebook> Message-ID: <15373.13561.525007.203007@slothrop.digicool.com> >>>>> "TH" == Thomas Heller writes: >> But the performance characteristics of class vs. function is >> really the bottleneck of your application? Are you sure that >> nested fuctions are really cheaper? TH> Calling a.test() takes 3.3 microseconds, and c.test() takes 3.16 TH> microseconds. So I 5% difference, based on a single set of measurements with one compiler/platform/etc., is enough to go out of your way to avoid classes? That seems conceivable for some very small set of applications, but not in general. Jeremy From thomas.heller@ion-tof.com Tue Dec 4 21:04:31 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 4 Dec 2001 22:04:31 +0100 Subject: [Python-Dev] nested scopes confusion References: <083a01c17cfa$e13f6120$e000a8c0@thomasnotebook><200112041947.fB4JlWR24198@odiug.zope.com><090401c17cfe$2ef12ea0$e000a8c0@thomasnotebook><15373.11485.440768.572641@slothrop.digicool.com><097901c17d02$46f6a3a0$e000a8c0@thomasnotebook> <15373.13561.525007.203007@slothrop.digicool.com> Message-ID: <0a2901c17d07$570bf8d0$e000a8c0@thomasnotebook> From: "Jeremy Hylton" > >>>>> "TH" == Thomas Heller writes: > > >> But the performance characteristics of class vs. function is > >> really the bottleneck of your application? Are you sure that > >> nested fuctions are really cheaper? > > TH> Calling a.test() takes 3.3 microseconds, and c.test() takes 3.16 > TH> microseconds. > > So I 5% difference, based on a single set of measurements with one > compiler/platform/etc., is enough to go out of your way to avoid > classes? No. The test only shows (or suggests for this particular compiler/platform) that calling methods (which are using nested scopes) has the same speed than normal methods and not a factor of 5 or whatever slowdown of callable classes. > That seems conceivable for some very small set of > applications, but not in general. True. Thomas From martin@v.loewis.de Tue Dec 4 20:17:46 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Tue, 4 Dec 2001 21:17:46 +0100 Subject: [Python-Dev] nested scopes confusion In-Reply-To: <090401c17cfe$2ef12ea0$e000a8c0@thomasnotebook> (thomas.heller@ion-tof.com) References: <083a01c17cfa$e13f6120$e000a8c0@thomasnotebook> <200112041947.fB4JlWR24198@odiug.zope.com> <090401c17cfe$2ef12ea0$e000a8c0@thomasnotebook> Message-ID: <200112042017.fB4KHkU01849@mira.informatik.hu-berlin.de> > So nested scopes are not really an alternative (at least in this case) > to the 'lambda self, i=i: i' trick. In this case, they aren't. In many other cases (e.g. when the variable from the outer scope is self), nobody will assign to this variable, in which case this does exactly the same thing as the i=i trick. Regards, Martin From nas@python.ca Tue Dec 4 22:36:06 2001 From: nas@python.ca (Neil Schemenauer) Date: Tue, 4 Dec 2001 14:36:06 -0800 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src configure,1.274,1.275 configure.in,1.283,1.284 In-Reply-To: ; from fdrake@users.sourceforge.net on Tue, Dec 04, 2001 at 12:55:49PM -0800 References: Message-ID: <20011204143606.A19738@glacier.arctrix.com> > + if test "$Py_DEBUG" = 'true'; then > + : > + else > + OPT="-DNDEBUG $OPT" > + fi > + Shouldn't this go into DEFS? Neil From fdrake@acm.org Tue Dec 4 23:26:27 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 4 Dec 2001 18:26:27 -0500 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src configure,1.274,1.275 configure.in,1.283,1.284 In-Reply-To: <20011204143606.A19738@glacier.arctrix.com> References: <20011204143606.A19738@glacier.arctrix.com> Message-ID: <15373.23459.612323.606576@grendel.zope.com> Neil Schemenauer writes: > > + if test "$Py_DEBUG" = 'true'; then > > + : > > + else > > + OPT="-DNDEBUG $OPT" > > + fi > > + > > Shouldn't this go into DEFS? I didn't see DEFS getting set anywhere else, and other places in configure.in add -D options to OPT, CC, or CCSHARED. I've no personal preference though; if someone with more autokarma than I changes it, I won't scream. ;-) -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From guido@python.org Tue Dec 4 23:12:51 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 04 Dec 2001 18:12:51 -0500 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src configure,1.274,1.275 configure.in,1.283,1.284 In-Reply-To: Your message of "Tue, 04 Dec 2001 18:26:27 EST." <15373.23459.612323.606576@grendel.zope.com> References: <20011204143606.A19738@glacier.arctrix.com> <15373.23459.612323.606576@grendel.zope.com> Message-ID: <200112042312.SAA07545@cj20424-a.reston1.va.home.com> > Neil Schemenauer writes: > > > + if test "$Py_DEBUG" = 'true'; then > > > + : > > > + else > > > + OPT="-DNDEBUG $OPT" > > > + fi > > > + > > > > Shouldn't this go into DEFS? > > I didn't see DEFS getting set anywhere else, and other places in > configure.in add -D options to OPT, CC, or CCSHARED. I've no personal > preference though; if someone with more autokarma than I changes it, I > won't scream. ;-) > > -Fred I think DEFS gets put in pyconfig.h, which gets included by Python.h, defeating the whole purpose. --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Wed Dec 5 05:49:19 2001 From: fdrake@acm.org (Fred L. Drake) Date: Wed, 5 Dec 2001 00:49:19 -0500 (EST) Subject: [Python-Dev] [development doc updates] Message-ID: <20011205054919.A1E7B28698@beowolf.digicool.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ A wide variety of minor additions and clarifications. From naliniprakash@earthlink.net Wed Dec 5 06:09:43 2001 From: naliniprakash@earthlink.net (Nalini Prakash) Date: Tue, 4 Dec 2001 22:09:43 -0800 Subject: [Python-Dev] Building PyXPCOM from Mozilla source/ImportError: cannot import name _xpcom Message-ID: <000801c17d53$6fd608a0$3953afd8@popcorn> This is a multi-part message in MIME format. ------=_NextPart_000_0005_01C17D10.60B38110 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Apologize if this is not the right group for this question. I am trying = to build PyXPCOM from Mozilla 0.9.6 source tree on Win 2000. I have = managed to do make successfully and I can import xpcom. When I import = xpcom._xpcom, I get "ImportError: cannot import name _xpcom" and the = path includes Mozilla bin directory. The configure.html file hints at = something being wrong with _xpcom(_d).pyd.=20 How do I fix this problem? TIA, Praki ------=_NextPart_000_0005_01C17D10.60B38110 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Apologize if this is not the right = group for this=20 question. I am trying to build PyXPCOM from Mozilla 0.9.6 source tree on = Win=20 2000. I have managed to do make successfully and I can import xpcom. = When I=20 import xpcom._xpcom, I get "ImportError: cannot import name _xpcom" and = the path=20 includes Mozilla bin directory. The configure.html file hints at = something being=20 wrong with _xpcom(_d).pyd.
 
How do I fix this problem?
 
TIA, Praki
------=_NextPart_000_0005_01C17D10.60B38110-- From MarkH@ActiveState.com Wed Dec 5 06:38:38 2001 From: MarkH@ActiveState.com (Mark Hammond) Date: Wed, 5 Dec 2001 17:38:38 +1100 Subject: [Python-Dev] Building PyXPCOM from Mozilla source/ImportError: cannot import name _xpcom In-Reply-To: <000801c17d53$6fd608a0$3953afd8@popcorn> Message-ID: The correct forum for this is the Python XPCOM mailing list - details at http://listserv.ActiveState.com/mailman/listinfo/pyxpcom Please direct all followups there. Your PYTHONPATH needs to be set so that the main "xpcom" Python package is set on it. This directory should have an "_xpcom.pyd" (or xpcom_d.pyd for debug builds). I am guessing that this .pyd simply does not exist in that directory. If you receive another message that you omitted mentioning, regarding a required .dll file not being found, then the problem is likely that the PATH is not set correctly - but it appears from your message that it is OK. Hope this helps, Mark. -----Original Message----- From: python-dev-admin@python.org [mailto:python-dev-admin@python.org]On Behalf Of Nalini Prakash Sent: Wednesday, 5 December 2001 5:10 PM To: python-dev@python.org Subject: [Python-Dev] Building PyXPCOM from Mozilla source/ImportError: cannot import name _xpcom Apologize if this is not the right group for this question. I am trying to build PyXPCOM from Mozilla 0.9.6 source tree on Win 2000. I have managed to do make successfully and I can import xpcom. When I import xpcom._xpcom, I get "ImportError: cannot import name _xpcom" and the path includes Mozilla bin directory. The configure.html file hints at something being wrong with _xpcom(_d).pyd. How do I fix this problem? TIA, Praki From anthony@interlink.com.au Wed Dec 5 07:05:29 2001 From: anthony@interlink.com.au (Anthony Baxter) Date: Wed, 05 Dec 2001 18:05:29 +1100 Subject: [Python-Dev] python2.1.2 plans, plus the curse of PEP-006. Message-ID: <200112050705.fB575UN09520@mbuna.arbhome.com.au> Ok, after a bit of an unfortunate break I'm back into the backporting. My current plan is to have a release candidate for 2.1.2 due a couple of days before Christmas, and then try to get it tested as much as possible before a release either in the week between christmas/new year (not optimal) or shortly after. I'd hoped to get this stuff done much sooner, but unfortunately a combination of various personal matters has knocked me out for a couple of weeks. (I'm not sure if it's just me, but it's possible the reason no-one's done two bugfix releases in a row is that PEP-006 is accursed, and brings down the wrath of gods on anyone foolish enough to get involved. I'm hoping that the last few weeks horror is over and done now...) Anthony -- Anthony Baxter It's never too late to have a happy childhood. From mal@lemburg.com Wed Dec 5 08:28:18 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 05 Dec 2001 09:28:18 +0100 Subject: [Python-Dev] nested scopes confusion References: <083a01c17cfa$e13f6120$e000a8c0@thomasnotebook> <08dc01c17cfc$a5ef22c0$e000a8c0@thomasnotebook> Message-ID: <3C0DDAA2.F12F365C@lemburg.com> Thomas Heller wrote: > > From: "Thomas Heller" > > How do I achieve the desired effect? Note that the default argument > > trick (def mth(i=i): ...) does not work because *args is present. > Answering my own question (sorry), this works: > > def functions(): > result = [] > for i in range(10): > def make_function(index): > def mth(): > return index > return mth > mth = make_function(i) > result.append(mth) > i = 25 > return result > > for mth in functions(): > print mth() > > But how will I understand this 3 months from now? Why do you put the function definitions *inside* the functions() constructor ? AFAICTL the confusion is caused by this overly complicated nesting which doesn't seem necessary. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From guido@python.org Wed Dec 5 13:43:02 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Dec 2001 08:43:02 -0500 Subject: [Python-Dev] python2.1.2 plans, plus the curse of PEP-006. In-Reply-To: Your message of "Wed, 05 Dec 2001 18:05:29 +1100." <200112050705.fB575UN09520@mbuna.arbhome.com.au> References: <200112050705.fB575UN09520@mbuna.arbhome.com.au> Message-ID: <200112051343.IAA13534@cj20424-a.reston1.va.home.com> Welcome back, Anthony! Glad to hear you haven't given up. If there's anything we can do to alleviate the curse, let us know... --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Wed Dec 5 14:49:04 2001 From: barry@zope.com (Barry A. Warsaw) Date: Wed, 5 Dec 2001 09:49:04 -0500 Subject: [Python-Dev] python2.1.2 plans, plus the curse of PEP-006. References: <200112050705.fB575UN09520@mbuna.arbhome.com.au> Message-ID: <15374.13280.512488.799375@anthem.wooz.org> >>>>> "AB" == Anthony Baxter writes: AB> (I'm not sure if it's just me, but it's possible the reason AB> no-one's done two bugfix releases in a row is that PEP-006 is AB> accursed, and brings down the wrath of gods on anyone foolish AB> enough to get involved. I'm hoping that the last few weeks AB> horror is over and done now...) Anthony, First, thanks for taking on the responsibility, and glad you're back! Second, remember that PEP 6 is a work in progress, especially since you're breaking new ground. If PEP 6 actually hinders patch releases rather than helps them, then IMHO the PEP is probably broken. Is your problem with it the prohibitions? Or is it the procedure? Is the PEP too incomplete to be helpful? Please do suggest changes! If you want to co-own it so you can update it yourself, we can do that too. Patch releases should be easy(er). -Barry From fdrake@acm.org Wed Dec 5 16:36:19 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Wed, 5 Dec 2001 11:36:19 -0500 Subject: [Python-Dev] pychecker on 2.2 current In-Reply-To: <3C0C3806.B7325138@metaslash.com> References: <3C0C3806.B7325138@metaslash.com> Message-ID: <15374.19715.930571.37939@grendel.zope.com> Neal Norwitz writes: > mimetypes.py:405: No global (sys) found > > This is inside usage() which presumably should only be used internally. > sys is imported if __name__ == '__main__'. > So the problem can only be triggered if usage() is called by external source. > > Possible fixes are: > sys could be locally imported in usage() > sys could be imported at the module scope > usage could be made _usage() to denote it's private Neal, I'll propose another fix, but I don't know what pychecker would say. ;-) I'd like to move the definition of usage() into the if __name__ == "__main__" block, in which case the problem goes away since there wouldn't be a usage() unless sys has been imported. I'll check it in that way. ;-) -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From jack@oratrix.nl Wed Dec 5 15:59:39 2001 From: jack@oratrix.nl (Jack Jansen) Date: Wed, 05 Dec 2001 16:59:39 +0100 Subject: [Python-Dev] Mac OS X sys.platform value Message-ID: <20011205155939.8936A303183@snelboot.oratrix.nl> Folks, Apple "wisely" changed their uname version numbering scheme with a micro-release (10.1.1). Of course, nobody cares about this, except Python, which stuffs the version into sys.platform, from where it is used by a gazillion things. Anyway, where sys.platform used to be "darwin1" upto and including 10.1.0 it is now "darwin5" for 10.1.1 (and it will go up to "darwin6" for 10.2, etc). I am of a mind to take the "1" out of sys.platform, so that it becomes "darwin". Good idea or not? -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | ++++ see http://www.xs4all.nl/~tank/ ++++ From guido@python.org Wed Dec 5 16:07:19 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Dec 2001 11:07:19 -0500 Subject: [Python-Dev] Mac OS X sys.platform value In-Reply-To: Your message of "Wed, 05 Dec 2001 16:59:39 +0100." <20011205155939.8936A303183@snelboot.oratrix.nl> References: <20011205155939.8936A303183@snelboot.oratrix.nl> Message-ID: <200112051607.fB5G7Jw28901@odiug.zope.com> > Anyway, where sys.platform used to be "darwin1" upto and including 10.1.0 it > is now "darwin5" for 10.1.1 (and it will go up to "darwin6" for 10.2, etc). > > I am of a mind to take the "1" out of sys.platform, so that it becomes > "darwin". > > Good idea or not? Fine with me as long as it doesn't affect other platforms. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz@rahul.net Wed Dec 5 17:43:38 2001 From: aahz@rahul.net (Aahz Maruch) Date: Wed, 5 Dec 2001 09:43:38 -0800 (PST) Subject: [Python-Dev] python2.1.2 plans, plus the curse of PEP-006. In-Reply-To: <200112050705.fB575UN09520@mbuna.arbhome.com.au> from "Anthony Baxter" at Dec 05, 2001 06:05:29 PM Message-ID: <20011205174339.3EC2CE8C1@waltz.rahul.net> Anthony Baxter wrote: > > (I'm not sure if it's just me, but it's possible the reason no-one's > done two bugfix releases in a row is that PEP-006 is accursed, and > brings down the wrath of gods on anyone foolish enough to get involved. > I'm hoping that the last few weeks horror is over and done now...) Hmmmm.... I think the fact that three different people have stepped up to the plate to handle bugfix releases is a *feature*. In addition, three releases in six months does not a long-term trend make. As Barry said, if there's a problem with PEP6, I'd love to know about it; other than that, there haven't IMO been enough Python releases and patches to warrant more bugfix releases than have occurred or are on the plate. Good work so far! -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many. From Anthony Baxter Wed Dec 5 22:09:40 2001 From: Anthony Baxter (Anthony Baxter) Date: Thu, 06 Dec 2001 09:09:40 +1100 Subject: [Python-Dev] python2.1.2 plans, plus the curse of PEP-006. In-Reply-To: Message from barry@zope.com (Barry A. Warsaw) of "Wed, 05 Dec 2001 09:49:04 CDT." <15374.13280.512488.799375@anthem.wooz.org> Message-ID: <200112052209.fB5M9eZ14675@mbuna.arbhome.com.au> >>> Barry A. Warsaw wrote > Please do suggest changes! If you want to co-own it so you can update > it yourself, we can do that too. > Patch releases should be easy(er). I had a think about this, and I think the major issue is that there was quite a gap to backfill between 2.1.1 and me starting 2.1.2. There was a bit of backporting (thank you twouters!), but not that much - and not that much done as far as marking bugfixes in the changelogs. Given the rate of change in the python core, I'd say that it's more likely that people are going to want the ability to switch versions when it suits them, and thus keeping up the version before the current bleeding edge is appropriate. I'd say that something like the following would make life easier The branch for 2.N-maint should be maintained until 2.(N+1) gets a first patch release. That is, 2.1-maint gets bugfixes until 2.2.1 gets out the door. Alternately, waiting X months after 2.(N+1).0 would be another approach. In any case, marking bugfix candidates in CVS logs should be a standard thing to do. In terms of concrete things I'm looking for right now: . someone who's either a HP/UX victim^W user who's prepared to try and work out what bits need backporting for the threading fix, or someone who's got a HP/UX box and is prepared to let me ssh into the box and try and do it myself. . people to test the 2.1.2 release candidate when it's ready on a bunch of different platforms - I only have access to solaris, linux, and whatever's in the sf compile farm. Anthony From martin@v.loewis.de Wed Dec 5 20:00:18 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Wed, 5 Dec 2001 21:00:18 +0100 Subject: [Python-Dev] Mac OS X sys.platform value In-Reply-To: <20011205155939.8936A303183@snelboot.oratrix.nl> (message from Jack Jansen on Wed, 05 Dec 2001 16:59:39 +0100) References: <20011205155939.8936A303183@snelboot.oratrix.nl> Message-ID: <200112052000.fB5K0Ip01526@mira.informatik.hu-berlin.de> > Anyway, where sys.platform used to be "darwin1" upto and including 10.1.0 it > is now "darwin5" for 10.1.1 (and it will go up to "darwin6" for 10.2, etc). > > I am of a mind to take the "1" out of sys.platform, so that it becomes > "darwin". > > Good idea or not? Bad idea. Anybody comparing to "darwin1" will have to update the code. When they do so, they could just as well write sys.platform.startswith("darwin"). The same happens on Linux, which happens to be "linux2" since Linux kernel version 2.x is in use. That says little about the system, though, since glibc version or name of the distribution may be relevant. On OS X, it seems sys.platform could be used to distinguish minor differences. We should not take away that opportunity. Regards, Martin From aahz@rahul.net Wed Dec 5 22:43:00 2001 From: aahz@rahul.net (Aahz Maruch) Date: Wed, 5 Dec 2001 14:43:00 -0800 (PST) Subject: [Python-Dev] python2.1.2 plans, plus the curse of PEP-006. In-Reply-To: <200112052209.fB5M9eZ14675@mbuna.arbhome.com.au> from "Anthony Baxter" at Dec 06, 2001 09:09:40 AM Message-ID: <20011205224300.9EFFEE8C7@waltz.rahul.net> Anthony Baxter wrote: > > I'd say that something like the following would make life easier > > The branch for 2.N-maint should be maintained until 2.(N+1) gets > a first patch release. That is, 2.1-maint gets bugfixes until 2.2.1 > gets out the door. Alternately, waiting X months after 2.(N+1).0 > would be another approach. What does "should be maintained" mean? With CVS, IIUC, there's no reason not to keep branches around forever. If instead what you're talking about is policy for work to be assigned, I won't let that into PEP6; the entire point of PEP6 is that it only prescribes procedure for someone who is already working, it does not specify any work to be accomplished in the abstract. The original form of PEP6 was looser in this respect (in terms of specifying when patch releases would occur and so on), but I was convinced by Guido et al to stick strictly to "how" and not "when" or "what". At the very least, I think we should stick with the current base plan for one or two more release cycles before making any structured attempts to integrate bugfixes into the standard Python release cycle. > In any case, marking bugfix candidates in CVS logs should be a standard > thing to do. The problem lies in creating the social structure to make that happen. People who monitor CVS need to get into the habit of reminding each other to mark bugfix candidates. But that's beyond the scope of PEP6. -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many. From neal@metaslash.com Thu Dec 6 01:28:43 2001 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 05 Dec 2001 20:28:43 -0500 Subject: [Python-Dev] pychecker on 2.2 current References: <3C0C3806.B7325138@metaslash.com> <15374.19715.930571.37939@grendel.zope.com> Message-ID: <3C0EC9CB.48BBD0F0@metaslash.com> "Fred L. Drake, Jr." wrote: > > Neal Norwitz writes: > > mimetypes.py:405: No global (sys) found > > > > This is inside usage() which presumably should only be used internally. > > sys is imported if __name__ == '__main__'. > > So the problem can only be triggered if usage() is called > > by external source. > > > > Possible fixes are: > > sys could be locally imported in usage() > > sys could be imported at the module scope > > usage could be made _usage() to denote it's private > > I'll propose another fix, but I don't know what pychecker would > say. ;-) I'd like to move the definition of usage() into the if > __name__ == "__main__" block, in which case the problem goes away > since there wouldn't be a usage() unless sys has been imported. > I'll check it in that way. ;-) I like your fix better than my suggestions. :-) I thought pychecker would complain, but it's happy too. :-) Neal From guido@python.org Thu Dec 6 02:15:15 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Dec 2001 21:15:15 -0500 Subject: [Python-Dev] pychecker on 2.2 current In-Reply-To: Your message of "Wed, 05 Dec 2001 20:28:43 EST." <3C0EC9CB.48BBD0F0@metaslash.com> References: <3C0C3806.B7325138@metaslash.com> <15374.19715.930571.37939@grendel.zope.com> <3C0EC9CB.48BBD0F0@metaslash.com> Message-ID: <200112060215.VAA18132@cj20424-a.reston1.va.home.com> > I thought pychecker would complain, but it's happy too. :-) But isn't that because the new code is all at the module-global level, which pychecker doesn't "see"? Or have you fixed that yet? --Guido van Rossum (home page: http://www.python.org/~guido/) From neal@metaslash.com Thu Dec 6 02:26:15 2001 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 05 Dec 2001 21:26:15 -0500 Subject: [Python-Dev] pychecker on 2.2 current References: <3C0C3806.B7325138@metaslash.com> <15374.19715.930571.37939@grendel.zope.com> <3C0EC9CB.48BBD0F0@metaslash.com> <200112060215.VAA18132@cj20424-a.reston1.va.home.com> Message-ID: <3C0ED747.8E6E519C@metaslash.com> Guido van Rossum wrote: > > > I thought pychecker would complain, but it's happy too. :-) > > But isn't that because the new code is all at the module-global level, > which pychecker doesn't "see"? Or have you fixed that yet? pychecker does "see" all code at the module level, but it isn't perfect. If there is code like: import sys if sys.platform == 'win32': def x(a, b): pass else: def x(a): pass The following incorrect warning is produced: t2.py:5: Redefining attribute (x) original line (3) Because both versions of the function x() are seen. Neal From guido@python.org Thu Dec 6 02:35:25 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Dec 2001 21:35:25 -0500 Subject: [Python-Dev] pychecker on 2.2 current In-Reply-To: Your message of "Wed, 05 Dec 2001 21:26:15 EST." <3C0ED747.8E6E519C@metaslash.com> References: <3C0C3806.B7325138@metaslash.com> <15374.19715.930571.37939@grendel.zope.com> <3C0EC9CB.48BBD0F0@metaslash.com> <200112060215.VAA18132@cj20424-a.reston1.va.home.com> <3C0ED747.8E6E519C@metaslash.com> Message-ID: <200112060235.VAA18752@cj20424-a.reston1.va.home.com> > > But isn't that because the new code is all at the module-global level, > > which pychecker doesn't "see"? Or have you fixed that yet? > > pychecker does "see" all code at the module level, I stand corrected. I think this was the case for an older version? > but it isn't perfect. If there is code like: > > import sys > if sys.platform == 'win32': > def x(a, b): pass > else: > def x(a): pass > > The following incorrect warning is produced: > t2.py:5: Redefining attribute (x) original line (3) > > Because both versions of the function x() are seen. Hm, I would call that warning correct. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From neal@metaslash.com Thu Dec 6 02:50:24 2001 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 05 Dec 2001 21:50:24 -0500 Subject: [Python-Dev] pychecker on 2.2 current References: <3C0C3806.B7325138@metaslash.com> <15374.19715.930571.37939@grendel.zope.com> <3C0EC9CB.48BBD0F0@metaslash.com> <200112060215.VAA18132@cj20424-a.reston1.va.home.com> <3C0ED747.8E6E519C@metaslash.com> <200112060235.VAA18752@cj20424-a.reston1.va.home.com> Message-ID: <3C0EDCF0.94AB3B88@metaslash.com> Guido van Rossum wrote: > > > > But isn't that because the new code is all at the module-global level, > > > which pychecker doesn't "see"? Or have you fixed that yet? > > > > pychecker does "see" all code at the module level, > > I stand corrected. I think this was the case for an older version? Yes, you are correct it was the case a while ago. BTW, same is true for code at the class scope, ie, it used to be missed, but it is checked now. > > but it isn't perfect. If there is code like: > > > > import sys > > if sys.platform == 'win32': > > def x(a, b): pass > > else: > > def x(a): pass > > > > The following incorrect warning is produced: > > t2.py:5: Redefining attribute (x) original line (3) > > > > Because both versions of the function x() are seen. > > Hm, I would call that warning correct. :-) :-) From neal@metaslash.com Thu Dec 6 02:50:38 2001 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 05 Dec 2001 21:50:38 -0500 Subject: [Python-Dev] memory leaks in 2.2 Message-ID: <3C0EDCFE.D01E6EA3@metaslash.com> Here is a list of all the outstanding memory leaks from the regression tests. All of these leaks have a bug report on SF. test_descr: 117 bytes (0.0407%) test_email: 30 bytes (0.0149%) test_re: 30 bytes (0.0149%) test_richcmp: 16 bytes (0.0106%) test_scope: 39 bytes (0.031%) test_sre: 188 bytes (0.137%) test_symtable: 12 bytes (0.00991%) test_unicode: 66 bytes (0.0224%) syntax error in interpreter: 38 bytes (0.05%) From Anthony Baxter Thu Dec 6 02:59:11 2001 From: Anthony Baxter (Anthony Baxter) Date: Thu, 06 Dec 2001 13:59:11 +1100 Subject: [Python-Dev] python2.1.2 plans, plus the curse of PEP-006. In-Reply-To: Message from aahz@rahul.net (Aahz Maruch) of "Wed, 05 Dec 2001 14:43:00 -0800." <20011205224300.9EFFEE8C7@waltz.rahul.net> Message-ID: <200112060259.fB62xBi16480@mbuna.arbhome.com.au> >>> Aahz Maruch wrote > What does "should be maintained" mean? With CVS, IIUC, there's no > reason not to keep branches around forever. If instead what you're I mean "should continually have backported patches applied". > talking about is policy for work to be assigned, I won't let that into > PEP6; the entire point of PEP6 is that it only prescribes procedure for > someone who is already working, it does not specify any work to be > accomplished in the abstract. Oh, sure - I'm not talking about adding it to PEP006 - it's not appropriate for it. More talking about the abstract here. > The problem lies in creating the social structure to make that happen. > People who monitor CVS need to get into the habit of reminding each > other to mark bugfix candidates. But that's beyond the scope of PEP6. *nod* absolutely. From tim.one@home.com Thu Dec 6 04:14:00 2001 From: tim.one@home.com (Tim Peters) Date: Wed, 5 Dec 2001 23:14:00 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <3C0EDCFE.D01E6EA3@metaslash.com> Message-ID: [Neal Norwitz] > Here is a list of all the outstanding memory leaks from the > regression tests. > All of these leaks have a bug report on SF. > > test_descr: 117 bytes (0.0407%) > test_email: 30 bytes (0.0149%) > test_re: 30 bytes (0.0149%) > test_richcmp: 16 bytes (0.0106%) > test_scope: 39 bytes (0.031%) > test_sre: 188 bytes (0.137%) > test_symtable: 12 bytes (0.00991%) > test_unicode: 66 bytes (0.0224%) > > syntax error in interpreter: 38 bytes (0.05%) Can anyone lend a hand here? Take a test, stick it in an infinite loop, watch your process size blow off the roof, then whittle the test down until the trigger becomes clear. We found a very serious 2.2 leak today doing this to test_binop.py, and not even Guido (let alone God ...) knows how many more may be lurking. It's not a Good Sign that test_binop has vanished from the list now but everything else is still there. have-some-fun-plug-a-leak-ly y'rs - tim From fdrake@acm.org Thu Dec 6 05:17:58 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Thu, 6 Dec 2001 00:17:58 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: References: <3C0EDCFE.D01E6EA3@metaslash.com> Message-ID: <15374.65414.640831.895781@grendel.zope.com> Tim Peters writes: > more may be lurking. It's not a Good Sign that test_binop has vanished from > the list now but everything else is still there. Err, test_parser is no longer on the list! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tim.one@home.com Thu Dec 6 04:36:51 2001 From: tim.one@home.com (Tim Peters) Date: Wed, 5 Dec 2001 23:36:51 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <15374.65414.640831.895781@grendel.zope.com> Message-ID: [Tim] > more may be lurking. It's not a Good Sign that test_binop has > vanished from the list now but everything else is still there. [Fred L. Drake, Jr.] > Err, test_parser is no longer on the list! Which is bad news: it means that test_binop and test_parser had leaks for reasons independent of each other, and of all the others on the list: we're not making swift progress on these. You only looked at test_parser because Guido assigned it to you. I only looked at test_binop because I was coughing too hard to get any difficult work done . From fdrake@acm.org Thu Dec 6 06:58:38 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Thu, 6 Dec 2001 01:58:38 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: References: <15374.65414.640831.895781@grendel.zope.com> Message-ID: <15375.5918.776405.747630@grendel.zope.com> Tim Peters writes: > Which is bad news: it means that test_binop and test_parser had leaks for > reasons independent of each other, and of all the others on the list: we're > not making swift progress on these. You only looked at test_parser because > Guido assigned it to you. I only looked at test_binop because I was > coughing too hard to get any difficult work done . Aside from these specific memory-leak reports, we've generally fallen down on bug triage. I'm not sure what we can do about this in the short term, but I really think we're not doing as good a job at this as we should be. I wonder if we could take a lesson from the Mozilla community and organize Bug Days (group hack days where the goal is to resolve bugs) or have a "BugAThon" where we call for smaller/simpler test cases that exercise reported bugs. I suspect a page about how to narrow down a bug to a smaller test case would be very helpful, especially if it also discusses how to use the various available tools for things like pinpointing leaks, etc. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tim.one@home.com Thu Dec 6 07:00:28 2001 From: tim.one@home.com (Tim Peters) Date: Thu, 6 Dec 2001 02:00:28 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <15375.5918.776405.747630@grendel.zope.com> Message-ID: [Fred L. Drake, Jr.] > Aside from these specific memory-leak reports, we've generally > fallen down on bug triage. Look on the bright side: we're doing a much better job on bugs than on patch review, and our patch review work in turn is stellar compared to what we've accomplished in the PSF . > I'm not sure what we can do about this in the short term, but I really > think we're not doing as good a job at this as we should be. I think it's just one of many areas where we "should be" doing a better job. > I wonder if we could take a lesson from the Mozilla community and > organize Bug Days (group hack days where the goal is to resolve bugs) > or have a "BugAThon" where we call for smaller/simpler test cases that > exercise reported bugs. I suspect a page about how to narrow down a > bug to a smaller test case would be very helpful, especially if it > also discusses how to use the various available tools for things like > pinpointing leaks, etc. Frankly, I don't think it will help. If you look at the bug statistics, we've made steady and dramatic improvement in bug turnaround time over the last six months, and we can't sustain even that improvement. Fact is we have very few people fixing bugs or reviewing patches. It's not like the majority of Python developers work on this stuff in spurts, it's that most don't work on it at all, and almost nobody works on it routinely. We couldn't even meet our modest internal (admittedly imposed by me ) "close one bug or patch per day per person" goal last week (you'll never see that idea again either ...). Maybe we should resume offering a free copy of the next Python release to contributors working beyond the bounds of sanity? Na -- even Thomas eventually figured out he could get a free copy anyway -- let's just pray he doesn't tell Martin . better-at-eating-kung-pao-chicken-than-motivating-the-overburdened- to-spread-themselves-thinner-than-gas-ly y'rs - tim From DavidA@ActiveState.com Thu Dec 6 07:12:45 2001 From: DavidA@ActiveState.com (David Ascher) Date: Wed, 05 Dec 2001 23:12:45 -0800 Subject: [Python-Dev] memory leaks in 2.2 References: <15374.65414.640831.895781@grendel.zope.com> <15375.5918.776405.747630@grendel.zope.com> Message-ID: <3C0F1A6D.417B603C@ActiveState.com> "Fred L. Drake, Jr." wrote: > Aside from these specific memory-leak reports, we've generally > fallen down on bug triage. I'm not sure what we can do about this in > the short term, but I really think we're not doing as good a job at > this as we should be. It's hard. Bugzilla has some good features there w.r.t. "unverified" vs. "new" vs. "accepted" and "fixed" vs "resolved", QA contacts, keyword handling for milestone/release targets, etc. I'm not sure how the SF bugtracker compares in practice but from a distance it seems a little weak there. [BTW -- I know this isn't the right channel, but we (the Komodo team) have found a couple of bugs in popen recently which y'all may not have seen yet and you might want to tackle for 2.2 -- we'll get them in SF in the next few days hopefully, but this is better than nothing: - popen on Windows XP seems problematic (needs a repro test there -- not analyzed yet) - popen on Posix has a nasty failure mode if the cmd argument is Unicode (see the first line in Popen3._run_child for a hint =)] > I wonder if we could take a lesson from the Mozilla community and > organize Bug Days (group hack days where the goal is to resolve bugs) > or have a "BugAThon" where we call for smaller/simpler test cases that > exercise reported bugs. I suspect a page about how to narrow down a > bug to a smaller test case would be very helpful, especially if it > also discusses how to use the various available tools for things like > pinpointing leaks, etc. One thing the Mozilla community has going for it is really dedicated QA folks. One in particular, Asa Dotzler, does an amazing job at bug triage, categorization, etc. He's now a Netscape employee, but used to do this for fun. It would be a good way for someone who'se 'on the periphery' to get more deeply involved and learn a lot. In general, that kind of QA job can be a gateway to fame and fortune much like documentation writing can be, and you really get to learn the product/technology. I don't have first-hand experience of the success of bug days, but they're doing more of them, so they must get some value out of them. --david guessing-from-the-"it hurts! then don't do that!"-corrolary ascher From martin@v.loewis.de Thu Dec 6 07:27:33 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Thu, 6 Dec 2001 08:27:33 +0100 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: References: Message-ID: <200112060727.fB67RXZ08417@mira.informatik.hu-berlin.de> > Maybe we should resume offering a free copy of the next Python release to > contributors working beyond the bounds of sanity? Na -- even Thomas > eventually figured out he could get a free copy anyway -- let's just pray he > doesn't tell Martin . :-) I personally don't see it as a problem that bug reports are piling up. The GCC GNATS has 1341 "open" bug reports (meaning either "open" or "analyzed") at the moment. bugzilla.mozilla.org reports 19611 bugs in the "open" categories. Microsoft reportedly has over 60000 open reports for Windows 2000. Neither of these groups despair :-) As for the things we should change: I believe it is pointless to sit on a report for a long time without doing anything about this. If nothing happens for a year, the person that was assigned this report likely won't do anything about it for the next months, either. Unassigning things you know you won't do is particularly important for patches, but to a degree also important for bug reports. It signals that somebody else may need to step forward, or that some deity should just decide on the fate of this report. Regards, Martin From DavidA@ActiveState.com Thu Dec 6 07:39:08 2001 From: DavidA@ActiveState.com (David Ascher) Date: Wed, 05 Dec 2001 23:39:08 -0800 Subject: [Python-Dev] memory leaks in 2.2 References: Message-ID: <3C0F209C.9FAE0599@ActiveState.com> Tim Peters wrote: > > [Fred L. Drake, Jr.] > > Aside from these specific memory-leak reports, we've generally > > fallen down on bug triage. > > Look on the bright side: we're doing a much better job on bugs than on > patch review, and our patch review work in turn is stellar compared to what > we've accomplished in the PSF . I'll pipe in with our experience with Mozilla & Komodo, which have related but different processes. Mozilla has a _very strict_ review process. Up to two reviews and one super-review for each checkin (http://www.mozilla.org/hacking/reviewers.html). 99% of all communication is done through comments & patches attached to bugs. It works for them, in that the quality of the code checkins tends to be high, but it's a fairly imposing process (which is limiting the number of contributors, I expect -- not because the process is _that_ hard to follow, but it's hard to discover and learn and adopt, especially as part of a side project). In the Komodo team, the number and type of reviews we require depends on the stage of the product (how close we are to release). We used to have periods when no review was needed, but I think we're going to keep the minimal one-review requirement on indefinitely. At critical times we've even had a "change review board" process in place. Having tools that generate good patches with 'proposed changelogs' and apply them safely, good integration between the source code control system and the bug tracking system are all good things which have made our code quality go up as well. Bugzilla is a good patch manager as well. It takes time to learn/teach how to be a good reviewer -- it's not the same kind of skill as writing a patch or logging a good bug. Review processes also affect 'superficial development speed' -- the patch author has to sit on a patch until the process is complete, and that can be frustrating. Also, to the extent that patches uncover other bugs, it can slow down the bug discovery process. Of course we wouldn't do it if we didn't feel it increased the long-term effective rate of quality increase. Note that in both of the above cases, _everyone_ must go through the review process, including the top dogs, for any change to the code base, build system, etc. (the documentation and test suite subtree are excluded usually). Note also that I don't mean to imply that the above process is the right one for Python. I'm just speaking on the processes I know something about =). --david From martin@v.loewis.de Thu Dec 6 07:40:22 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Thu, 6 Dec 2001 08:40:22 +0100 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <3C0F1A6D.417B603C@ActiveState.com> (message from David Ascher on Wed, 05 Dec 2001 23:12:45 -0800) References: <15374.65414.640831.895781@grendel.zope.com> <15375.5918.776405.747630@grendel.zope.com> <3C0F1A6D.417B603C@ActiveState.com> Message-ID: <200112060740.fB67eMV08466@mira.informatik.hu-berlin.de> > It's hard. Bugzilla has some good features there w.r.t. "unverified" > vs. "new" vs. "accepted" and "fixed" vs "resolved", QA contacts, keyword > handling for milestone/release targets, etc. I'm not sure how the SF > bugtracker compares in practice but from a distance it seems a little > weak there. SF recommends to use the Group for "unsupported", "unverified". Unfortunately, the filtering capabilities leave a lot of wishes. > - popen on Posix has a nasty failure mode if the cmd argument is > Unicode (see the first line in Popen3._run_child for a hint =)] You mean, def _run_child(self, cmd): if isinstance(cmd, types.StringTypes): cmd = ['/bin/sh', '-c', cmd] :-? Fixed since 2001/12/02. > It would be a good way for someone who'se 'on the periphery' to get > more deeply involved and learn a lot. This is free software. Most contributors work on it for fun, or because they need it for their job. Posting "volunteers needed" may be a good idea, but then, perhaps not so shortly before 2.2. Regards, Martin From DavidA@ActiveState.com Thu Dec 6 07:49:39 2001 From: DavidA@ActiveState.com (David Ascher) Date: Wed, 05 Dec 2001 23:49:39 -0800 Subject: [Python-Dev] memory leaks in 2.2 References: <15374.65414.640831.895781@grendel.zope.com> <15375.5918.776405.747630@grendel.zope.com> <3C0F1A6D.417B603C@ActiveState.com> <200112060740.fB67eMV08466@mira.informatik.hu-berlin.de> Message-ID: <3C0F2313.C9B7824F@ActiveState.com> > if isinstance(cmd, types.StringTypes): > cmd = ['/bin/sh', '-c', cmd] > > :-? Fixed since 2001/12/02. Ha! Good. One more resolved bug then =). > > It would be a good way for someone who'se 'on the periphery' to get > > more deeply involved and learn a lot. > > This is free software. Most contributors work on it for fun, or > because they need it for their job. Sure. Are you disagreeing with something I said? --david From anthony@interlink.com.au Thu Dec 6 07:45:04 2001 From: anthony@interlink.com.au (Anthony Baxter) Date: Thu, 06 Dec 2001 18:45:04 +1100 Subject: [Python-Dev] on the subject of SF... Message-ID: <200112060745.fB67j4e18189@mbuna.arbhome.com.au> can someone with the appropriate level of magickness add a 'Python 2.1.2' group to the bug tracker and patch tracker? I've been meaning to get this done for a while... ta. Anthony From martin@v.loewis.de Thu Dec 6 07:47:31 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Thu, 6 Dec 2001 08:47:31 +0100 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <3C0F209C.9FAE0599@ActiveState.com> (message from David Ascher on Wed, 05 Dec 2001 23:39:08 -0800) References: <3C0F209C.9FAE0599@ActiveState.com> Message-ID: <200112060747.fB67lV108506@mira.informatik.hu-berlin.de> > Note that in both of the above cases, _everyone_ must go through the > review process, including the top dogs, for any change to the code base, > build system, etc. (the documentation and test suite subtree are > excluded usually). Note also that I don't mean to imply that the above > process is the right one for Python. I'm just speaking on the processes > I know something about =). To report on the processes of another project; GCC organizes people in "blanket write priviledge", "maintainers" (CPU port, OS, misc), and "write after approval". The blanket priviledge allows people to commit contributions they have reviewed from contributors without any write access; those are used sparingly. Maintainers can change "their" files any time they like, except after a freeze. Everybody else must wait for a patch review. Regards, Martin From barry@zope.com Thu Dec 6 07:49:02 2001 From: barry@zope.com (Barry A. Warsaw) Date: Thu, 6 Dec 2001 02:49:02 -0500 Subject: [Python-Dev] on the subject of SF... References: <200112060745.fB67j4e18189@mbuna.arbhome.com.au> Message-ID: <15375.8942.722195.533597@anthem.wooz.org> >>>>> "AB" == Anthony Baxter writes: AB> can someone with the appropriate level of magickness add a AB> 'Python 2.1.2' group to the bug tracker and patch tracker? AB> I've been meaning to get this done for a while... ta. Anthony Done! -Barry From martin@v.loewis.de Thu Dec 6 07:52:38 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Thu, 6 Dec 2001 08:52:38 +0100 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <3C0F2313.C9B7824F@ActiveState.com> (message from David Ascher on Wed, 05 Dec 2001 23:49:39 -0800) References: <15374.65414.640831.895781@grendel.zope.com> <15375.5918.776405.747630@grendel.zope.com> <3C0F1A6D.417B603C@ActiveState.com> <200112060740.fB67eMV08466@mira.informatik.hu-berlin.de> <3C0F2313.C9B7824F@ActiveState.com> Message-ID: <200112060752.fB67qcw08535@mira.informatik.hu-berlin.de> > > > It would be a good way for someone who'se 'on the periphery' to get > > > more deeply involved and learn a lot. > > > > This is free software. Most contributors work on it for fun, or > > because they need it for their job. > > Sure. Are you disagreeing with something I said? No. I just think this won't help much. Just because it would be good for somebody doesn't mean that this somebody will actually invest time into the project. People normally get involved by other means than starting as a bug/patch reviewer. Typically, they get involved because they need something fixed that apparently nobody else has fixed for them while they were waiting for it (atleast that's how I got involved in all free software projects I was working on). Regards, Martin From mal@lemburg.com Thu Dec 6 08:40:00 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 06 Dec 2001 09:40:00 +0100 Subject: [Python-Dev] on the subject of SF... References: <200112060745.fB67j4e18189@mbuna.arbhome.com.au> <15375.8942.722195.533597@anthem.wooz.org> Message-ID: <3C0F2EE0.BE1D2980@lemburg.com> "Barry A. Warsaw" wrote: > > >>>>> "AB" == Anthony Baxter writes: > > AB> can someone with the appropriate level of magickness add a > AB> 'Python 2.1.2' group to the bug tracker and patch tracker? > AB> I've been meaning to get this done for a while... ta. Anthony > > Done! And please also a Group "Python 2.3" so that feature requests for that version can be assigned to it ?! -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Thu Dec 6 09:08:59 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 06 Dec 2001 10:08:59 +0100 Subject: [Python-Dev] Mac OS X sys.platform value References: <20011205155939.8936A303183@snelboot.oratrix.nl> <200112052000.fB5K0Ip01526@mira.informatik.hu-berlin.de> Message-ID: <3C0F35AB.2CF9D067@lemburg.com> "Martin v. Loewis" wrote: > > > Anyway, where sys.platform used to be "darwin1" upto and including 10.1.0 it > > is now "darwin5" for 10.1.1 (and it will go up to "darwin6" for 10.2, etc). > > > > I am of a mind to take the "1" out of sys.platform, so that it becomes > > "darwin". > > > > Good idea or not? > > Bad idea. Anybody comparing to "darwin1" will have to update the > code. When they do so, they could just as well write > sys.platform.startswith("darwin"). > > The same happens on Linux, which happens to be "linux2" since Linux > kernel version 2.x is in use. That says little about the system, > though, since glibc version or name of the distribution may be > relevant. Exactly and that's why I think that version numbers in sys.platform are plain wrong. There are better tools for finding out details about the system, e.g. platform.py which you can download from my Python pages. > On OS X, it seems sys.platform could be used to distinguish minor > differences. We should not take away that opportunity. As you already noted, the information given in the current version number is too inaccurate to be of any real value. Why not add platform.py to the std lib in Python 2.3 and remove all version info from sys.platform ?! -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From ping@lfw.org Thu Dec 6 09:57:41 2001 From: ping@lfw.org (Ka-Ping Yee) Date: Thu, 6 Dec 2001 01:57:41 -0800 (PST) Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <3C0EDCFE.D01E6EA3@metaslash.com> Message-ID: On Wed, 5 Dec 2001, Neal Norwitz wrote: > Here is a list of all the outstanding memory leaks from the regression tests. > All of these leaks have a bug report on SF. > > test_descr: 117 bytes (0.0407%) Okay, i have tested this case and narrowed it down to: while 1: class C: __metaclass__ = type Try it on a Python interpreter near you. With mine, it grows without bound. As far as i can tell, of all the calls in test_descr.test_main, only metaclass() leaks memory. -- ?!ng From mal@lemburg.com Thu Dec 6 12:12:46 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 06 Dec 2001 13:12:46 +0100 Subject: [Python-Dev] Comparing sequences ?! Message-ID: <3C0F60BE.5616F544@lemburg.com> I just tried (1,2,3) == [1,2,3] and found that the compare returns false. Is that intended ? Background: 1.0 == 1 == 1L also works, so it seems natural that comparing sequences of different types should work in the same way. I would even expect a two iterators to compare equal if they always return the same values (this could cause an endless loop for some endless iterators ;-). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From fdrake@acm.org Thu Dec 6 13:25:15 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Thu, 6 Dec 2001 08:25:15 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: References: <15375.5918.776405.747630@grendel.zope.com> Message-ID: <15375.29115.974845.732239@grendel.zope.com> Tim Peters writes: > better-at-eating-kung-pao-chicken-than-motivating-the-overburdened- > to-spread-themselves-thinner-than-gas-ly y'rs - tim Ah, but gas can be spread infinately thin -- something that can't be said of most developers! ;-) -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From guido@python.org Thu Dec 6 13:30:58 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Dec 2001 08:30:58 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: Your message of "Wed, 05 Dec 2001 23:14:00 EST." References: Message-ID: <200112061330.IAA21262@cj20424-a.reston1.va.home.com> > Can anyone lend a hand here? Take a test, stick it in an infinite loop, > watch your process size blow off the roof, then whittle the test down until > the trigger becomes clear. We found a very serious 2.2 leak today doing > this to test_binop.py, and not even Guido (let alone God ...) knows how many > more may be lurking. It's not a Good Sign that test_binop has vanished from > the list now but everything else is still there. I'll take test_descr.py right now. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Dec 6 13:38:54 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Dec 2001 08:38:54 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: Your message of "Wed, 05 Dec 2001 23:36:51 EST." References: Message-ID: <200112061338.IAA21671@cj20424-a.reston1.va.home.com> I can't find a leak in test_descr.py when I simply call test_main() in an infinite loop. (I can when I use reload() in the loop, but that's an import issue I believe.) Neil, did you test this before or after I fixed the __slots__ leak? --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh@python.net Thu Dec 6 13:42:23 2001 From: mwh@python.net (Michael Hudson) Date: 06 Dec 2001 08:42:23 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: "Martin v. Loewis"'s message of "Thu, 6 Dec 2001 08:40:22 +0100" References: <15374.65414.640831.895781@grendel.zope.com> <15375.5918.776405.747630@grendel.zope.com> <3C0F1A6D.417B603C@ActiveState.com> <200112060740.fB67eMV08466@mira.informatik.hu-berlin.de> Message-ID: <2my9kgcvnk.fsf@starship.python.net> "Martin v. Loewis" writes: > > It's hard. Bugzilla has some good features there w.r.t. "unverified" > > vs. "new" vs. "accepted" and "fixed" vs "resolved", QA contacts, keyword > > handling for milestone/release targets, etc. I'm not sure how the SF > > bugtracker compares in practice but from a distance it seems a little > > weak there. > > SF recommends to use the Group for "unsupported", > "unverified". Unfortunately, the filtering capabilities leave a lot of > wishes. Is roundup usable yet? I played with it a bit yesterday, but I'm not really sure what's needed. It would have the advantage that if it didn't do something we needed, we could probably club it into doing it easily enough. Cheers, M. -- I never disputed the Perl hacking skill of the Slashdot creators. My objections are to the editors' taste, the site's ugly visual design, and the Slashdot community's raging stupidity. -- http://www.cs.washington.edu/homes/klee/misc/slashdot.html#faq From guido@python.org Thu Dec 6 13:48:25 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Dec 2001 08:48:25 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: Your message of "Thu, 06 Dec 2001 08:27:33 +0100." <200112060727.fB67RXZ08417@mira.informatik.hu-berlin.de> References: <200112060727.fB67RXZ08417@mira.informatik.hu-berlin.de> Message-ID: <200112061348.IAA22096@cj20424-a.reston1.va.home.com> > As for the things we should change: I believe it is pointless to sit > on a report for a long time without doing anything about this. If > nothing happens for a year, the person that was assigned this report > likely won't do anything about it for the next months, either. > Unassigning things you know you won't do is particularly important for > patches, but to a degree also important for bug reports. It signals > that somebody else may need to step forward, or that some deity should > just decide on the fate of this report. Yup. We've missed important bugs in the past because they happened to be assigned to someone who didn't care. (I noticed that Jack recommended some bug reporter to assign bugs to him rather than let them sit anonymously. That may work in his case, but in general it's the wrong thing to do.) --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik@pythonware.com Thu Dec 6 13:50:51 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 6 Dec 2001 14:50:51 +0100 Subject: [Python-Dev] memory leaks in 2.2 References: <15374.65414.640831.895781@grendel.zope.com> <15375.5918.776405.747630@grendel.zope.com> <3C0F1A6D.417B603C@ActiveState.com> <200112060740.fB67eMV08466@mira.informatik.hu-berlin.de> <2my9kgcvnk.fsf@starship.python.net> Message-ID: <01ad01c17e5d$051fe840$0900a8c0@spiff> Michael Hudson wrote: > Is roundup usable yet? I played with it a bit yesterday, but I'm not > really sure what's needed. It would have the advantage that if it > didn't do something we needed, we could probably club it into doing it > easily enough. the drawback with "easily enough" is that you cannot resist modifying it -- so instead of learning how to use the tool you have in the best way, you end up hacking on the tool instead of doing more important work. (fwiw, we're using a very slightly modified version of Ping's original one-day hack. took a couple of months to find the right "use patterns", but now that we know how to use it in a way that fits our development process, I doubt you can find a "more usable" tool...) From guido@python.org Thu Dec 6 13:59:53 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Dec 2001 08:59:53 -0500 Subject: [Python-Dev] Comparing sequences ?! In-Reply-To: Your message of "Thu, 06 Dec 2001 13:12:46 +0100." <3C0F60BE.5616F544@lemburg.com> References: <3C0F60BE.5616F544@lemburg.com> Message-ID: <200112061359.IAA22524@cj20424-a.reston1.va.home.com> > I just tried (1,2,3) == [1,2,3] and found that the compare > returns false. Is that intended ? Yes. --Guido van Rossum (home page: http://www.python.org/~guido/) From neal@metaslash.com Thu Dec 6 14:01:16 2001 From: neal@metaslash.com (Neal Norwitz) Date: Thu, 06 Dec 2001 09:01:16 -0500 Subject: [Python-Dev] memory leaks in 2.2 References: <200112061338.IAA21671@cj20424-a.reston1.va.home.com> Message-ID: <3C0F7A2C.1B416FF0@metaslash.com> Guido van Rossum wrote: > > I can't find a leak in test_descr.py when I simply call test_main() in > an infinite loop. (I can when I use reload() in the loop, but that's > an import issue I believe.) > > Neil, did you test this before or after I fixed the __slots__ leak? I believe I tested after your initial checkin, but before the most recent. I just updated and re-ran. 117 bytes were still leaked. I'll try to narrow down the test to get a smaller test case. Neal From guido@python.org Thu Dec 6 14:01:06 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Dec 2001 09:01:06 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: Your message of "Thu, 06 Dec 2001 08:38:54 EST." <200112061338.IAA21671@cj20424-a.reston1.va.home.com> References: <200112061338.IAA21671@cj20424-a.reston1.va.home.com> Message-ID: <200112061401.JAA22566@cj20424-a.reston1.va.home.com> > I can't find a leak in test_descr.py when I simply call test_main() in > an infinite loop. (I can when I use reload() in the loop, but that's > an import issue I believe.) > > Neil, did you test this before or after I fixed the __slots__ leak? Never mind, with Ping's hint I see a leak. I'll fixit momentarily. --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy@zope.com Thu Dec 6 14:15:34 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Thu, 6 Dec 2001 09:15:34 -0500 (EST) Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <3C0EDCFE.D01E6EA3@metaslash.com> References: <3C0EDCFE.D01E6EA3@metaslash.com> Message-ID: <15375.32134.907220.808367@slothrop.digicool.com> >>>>> "NN" == Neal Norwitz writes: NN> Here is a list of all the outstanding memory leaks from the NN> regression tests. All of these leaks have a bug report on SF. I don't see an SF bug report for test_scope. If you have any data from Purify, that would be helpful. It certainly looks like it's leaking. Jeremy From neal@metaslash.com Thu Dec 6 14:18:47 2001 From: neal@metaslash.com (Neal Norwitz) Date: Thu, 06 Dec 2001 09:18:47 -0500 Subject: [Python-Dev] memory leaks in 2.2 References: <3C0EDCFE.D01E6EA3@metaslash.com> <15375.32134.907220.808367@slothrop.digicool.com> Message-ID: <3C0F7E47.BF969740@metaslash.com> Jeremy Hylton wrote: > > >>>>> "NN" == Neal Norwitz writes: > > NN> Here is a list of all the outstanding memory leaks from the > NN> regression tests. All of these leaks have a bug report on SF. > > I don't see an SF bug report for test_scope. If you have any data > from Purify, that would be helpful. It certainly looks like it's > leaking. https://sourceforge.net/tracker/index.php?func=detail&aid=485152&group_id=5470&atid=105470 From neal@metaslash.com Thu Dec 6 14:24:19 2001 From: neal@metaslash.com (Neal Norwitz) Date: Thu, 06 Dec 2001 09:24:19 -0500 Subject: [Python-Dev] memory leaks in 2.2 References: <200112061338.IAA21671@cj20424-a.reston1.va.home.com> <200112061401.JAA22566@cj20424-a.reston1.va.home.com> Message-ID: <3C0F7F93.5555623B@metaslash.com> Guido van Rossum wrote: > > > I can't find a leak in test_descr.py when I simply call test_main() in > > an infinite loop. (I can when I use reload() in the loop, but that's > > an import issue I believe.) > > > > Neil, did you test this before or after I fixed the __slots__ leak? > > Never mind, with Ping's hint I see a leak. I'll fixit momentarily. In test_descr, buffer_inherit() leaks 35 bytes which may indicate the problem. Neal From mal@lemburg.com Thu Dec 6 14:26:28 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 06 Dec 2001 15:26:28 +0100 Subject: [Python-Dev] Comparing sequences ?! References: <3C0F60BE.5616F544@lemburg.com> <200112061359.IAA22524@cj20424-a.reston1.va.home.com> Message-ID: <3C0F8014.FA2ECF58@lemburg.com> Guido van Rossum wrote: > > > I just tried (1,2,3) == [1,2,3] and found that the compare > > returns false. Is that intended ? > > Yes. I presume that means "...and we're not going to change it." ;-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From barry@zope.com Thu Dec 6 14:36:11 2001 From: barry@zope.com (Barry A. Warsaw) Date: Thu, 6 Dec 2001 09:36:11 -0500 Subject: [Python-Dev] on the subject of SF... References: <200112060745.fB67j4e18189@mbuna.arbhome.com.au> <15375.8942.722195.533597@anthem.wooz.org> <3C0F2EE0.BE1D2980@lemburg.com> Message-ID: <15375.33371.540885.525907@anthem.wooz.org> >>>>> "M" == M writes: M> And please also a Group "Python 2.3" so that feature requests M> for that version can be assigned to it ?! So far we have no groups defined for the feature request tracker. I wonder if that's intentional. -Barry From guido@python.org Thu Dec 6 14:47:18 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Dec 2001 09:47:18 -0500 Subject: [Python-Dev] Comparing sequences ?! In-Reply-To: Your message of "Thu, 06 Dec 2001 15:26:28 +0100." <3C0F8014.FA2ECF58@lemburg.com> References: <3C0F60BE.5616F544@lemburg.com> <200112061359.IAA22524@cj20424-a.reston1.va.home.com> <3C0F8014.FA2ECF58@lemburg.com> Message-ID: <200112061447.JAA23518@cj20424-a.reston1.va.home.com> > > > I just tried (1,2,3) == [1,2,3] and found that the compare > > > returns false. Is that intended ? > > > > Yes. > > I presume that means "...and we're not going to change it." ;-) Not any time soon. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Dec 6 21:52:16 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Dec 2001 16:52:16 -0500 Subject: [Python-Dev] Need help with distutils Message-ID: <200112062152.fB6LqGT11265@odiug.zope.com> Andrew Kuchling just unassigned from himself a whole slew of distutils-related bug reports. It appears that neither Andrew nor Greg Ward (the original maintainer) has the time to maintain this code any more. No-one at PythonLabs understands it or has the time to learn about it. So it's effectively orphaned. Does anybody want to volunteer? --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Thu Dec 6 22:39:05 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 06 Dec 2001 23:39:05 +0100 Subject: [Python-Dev] Need help with distutils References: <200112062152.fB6LqGT11265@odiug.zope.com> Message-ID: <3C0FF389.9C5FF5A5@lemburg.com> Guido van Rossum wrote: > > Andrew Kuchling just unassigned from himself a whole slew of > distutils-related bug reports. It appears that neither Andrew nor > Greg Ward (the original maintainer) has the time to maintain this code > any more. No-one at PythonLabs understands it or has the time to > learn about it. So it's effectively orphaned. > > Does anybody want to volunteer? The distutils list can take over, I guess. I will certainly want to see distutils have a bright future and so do many other users like e.g. Thomas Heller, Jack Jansen, etc. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From DavidA@ActiveState.com Fri Dec 7 00:56:46 2001 From: DavidA@ActiveState.com (David Ascher) Date: Thu, 06 Dec 2001 16:56:46 -0800 Subject: [Python-Dev] memory leaks in 2.2 References: <15374.65414.640831.895781@grendel.zope.com> <15375.5918.776405.747630@grendel.zope.com> <3C0F1A6D.417B603C@ActiveState.com> <200112060740.fB67eMV08466@mira.informatik.hu-berlin.de> <2my9kgcvnk.fsf@starship.python.net> <01ad01c17e5d$051fe840$0900a8c0@spiff> Message-ID: <3C1013CE.B1AE8D82@ActiveState.com> Fredrik Lundh wrote: > > the drawback with "easily enough" is that you cannot resist > modifying it -- so instead of learning how to use the tool you > have in the best way, you end up hacking on the tool instead > of doing more important work. Indeed =) > (fwiw, we're using a very slightly modified version of Ping's > original one-day hack. took a couple of months to find the > right "use patterns", but now that we know how to use it in > a way that fits our development process, I doubt you can > find a "more usable" tool...) FWIW, we used to use a fairly seriously modified version of Ping's original one-day hack, and gave it up. It was generating too much email and the 'nosylist' concept wasn't working for us. The more explicit list management that bugzilla uses seems to work better for us. YMMV. --david From Keila - Curitiba" Ol=E1! Veja meu site pessoal. Basta clicar no endere=E7o abaixo. GARANTO SER SUI-GENERIS - CLIQUE ABAIXO: http://www.pastorinha.atfreeweb.com Mais de 162.000 internautas visitaram a PG., existe 6 =C1lbuns: Se voc=EA quiser, por favor, indique minha Home Page, a outros Internautas. Mais detalhes, se comunique, passe um e-mail, que responderei brevemente. Dentro da Home Page, ao lado das fotos, voc=EA poder=E1 saber muito mais sobre mim! Obrigada. e-mail: arosadesaron@zipmail.com.br Beijos:- Keila - Curitiba - Pr - Podes falar comigo, direto dela. Brevemente uma Carta Aberta. - Embora derrubem a p=E1gina eu a subo em 3 horas novamente. "Esta mensagem =E9 enviada com a complac=EAncia da nova legisla=E7=E3o=20 sobre correio eletr=F4nico, Se=E7=E3o 301, Par=E1grafo (a) (2) (c) Decreto S. 1= 618, T=EDtulo Terceiro aprovado pelo "105=BA Congresso Base das Normativas Internacionais sobre o SPAM". Este E-mail n=E3o poder=E1 ser considerado SPAM quando incluir uma forma de ser removido. Para ser removido de futuros correios, simplesmente responda indicando no Assunto: REMOVER" From gward@python.net Fri Dec 7 02:27:57 2001 From: gward@python.net (Greg Ward) Date: Thu, 6 Dec 2001 21:27:57 -0500 Subject: [Python-Dev] Mac OS X sys.platform value In-Reply-To: <20011205155939.8936A303183@snelboot.oratrix.nl> References: <20011205155939.8936A303183@snelboot.oratrix.nl> Message-ID: <20011206212757.A9167@gerg.ca> On 05 December 2001, Jack Jansen said: > Anyway, where sys.platform used to be "darwin1" upto and including 10.1.0 it > is now "darwin5" for 10.1.1 (and it will go up to "darwin6" for 10.2, etc). > > I am of a mind to take the "1" out of sys.platform, so that it becomes > "darwin". +1 from me. If we could turn the clock back 5 or 6 years, I would also do the same on Linux. Note that I share Marc-André's dim view of tacking the first digit of the kernel version number (or whatever) onto sys.platform -- the Distutils has Yet Another way of creating platform descriptions. It's more complex than sys.platform, but quite a lot less fancy than Marc-André's platform.py. See distutils.util.get_platform(), if you're curious. Greg -- Greg Ward - Linux weenie gward@python.net http://starship.python.net/~gward/ Never put off till tomorrow what you can put off till the day after tomorrow. From anthony@interlink.com.au Fri Dec 7 03:58:41 2001 From: anthony@interlink.com.au (Anthony Baxter) Date: Fri, 07 Dec 2001 14:58:41 +1100 Subject: [Python-Dev] dumbdbm.py Message-ID: <200112070358.fB73wf425864@mbuna.arbhome.com.au> just had reason to be looking at dumbdbm.py - perhaps "terminallybrokendbm.py" would be a better name. There's a whole host of "oh. my. god." type things in there that are just broken, with massive dataloss problems if you end up (innocently) getting it from anydbm.py. I'm going to fix it either this arvo/evening or tomorrow, but it really shouldn't go out as part of 2.2 in it's current form. Is it too late to consider either fixing it or removing it from anydbm's list? Anthony From guido@python.org Fri Dec 7 04:03:29 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Dec 2001 23:03:29 -0500 Subject: [Python-Dev] dumbdbm.py In-Reply-To: Your message of "Fri, 07 Dec 2001 14:58:41 +1100." <200112070358.fB73wf425864@mbuna.arbhome.com.au> References: <200112070358.fB73wf425864@mbuna.arbhome.com.au> Message-ID: <200112070403.XAA07071@cj20424-a.reston1.va.home.com> > just had reason to be looking at dumbdbm.py - perhaps > "terminallybrokendbm.py" would be a better name. There's a whole host of > "oh. my. god." type things in there that are just broken, with massive > dataloss problems if you end up (innocently) getting it from anydbm.py. > > I'm going to fix it either this arvo/evening or tomorrow, but it really > shouldn't go out as part of 2.2 in it's current form. Is it too late to > consider either fixing it or removing it from anydbm's list? Please fix it. It was never intended to be production quality. But it has found more use than it should. Then again, dbhash.py isn't much better...[*] :-( --Guido van Rossum (home page: http://www.python.org/~guido/) [*] http://www.sleepycat.com/historic.html From mwh@python.net Fri Dec 7 15:59:13 2001 From: mwh@python.net (Michael Hudson) Date: 07 Dec 2001 10:59:13 -0500 Subject: [Python-Dev] on the subject of SF... In-Reply-To: barry@zope.com's message of "Thu, 6 Dec 2001 09:36:11 -0500" References: <200112060745.fB67j4e18189@mbuna.arbhome.com.au> <15375.8942.722195.533597@anthem.wooz.org> <3C0F2EE0.BE1D2980@lemburg.com> <15375.33371.540885.525907@anthem.wooz.org> Message-ID: <2mu1v33tta.fsf@starship.python.net> barry@zope.com (Barry A. Warsaw) writes: > >>>>> "M" == M writes: > > M> And please also a Group "Python 2.3" so that feature requests > M> for that version can be assigned to it ?! > > So far we have no groups defined for the feature request tracker. I > wonder if that's intentional. Does anybody look at the feature request tracker ever? I notice I'm one of four possible assignees (presumably 'cause I'm new). Does email from it go anywhere? Cheers, M. -- Slim Shady is fed up with your shit, and he's going to kill you. -- Eminem, "Public Service Announcement 2000" From guido@python.org Fri Dec 7 16:13:49 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Dec 2001 11:13:49 -0500 Subject: [Python-Dev] on the subject of SF... In-Reply-To: Your message of "07 Dec 2001 10:59:13 EST." <2mu1v33tta.fsf@starship.python.net> References: <200112060745.fB67j4e18189@mbuna.arbhome.com.au> <15375.8942.722195.533597@anthem.wooz.org> <3C0F2EE0.BE1D2980@lemburg.com> <15375.33371.540885.525907@anthem.wooz.org> <2mu1v33tta.fsf@starship.python.net> Message-ID: <200112071613.LAA09963@cj20424-a.reston1.va.home.com> > Does anybody look at the feature request tracker ever? I notice I'm > one of four possible assignees (presumably 'cause I'm new). Does > email from it go anywhere? New features were supposed to be sent to python-bugs-list@python.org, but somehow that wasn't set, so no, nobody has looked at this in a long time... :-( I'll have a look now, and I've set the email address. --Guido van Rossum (home page: http://www.python.org/~guido/) From neal@metaslash.com Fri Dec 7 16:15:24 2001 From: neal@metaslash.com (Neal Norwitz) Date: Fri, 07 Dec 2001 11:15:24 -0500 Subject: [Python-Dev] update on memory leaks in 2.2 Message-ID: <3C10EB1C.9302CA8@metaslash.com> Command line: (longexp crashes purify for some reason) ./python -E -tt ./Lib/test/regrtest.py -u network -x test_longexp.py Results: 168 tests OK. 1 test failed: test_socket_ssl 15 tests skipped: test_al test_bsddb test_cd test_cl test_curses test_dl test_gl test_imgfile test_linuxaudiodev test_nis test_ntpath test_openpty test_sundry test_winreg test_winsound test test_socket_ssl crashed -- socket.sslerror: SSL_CTX_new error When running test_curses: test test_curses crashed -- _curses.error: curs_set() returned ERR Uninitialized Memory Reads (don't think these are python's fault): gdbm_open() gdbmmodule.c:68 getaddrinfo() socketmodule.c:636 Memory Leaks: NONE Potential Memory Leaks: 1012 from 4 different allocating points -- all in WeakRef Things look great! Let me know if any of the skipped tests ought to have been run. Most likely possibility seems to be bsddb, sundry also needs bsddb. Neal From guido@python.org Fri Dec 7 16:26:58 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Dec 2001 11:26:58 -0500 Subject: [Python-Dev] tracker admin Message-ID: <200112071626.LAA09993@cj20424-a.reston1.va.home.com> I just discovered a few new (?) features of the bug/patch tracker admin page. - You can disable anonymous submission. I've done this for all our trackers (bugs, patches, features), since we have way too many problems with anonymous submissions where the user doesn't provide enough info and we have no way to get back to them. - You can provide custom text for the "submit new" page. I've added a warning about the annoying upload checkbox, and some additional suggestions for patches. Please feel free to provide additional suggestions. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Dec 7 16:38:35 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Dec 2001 11:38:35 -0500 Subject: [Python-Dev] update on memory leaks in 2.2 In-Reply-To: Your message of "Fri, 07 Dec 2001 11:15:24 EST." <3C10EB1C.9302CA8@metaslash.com> References: <3C10EB1C.9302CA8@metaslash.com> Message-ID: <200112071638.LAA12876@cj20424-a.reston1.va.home.com> > Command line: (longexp crashes purify for some reason) > ./python -E -tt ./Lib/test/regrtest.py -u network -x test_longexp.py > > Results: > 168 tests OK. > 1 test failed: > test_socket_ssl > 15 tests skipped: > test_al test_bsddb test_cd test_cl test_curses test_dl test_gl > test_imgfile test_linuxaudiodev test_nis test_ntpath test_openpty > test_sundry test_winreg test_winsound A few of these skips weren't expected: test_sundry, test_openpty, test_bsddb. What's missing to run those? > test test_socket_ssl crashed -- socket.sslerror: SSL_CTX_new error Works for me. Probably your platform. The SSL code sucks anyway. > When running test_curses: > > test test_curses crashed -- _curses.error: curs_set() returned ERR Works for me. > > Uninitialized Memory Reads (don't think these are python's fault): > > gdbm_open() gdbmmodule.c:68 > getaddrinfo() socketmodule.c:636 > > Memory Leaks: > > NONE Woohoo! So I'll close the remaining leak report! > Potential Memory Leaks: > > 1012 from 4 different allocating points -- all in WeakRef Any more info on these, or are these just red herrings? > Things look great! Agreed. > Let me know if any of the skipped tests ought to have been run. > Most likely possibility seems to be bsddb, sundry also needs bsddb. Ah, that's why sundry fails. I think it shouldn't depend on bsddb; I'll look into that. --Guido van Rossum (home page: http://www.python.org/~guido/) From neal@metaslash.com Fri Dec 7 16:54:54 2001 From: neal@metaslash.com (Neal Norwitz) Date: Fri, 07 Dec 2001 11:54:54 -0500 Subject: [Python-Dev] update on memory leaks in 2.2 References: <3C10EB1C.9302CA8@metaslash.com> <200112071638.LAA12876@cj20424-a.reston1.va.home.com> Message-ID: <3C10F45E.6476A376@metaslash.com> Guido van Rossum wrote: > > > Command line: (longexp crashes purify for some reason) > > ./python -E -tt ./Lib/test/regrtest.py -u network -x test_longexp.py > > > > Results: > > 168 tests OK. > > 1 test failed: > > test_socket_ssl > > 15 tests skipped: > > test_al test_bsddb test_cd test_cl test_curses test_dl test_gl > > test_imgfile test_linuxaudiodev test_nis test_ntpath test_openpty > > test_sundry test_winreg test_winsound > > A few of these skips weren't expected: test_sundry, test_openpty, > test_bsddb. What's missing to run those? The platform is Sun Solaris 8 (2.8). Purify 2002. For bsddb, do I need to install something that isn't "standard"? test test_openpty skipped -- No openpty() available. > > Potential Memory Leaks: > > > > 1012 from 4 different allocating points -- all in WeakRef > > Any more info on these, or are these just red herrings? The details are below. Neal -- PLK: 704 bytes potentially leaked in 16 blocks This memory was allocated from: malloc [rtlib.o] _PyObject_GC_New [gcmodule.c:832] PyWeakref_NewRef [weakrefobject.c:37] weakref_ref [_weakref.c:73] PyCFunction_Call [methodobject.c:101] eval_frame [ceval.c:1989] PyEval_EvalCodeEx [ceval.c:2570] function_call [funcobject.c:374] PyObject_Call [abstract.c:1665] instancemethod_call [classobject.c:2276] PyObject_Call [abstract.c:1665] PyEval_CallObjectWithKeywords [ceval.c:3034] Block of 44 bytes (16 times); last block at 0x18bf0a8 -- PLK: 220 bytes potentially leaked in 5 blocks This memory was allocated from: malloc [rtlib.o] _PyObject_GC_New [gcmodule.c:832] PyWeakref_NewRef [weakrefobject.c:37] add_subclass [typeobject.c:2054] PyType_Ready [typeobject.c:2026] type_new [typeobject.c:1142] type_call [typeobject.c:148] PyObject_Call [abstract.c:1665] PyEval_CallObjectWithKeywords [ceval.c:3034] PyObject_CallFunction [abstract.c:1656] build_class [ceval.c:3583] eval_frame [ceval.c:1507] Block of 44 bytes (5 times); last block at 0xced9b8 -- PLK: 44 bytes potentially leaked at 0xcb2ba8 This memory was allocated from: malloc [rtlib.o] _PyObject_GC_New [gcmodule.c:832] PyWeakref_NewRef [weakrefobject.c:37] add_subclass [typeobject.c:2054] PyType_Ready [typeobject.c:2026] type_new [typeobject.c:1142] tp_new_wrapper [typeobject.c:2586] PyCFunction_Call [methodobject.c:69] PyObject_Call [abstract.c:1665] do_call [ceval.c:3247] eval_frame [ceval.c:1984] PyEval_EvalCodeEx [ceval.c:2570] -- PLK: 44 bytes potentially leaked at 0xcb3470 This memory was allocated from: malloc [rtlib.o] _PyObject_GC_New [gcmodule.c:832] PyWeakref_NewRef [weakrefobject.c:37] add_subclass [typeobject.c:2054] PyType_Ready [typeobject.c:2026] type_new [typeobject.c:1142] tp_new_wrapper [typeobject.c:2586] PyCFunction_Call [methodobject.c:69] PyObject_Call [abstract.c:1665] slot_tp_new [typeobject.c:3355] type_call [typeobject.c:148] PyObject_Call [abstract.c:1665] From neal@metaslash.com Fri Dec 7 16:58:53 2001 From: neal@metaslash.com (Neal Norwitz) Date: Fri, 07 Dec 2001 11:58:53 -0500 Subject: [Python-Dev] update on memory leaks in 2.2 References: <3C10EB1C.9302CA8@metaslash.com> <200112071638.LAA12876@cj20424-a.reston1.va.home.com> Message-ID: <3C10F54D.F20C2C71@metaslash.com> Guido van Rossum wrote: > Ah, that's why sundry fails. I think it shouldn't depend on bsddb; > I'll look into that. test_sundry passes with no leaks now. Neal From fdrake@acm.org Fri Dec 7 18:01:36 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 7 Dec 2001 13:01:36 -0500 Subject: [Python-Dev] update on memory leaks in 2.2 In-Reply-To: <3C10F45E.6476A376@metaslash.com> References: <3C10EB1C.9302CA8@metaslash.com> <200112071638.LAA12876@cj20424-a.reston1.va.home.com> <3C10F45E.6476A376@metaslash.com> Message-ID: <15377.1024.829156.22274@grendel.zope.com> Neal Norwitz's Purify writes: > PLK: 704 bytes potentially leaked in 16 blocks > This memory was allocated from: > malloc [rtlib.o] > _PyObject_GC_New [gcmodule.c:832] > PyWeakref_NewRef [weakrefobject.c:37] > weakref_ref [_weakref.c:73] > PyCFunction_Call [methodobject.c:101] > eval_frame [ceval.c:1989] > PyEval_EvalCodeEx [ceval.c:2570] > function_call [funcobject.c:374] > PyObject_Call [abstract.c:1665] > instancemethod_call [classobject.c:2276] > PyObject_Call [abstract.c:1665] > PyEval_CallObjectWithKeywords [ceval.c:3034] > Block of 44 bytes (16 times); last block at 0x18bf0a8 Free'd weakref objects are stored in a free list, so it makes sense that these are labelled potential leaks. We could either give up using a free list for these, or we could allocate blocks of these rather than allocating them individually. The latter would allow still better performance and would reduce the malloc overhead. That change would be a higher risk than tossing the free list, and tossing it would be a higher risk than keeping it this close to the release. But we're probably far enough away that any of the options (no free list, blocking allocation, and leaving it alone) are manageable. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From barry@zope.com Fri Dec 7 17:20:22 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 7 Dec 2001 12:20:22 -0500 Subject: [Python-Dev] update on memory leaks in 2.2 References: <3C10EB1C.9302CA8@metaslash.com> <200112071638.LAA12876@cj20424-a.reston1.va.home.com> <3C10F45E.6476A376@metaslash.com> <15377.1024.829156.22274@grendel.zope.com> Message-ID: <15376.64086.678564.584731@anthem.wooz.org> Note that there are tons of memory that is validly still alive when Python exists. As we've seen, trying to clean all that up /can/ have a big negative impact on shutdown. So I wouldn't worry about them. Insure labels them as (paraphrasing) "memory still allocated at exit", distinct from memory leaked. I think this is equivalent to Purify's "potentially leaked" and "leaked" memory respectively. Note that at least stringobject.c has a hook for clearing out all those interned strings at program shutdown time, which I added to clean up Insure's output. When INTERN_STRINGS is defined, the extra function _Py_ReleaseInternedStrings() is defined. Then, in Modules/main.c, in Py_Main(), if __INSURE__ is defined that function is called, so this memory doesn't show up in the report. It may be worthwhile generalizing this approach, and adding it to other static long-lived data structures, simply as an aid to memory debugging. As long as it isn't enabled by default, that should be fine. -Barry From guido@python.org Fri Dec 7 17:28:15 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Dec 2001 12:28:15 -0500 Subject: [Python-Dev] update on memory leaks in 2.2 In-Reply-To: Your message of "Fri, 07 Dec 2001 13:01:36 EST." <15377.1024.829156.22274@grendel.zope.com> References: <3C10EB1C.9302CA8@metaslash.com> <200112071638.LAA12876@cj20424-a.reston1.va.home.com> <3C10F45E.6476A376@metaslash.com> <15377.1024.829156.22274@grendel.zope.com> Message-ID: <200112071728.MAA13868@cj20424-a.reston1.va.home.com> > Neal Norwitz's Purify writes: > > PLK: 704 bytes potentially leaked in 16 blocks > > This memory was allocated from: > > malloc [rtlib.o] > > _PyObject_GC_New [gcmodule.c:832] > > PyWeakref_NewRef [weakrefobject.c:37] > > weakref_ref [_weakref.c:73] > > PyCFunction_Call [methodobject.c:101] > > eval_frame [ceval.c:1989] > > PyEval_EvalCodeEx [ceval.c:2570] > > function_call [funcobject.c:374] > > PyObject_Call [abstract.c:1665] > > instancemethod_call [classobject.c:2276] > > PyObject_Call [abstract.c:1665] > > PyEval_CallObjectWithKeywords [ceval.c:3034] > > Block of 44 bytes (16 times); last block at 0x18bf0a8 > > Free'd weakref objects are stored in a free list, so it makes sense > that these are labelled potential leaks. > We could either give up using a free list for these, or we could > allocate blocks of these rather than allocating them individually. > The latter would allow still better performance and would reduce the > malloc overhead. That change would be a higher risk than tossing the > free list, and tossing it would be a higher risk than keeping it this > close to the release. But we're probably far enough away that any of > the options (no free list, blocking allocation, and leaving it alone) > are manageable. Thanks for the explanation! The simplest way to avoid the Purify "potential leak" complaints then might be to add a way to explicitly zap this free list at Py_Finalize() time, like we do with other custom free lists. That should be a post-2.2 feature. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@home.com Fri Dec 7 17:35:21 2001 From: tim.one@home.com (Tim Peters) Date: Fri, 7 Dec 2001 12:35:21 -0500 Subject: [Python-Dev] update on memory leaks in 2.2 In-Reply-To: <15377.1024.829156.22274@grendel.zope.com> Message-ID: [Fred L. Drake, Jr.] > Free'd weakref objects are stored in a free list, so it makes sense > that these are labelled potential leaks. > We could either give up using a free list for these, or we could > allocate blocks of these rather than allocating them individually. > The latter would allow still better performance and would reduce the > malloc overhead. -0. When we turn on pymalloc in 2.3, blocks will get allocated by magic, and more space-efficiently than you're going to do by hand (it bites one malloc overhead per 256KB). pymalloc will likely be a little slower than a wholly customized free list, but it will get all the memory benefit, and unlike type-specific free lists allows memory to be recycled across types as programs progress. > That change would be a higher risk than tossing the free list, and > tossing it would be a higher risk than keeping it this close to the > release. But we're probably far enough away that any of the options > (no free list, blocking allocation, and leaving it alone) are > manageable. There's another possibility: as float objects do, add a module xxx_fini() routine to clear the free list at shutdown. From fdrake@acm.org Fri Dec 7 18:33:06 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 7 Dec 2001 13:33:06 -0500 Subject: [Python-Dev] update on memory leaks in 2.2 In-Reply-To: References: <15377.1024.829156.22274@grendel.zope.com> Message-ID: <15377.2914.145010.889503@grendel.zope.com> Tim Peters writes: > -0. When we turn on pymalloc in 2.3, blocks will get allocated by magic, > and more space-efficiently than you're going to do by hand (it bites one > malloc overhead per 256KB). pymalloc will likely be a little slower than a That's good enough for me! I'll look forward to ripping the free list out completely. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From guido@python.org Fri Dec 7 17:44:04 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Dec 2001 12:44:04 -0500 Subject: [Python-Dev] update on memory leaks in 2.2 In-Reply-To: Your message of "Fri, 07 Dec 2001 12:20:22 EST." <15376.64086.678564.584731@anthem.wooz.org> References: <3C10EB1C.9302CA8@metaslash.com> <200112071638.LAA12876@cj20424-a.reston1.va.home.com> <3C10F45E.6476A376@metaslash.com> <15377.1024.829156.22274@grendel.zope.com> <15376.64086.678564.584731@anthem.wooz.org> Message-ID: <200112071744.MAA13971@cj20424-a.reston1.va.home.com> > Insure labels them as (paraphrasing) "memory still allocated at exit", > distinct from memory leaked. I think this is equivalent to Purify's > "potentially leaked" and "leaked" memory respectively. I don't think so. Insure (uselessly) lists *all* memory that's still allocated and reachable at exit. Purify only reports certain blocks (in this case they were a bunch of weak refs) as potentially leaked. I'm not sure what makes them potentially leaked, but it must be a stronger condition than "still exists at exit". As Fred mentioned, this may have to do with the weaklist free list implementation. > Note that at least stringobject.c has a hook for clearing out all > those interned strings at program shutdown time, which I added to > clean up Insure's output. When INTERN_STRINGS is defined, the extra > function _Py_ReleaseInternedStrings() is defined. Then, in > Modules/main.c, in Py_Main(), if __INSURE__ is defined that function > is called, so this memory doesn't show up in the report. > > It may be worthwhile generalizing this approach, and adding it to > other static long-lived data structures, simply as an aid to memory > debugging. As long as it isn't enabled by default, that should be > fine. Most types that have a custom free list already have a cleanup routine that's always called by Py_Finalize(); the weakref free list is lacking one. The interned list is special because it's generally unsafe to clear it out when Py_Finalize() might be followed by Py_Initialize() again -- that's why it's only done when running Insure++. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Dec 7 18:02:30 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Dec 2001 13:02:30 -0500 Subject: [Python-Dev] update on memory leaks in 2.2 In-Reply-To: Your message of "Fri, 07 Dec 2001 12:56:51 EST." <3C1102E3.447B6D8B@metaslash.com> References: <3C10EB1C.9302CA8@metaslash.com> <200112071638.LAA12876@cj20424-a.reston1.va.home.com> <3C10F45E.6476A376@metaslash.com> <200112071713.MAA13808@cj20424-a.reston1.va.home.com> <3C1102E3.447B6D8B@metaslash.com> Message-ID: <200112071802.NAA14180@cj20424-a.reston1.va.home.com> > Guido van Rossum wrote: > > > What's a potential leak? Could it be that these are in cycles? The > > reported blocks are all weak reference objects; most of then are > > allocated by add_subclass(), which uses weak refs for the list of > > subclasses of a given (new-style) class. I suppose most weak refs are > > in cycles (else there would be no need for weak refs). Neal replied (in private email): > From page 157 of this doc: > > http://www.rational.com/support/documentation/manuals/unixsuites/pdf/purify/purify.pdf > > A PLK message describes heap memory that you might have > leaked. You have pointers only to the middle of the region. PLK is a > warning message. > > Memory in use can sometimes appear as a PLK if the pointer > returned by malloc is offset. A common cause is referencing a > substring within a large string. Another example is when a > pointer to a C++ object is cast to the second or later base class of a > multiply-inherited object. It is offset past the other base class > objects. > > Truly leaked memory can sometimes appear as a PLK, if some > non-pointer integer within the program space, when interpreted > as a pointer, points within an otherwise leaked block of memory. > This is rather rare. Inspect the code to differentiate between these > causes of PLK reports. Thanks, that's very helpful! Here's how I now understand what's happened. Weak references are garbage collected, which means that the malloc pointer points 12 bytes *before* the start of the object as the rest of Python knows about it. Objects in the free list are linked together via their object pointer, so they are only known via a pointer into their "middle". We don't see this warning for other GC'ed objects, because they are in one of the collector's linked lists, which point to the start of the GC header which is the malloc'ed pointer. But freed weak references are not known to the collector! So I'm confident that these are not leaks. --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Sat Dec 8 06:30:53 2001 From: fdrake@acm.org (Fred L. Drake) Date: Sat, 8 Dec 2001 01:30:53 -0500 (EST) Subject: [Python-Dev] [development doc updates] Message-ID: <20011208063053.2367D286BC@beowolf.digicool.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Another day of updates; various fixes and clarifications. From fredrik@pythonware.com Sun Dec 9 10:04:17 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Sun, 9 Dec 2001 11:04:17 +0100 Subject: [Python-Dev] update on memory leaks in 2.2 References: <3C10EB1C.9302CA8@metaslash.com> <200112071638.LAA12876@cj20424-a.reston1.va.home.com> <3C10F45E.6476A376@metaslash.com> <15377.1024.829156.22274@grendel.zope.com> <15376.64086.678564.584731@anthem.wooz.org> <200112071744.MAA13971@cj20424-a.reston1.va.home.com> Message-ID: <05ab01c18098$e2380130$ced241d5@hagrid> guido wrote: > I don't think so. Insure (uselessly) lists *all* memory that's still > allocated and reachable at exit. Purify only reports certain blocks > (in this case they were a bunch of weak refs) as potentially leaked. > I'm not sure what makes them potentially leaked, but it must be a > stronger condition than "still exists at exit". purify runs a garbage collection algorithm on your program to figure out if something has leaked. iirc, a potential memory leak is when you no longer have a pointer to the beginning of an allocated block, but there's a valid pointer to somewhere inside the block. a true leak is when you no longer point to the block (i.e when even a conservative garbage collector could safely destroy it). From tim.one@home.com Mon Dec 10 00:06:13 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 9 Dec 2001 19:06:13 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <3C0F1A6D.417B603C@ActiveState.com> Message-ID: [David Ascher] > - popen on Windows XP seems problematic (needs a repro test there -- > not analyzed yet) We (PythonLabs) don't have XP, so you're on your own for tracking it down. popen() fails in more than one bizarre way on Win9X, and always has, and likely always will. From tim.one@home.com Mon Dec 10 00:11:43 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 9 Dec 2001 19:11:43 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <200112060727.fB67RXZ08417@mira.informatik.hu-berlin.de> Message-ID: [Martin v. Loewis] > :-) I personally don't see it as a problem that bug reports are piling > up. The GCC GNATS has 1341 "open" bug reports (meaning either "open" > or "analyzed") at the moment. bugzilla.mozilla.org reports 19611 bugs > in the "open" categories. Microsoft reportedly has over 60000 open > reports for Windows 2000. Neither of these groups despair :-) If any language I worked on previously had a few hundred open bug reports (not counting feature requests), we would have shut the group down and pursued a new implementation. If you can get the open bugs to fit on one screen, it tends to stay that way; once it spills over a screen, it appears there's no bound on how many additional screens are tolerated (0, 1, infinity). I'm doing my part by reducing my font size and buying a much larger monitor . From jason@crash.org Mon Dec 10 00:30:00 2001 From: jason@crash.org (Jason L. Asbahr) Date: Sun, 9 Dec 2001 18:30:00 -0600 Subject: [Python-Dev] win32all Message-ID: Any thoughts on making this package more prominent on the Python.org download pages? Like under the primary Python 2.1.1 and Python 2.2 download links? ______________________________________________________________________ Jason Asbahr jason@asbahr.com From guido@python.org Mon Dec 10 01:14:26 2001 From: guido@python.org (Guido van Rossum) Date: Sun, 09 Dec 2001 20:14:26 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: Your message of "Sun, 09 Dec 2001 19:11:43 EST." References: Message-ID: <200112100114.UAA11756@cj20424-a.reston1.va.home.com> > [Martin v. Loewis] > > :-) I personally don't see it as a problem that bug reports are piling > > up. The GCC GNATS has 1341 "open" bug reports (meaning either "open" > > or "analyzed") at the moment. bugzilla.mozilla.org reports 19611 bugs > > in the "open" categories. Microsoft reportedly has over 60000 open > > reports for Windows 2000. Neither of these groups despair :-) [Tim] > If any language I worked on previously had a few hundred open bug reports > (not counting feature requests), we would have shut the group down and > pursued a new implementation. It depends on what the bugs are. Many bugs in our tracker are intractable because we don't have the platform where it occurs or the user didn't supply enough information to reproduce it. I imagine the GCC bugs are much of the same, just more (because GCC is more complex and wider used than Python -- most GCC users are even more clueless about GCC's implementation than Python users are about Python's implementation). I try to close bugs from clueless users if there's no response to a request for more info, but often something that might be a bug but isn't easily reproduced stays open for many many months. So I'm not sure that it's realistic to keep all bugs on a page. I did my part against bugs without someone to ask for more info by forbidding anonymous bugs, recently. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Dec 10 03:05:03 2001 From: guido@python.org (Guido van Rossum) Date: Sun, 09 Dec 2001 22:05:03 -0500 Subject: [Python-Dev] win32all In-Reply-To: Your message of "Sun, 09 Dec 2001 18:30:00 CST." References: Message-ID: <200112100305.WAA12125@cj20424-a.reston1.va.home.com> > Any thoughts on making this package more prominent on the Python.org > download > pages? Like under the primary Python 2.1.1 and Python 2.2 download links? Good idea. Done for 1.5.2, 2.0.1 and 2.1.1. Not for 2.2 yet, since there's no version of win32all for 2.2 available AFAIK. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@home.com Mon Dec 10 03:44:31 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 9 Dec 2001 22:44:31 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <200112100114.UAA11756@cj20424-a.reston1.va.home.com> Message-ID: [Tim] > If any language I worked on previously had a few hundred open > bug reports (not counting feature requests), we would have shut the > group down and pursued a new implementation. [Guido] > It depends on what the bugs are. Actually, it would not have: a backlog of that many bugs (regardless of nature) would mean they'd never get fixed -- it's hopeless. Time then to cut losses. The compilers I worked on previously sold multi-million dollar machines, though, and had to work only with our own OS(es) and libraries. Python's life is more complicated than that -- and Python doesn't have a few hundred open bugs anyway (there are only 183 right now ). The Python bug base also has gripes about everything from docs to python-mode.el in it. > Many bugs in our tracker are intractable because we don't have the > platform where it occurs or the user didn't supply enough information > to reproduce it. Those should be closed after a reasonable time: nobody is going to work on them. OTOH, I don't think there are many *open* bugs of this ilk (I routinely close such after a month, with "closed for lack of requested followup" -- and nobody has complained about that). > I imagine the GCC bugs are much of the same, just more (because GCC is > more complex and wider used than Python -- most GCC users are even more > clueless about GCC's implementation than Python users are about Python's > implementation). My only interaction with the gcc bug base was helping to whittle down oodles of bad codegen cases on new ports. I have no idea what else may be in there. > I try to close bugs from clueless users if there's no response to a > request for more info, but often something that might be a bug but > isn't easily reproduced stays open for many many months. I don't think there are many open bugs of that ilk either. > So I'm not sure that it's realistic to keep all bugs on a page. We were close once. Now the open patches don't even fit on a page. > I did my part against bugs without someone to ask for more info by > forbidding anonymous bugs, recently. Appreciated, but those were the easiest to close <0.9 wink>. the-things-that-get-done-are-the-things-that-get-worked-on-ly y'rs - tim From tim.one@home.com Mon Dec 10 05:22:15 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 10 Dec 2001 00:22:15 -0500 Subject: [Python-Dev] Subclassing varying length types (What's a PyStructSequence ?) In-Reply-To: <3C0B4152.3739F4EA@lemburg.com> Message-ID: [MAL] > Have you tried disabling all free list and using pymalloc > instead ? No, but I haven't tried anything -- it's a 2.3 issue. > If this pays off, I agree, we should get rid off all of them. When I do try it , it will be slower but more memory-efficient (both data and code) than the type-specific free lists, and faster and much more memory-efficient than using malloc(). > ... > I would consider moving from 8-bit strings to Unicode an > improvement in flexibility. Sure. Moving from one malloc to two is orthogonal. > It also results in better algroithms (== simpler, less error-prone, > etc. in this case). Unclear what "it" means; assuming it means using two mallocs instead of one for a Unicode string object, the 8-bit string algorithms haven't been a particular source of bugs. People mutating strings at the C level has been. > As I said, it's a tradeoff flexibility vs. memory consumption. > Whether it pays off depends on your application environment. It > certainly does for companies like Micron and pays off stock-wise > for a lot of people... uhm, getting off-topic here :-) I've got nothing against Unicode (apart from the larger issue that the whole world would obviously be a lot better off if they switched to American English ). >> Subclassing seems easy enough to me from the Python level; I >> don't have time to revisit C-level subclasssing here (and I don't >> know that it's hackish there either, but do think it's in need of >> docs). > It is beautifully easy for non-varying-length types. Unfortunately, > it happens that some of the basic types which would be attractive > for subclassing are varying length types (such as string and > tuples). It's easy to subclass from str and tuple in Python -- even to add your own instance data. > In my case, I'm looking for away to subclass strings, but I haven't > yet found an elegant solution to the problem of adding extra > data to the instances. It's easy if you're willing to use a dict: class STR(str): def __new__(cls, strguts, n): self = str.__new__(cls, strguts) self.n = n return self s = STR('abc', 42) print s # abc print s.n # 42 __slots__ doesn't work here, though. I admit I personally don't see much attraction to subclassing from str and tuple, apart from adding additional *methods*. I suppose someone could code up two-malloc variants ... From fredrik@pythonware.com Mon Dec 10 08:26:52 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Mon, 10 Dec 2001 09:26:52 +0100 Subject: [Python-Dev] memory leaks in 2.2 References: Message-ID: <00b701c18155$9b4bbd80$ced241d5@hagrid> tim wrote: > If any language I worked on previously had a few hundred open bug reports > (not counting feature requests). btw, what's the right way to flag something as a feature request? both the data type and the group fields can be set to "feature request", but I'm not allowed to change the former. (and all my feature requests are listed under "Python - Bugs") From martin@v.loewis.de Mon Dec 10 08:24:29 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Mon, 10 Dec 2001 09:24:29 +0100 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: References: Message-ID: <200112100824.fBA8OTa01801@mira.informatik.hu-berlin.de> > My only interaction with the gcc bug base was helping to whittle down oodles > of bad codegen cases on new ports. I have no idea what else may be in > there. Many of them are of this kind. As a C++ person, I notice also (with regrets) that a significant portion is of the kind "C++ compiler rejects valid code, or crashes". They are easy to reproduce, so they probably would have caused your former companies to throw away the C++ front-end and rewrite it from scratch. In g++, this won't happen: nobody has the resources to rewrite it from scratch. And even with these bugs, g++ still is much better than the competition :-) > > So I'm not sure that it's realistic to keep all bugs on a page. > > We were close once. Now the open patches don't even fit on a page. I think the only time we were close was when the database was new. Regards, Martin From martin@v.loewis.de Mon Dec 10 08:34:19 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Mon, 10 Dec 2001 09:34:19 +0100 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <00b701c18155$9b4bbd80$ced241d5@hagrid> (fredrik@pythonware.com) References: <00b701c18155$9b4bbd80$ced241d5@hagrid> Message-ID: <200112100834.fBA8YJ901851@mira.informatik.hu-berlin.de> > btw, what's the right way to flag something as a feature > request? I'd prefer to see them in the Feature Requests tracker. The Feature Requests group predates the separate feature requests tracker; if they are in a separate tracker, people won't have to go over them again and again > both the data type and the group fields can be set to "feature > request", but I'm not allowed to change the former. At the moment, it might be fastest if you post the numbers, so I can reclassify them. Alternatively, someone with appropriate powers could give you write access to the feature requests tracker, and the data type field (I think you need to be tracker admin for that). Regards, Martin From mal@lemburg.com Mon Dec 10 09:42:59 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 10 Dec 2001 10:42:59 +0100 Subject: [Python-Dev] on the subject of SF... References: <200112060745.fB67j4e18189@mbuna.arbhome.com.au> <15375.8942.722195.533597@anthem.wooz.org> <3C0F2EE0.BE1D2980@lemburg.com> <15375.33371.540885.525907@anthem.wooz.org> <2mu1v33tta.fsf@starship.python.net> Message-ID: <3C1483A3.AF10032F@lemburg.com> Michael Hudson wrote: > > barry@zope.com (Barry A. Warsaw) writes: > > > >>>>> "M" == M writes: > > > > M> And please also a Group "Python 2.3" so that feature requests > > M> for that version can be assigned to it ?! > > > > So far we have no groups defined for the feature request tracker. I > > wonder if that's intentional. > > Does anybody look at the feature request tracker ever? I notice I'm > one of four possible assignees (presumably 'cause I'm new). Does > email from it go anywhere? I was actually thinking of the bug tracker. I saw two group entries there for specific Python versions and thought it would be nice to use the group as feature request target specifier. Nevermind, -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Mon Dec 10 09:44:50 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 10 Dec 2001 10:44:50 +0100 Subject: [Python-Dev] tracker admin References: <200112071626.LAA09993@cj20424-a.reston1.va.home.com> Message-ID: <3C148412.3FA26450@lemburg.com> Guido van Rossum wrote: > > I just discovered a few new (?) features of the bug/patch tracker > admin page. > > - You can disable anonymous submission. I've done this for all our > trackers (bugs, patches, features), since we have way too many > problems with anonymous submissions where the user doesn't provide > enough info and we have no way to get back to them. > > - You can provide custom text for the "submit new" page. I've added a > warning about the annoying upload checkbox, and some additional > suggestions for patches. Please feel free to provide additional > suggestions. That's great news ! Both of these will help a lot with the PSF discussions we recently had about how to legally deal with submissions. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From fredrik@pythonware.com Mon Dec 10 10:16:38 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Mon, 10 Dec 2001 11:16:38 +0100 Subject: [Python-Dev] memory leaks in 2.2 References: <200112100824.fBA8OTa01801@mira.informatik.hu-berlin.de> Message-ID: <011e01c18163$d61f3780$ced241d5@hagrid> martin wrote: > ...rewrite it from scratch. more likely, buy one from EDG: http://www.edg.com/customers.html From mal@lemburg.com Mon Dec 10 10:57:10 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 10 Dec 2001 11:57:10 +0100 Subject: [Python-Dev] Subclassing varying length types (What's a PyStructSequence ?) References: Message-ID: <3C149506.A5689065@lemburg.com> Tim Peters wrote: > > [MAL] > > Have you tried disabling all free list and using pymalloc > > instead ? > > No, but I haven't tried anything -- it's a 2.3 issue. > > > If this pays off, I agree, we should get rid off all of them. > > When I do try it , it will be slower but more memory-efficient (both > data and code) than the type-specific free lists, and faster and much more > memory-efficient than using malloc(). Well, let's do some pybench runs next year and see what the results look like. > > ... > > I would consider moving from 8-bit strings to Unicode an > > improvement in flexibility. > > Sure. Moving from one malloc to two is orthogonal. You know that I know that you knew what I was talking about :-) > > It also results in better algroithms (== simpler, less error-prone, > > etc. in this case). > > Unclear what "it" means; assuming it means using two mallocs instead of one > for a Unicode string object, the 8-bit string algorithms haven't been a > particular source of bugs. People mutating strings at the C level has been. If you ever try to support more than ASCII text in a user program, you'll find that having to deal with only one encoding safes you a whole lot of trouble. I won't even start talking about variable length encodings, encodings with builtin shift state and other goodies which are a complete nightmare to handle (e.g. various character properties such as title case, upper/lower mappings, different ways to encode a single character, collation,...). > > As I said, it's a tradeoff flexibility vs. memory consumption. > > Whether it pays off depends on your application environment. It > > certainly does for companies like Micron and pays off stock-wise > > for a lot of people... uhm, getting off-topic here :-) > > I've got nothing against Unicode (apart from the larger issue that the whole > world would obviously be a lot better off if they switched to American > English ). I suppose Mandarin would reach a larger share in world population ... and they *need* Unicode :-) > >> Subclassing seems easy enough to me from the Python level; I > >> don't have time to revisit C-level subclasssing here (and I don't > >> know that it's hackish there either, but do think it's in need of > >> docs). > > > It is beautifully easy for non-varying-length types. Unfortunately, > > it happens that some of the basic types which would be attractive > > for subclassing are varying length types (such as string and > > tuples). > > It's easy to subclass from str and tuple in Python -- even to add your own > instance data. Yeah, but that's not the point. I want to do this in C... > > In my case, I'm looking for away to subclass strings, but I haven't > > yet found an elegant solution to the problem of adding extra > > data to the instances. > > It's easy if you're willing to use a dict: I would be willing to use a dictionary. It's only that even the dictionary trick doesn't seem to work at C level. > class STR(str): > def __new__(cls, strguts, n): > self = str.__new__(cls, strguts) > self.n = n > return self > > s = STR('abc', 42) > print s # abc > print s.n # 42 > > __slots__ doesn't work here, though. > > I admit I personally don't see much attraction to subclassing from str and > tuple, apart from adding additional *methods*. I suppose someone could code > up two-malloc variants ... If you look at mxURL you'll find an extension type which tries to play nice with strings -- it would be a good candidate for a string subtype. A string type which carries along an encoding attribute would be another good candidate for a string subtype. Both need extra attributes/data fields. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From fdrake@acm.org Mon Dec 10 15:14:27 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 10 Dec 2001 10:14:27 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <200112100824.fBA8OTa01801@mira.informatik.hu-berlin.de> References: <200112100824.fBA8OTa01801@mira.informatik.hu-berlin.de> Message-ID: <15380.53587.201337.1803@grendel.zope.com> Martin v. Loewis writes: > I think the only time we were close was when the database was new. No; when the DB was new (but old enough to notice), it contained an import of all the old JiterBug reports. I think we had it pretty well whittled down around the 2.0 release. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From jeremy@zope.com Mon Dec 10 10:20:01 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Mon, 10 Dec 2001 05:20:01 -0500 (EST) Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <15380.53587.201337.1803@grendel.zope.com> References: <200112100824.fBA8OTa01801@mira.informatik.hu-berlin.de> <15380.53587.201337.1803@grendel.zope.com> Message-ID: <15380.35921.588329.420402@slothrop.digicool.com> >>>>> "FLD" == Fred L Drake, writes: FLD> Martin v. Loewis writes: >> I think the only time we were close was when the database was >> new. FLD> No; when the DB was new (but old enough to notice), it FLD> contained an FLD> import of all the old JiterBug reports. I think we had it FLD> pretty well whittled down around the 2.0 release. That's right. We spent a lot of time fixing bugs prior to the 2.0 release. Jeremy From tim.one@home.com Mon Dec 10 17:30:37 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 10 Dec 2001 12:30:37 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <00b701c18155$9b4bbd80$ced241d5@hagrid> Message-ID: [Fredrik Lundh] > btw, what's the right way to flag something as a feature > request? > > both the data type and the group fields can be set to "feature > request", but I'm not allowed to change the former. After literally hundreds of mouse clicks, just about everyone should have permission to do that now. The "data type" field determines the tracker a thing lives in, so that's the more important of the two to change. Unfortunately, when you move a report, old URLs to the report become invalid, and bring up a bizarre "invalide artifact ID" (something like that) page. > (and all my feature requests are listed under "Python - Bugs") Then I'm going to raise them all to priority 9 . From tim.one@home.com Mon Dec 10 17:44:22 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 10 Dec 2001 12:44:22 -0500 Subject: Feature Requests (RE: [Python-Dev] memory leaks in 2.2) In-Reply-To: Message-ID: The SF Feature Request tracker didn't have any Categories defined. I've now added the same Categories as are available in the Bug tracker: Build Demos and Tools Distutils Documentation Extension Modules IDLE Installation Macintosh Parser/Compiler Python Interpreter Core Python Library Regular Expressions Threads Tkinter Type/class unification Unicode Windows XML Caution: it *could* be that when you change trackers, the Category is lost; you may have to set it again after it's moved (this was certainly the case when the FR tracker didn't have any Categories). From tim.one@home.com Tue Dec 11 04:31:39 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 10 Dec 2001 23:31:39 -0500 Subject: [Python-Dev] memory leaks in 2.2 In-Reply-To: <200112100824.fBA8OTa01801@mira.informatik.hu-berlin.de> Message-ID: [Tim] >> My only interaction with the gcc bug base was helping to >> whittle down oodles of bad codegen cases on new ports. ... [Martin v. Loewis] > Many of them are of this kind. As a C++ person, I notice also (with > regrets) that a significant portion is of the kind "C++ compiler > rejects valid code, or crashes". They are easy to reproduce, so they > probably would have caused your former companies to throw away the C++ > front-end and rewrite it from scratch. As Fredrik suggested, we bought front ends whenever possible. When a single machine sells for millions, and you can buy a highly regarded front end for a few percent of that, you'd be crazy to write your own. > In g++, this won't happen: nobody has the resources to rewrite it from > scratch. And even with these bugs, g++ still is much better than the > competition :-) I'd be more upset if anyone had proved that C++ is compilable . [Guido] >>> So I'm not sure that it's realistic to keep all bugs on a page. >> We were close once. Now the open patches don't even fit on a page. > I think the only time we were close was when the database was new. As Fred and Jeremy said, when we (PythonLabs) worked intensely on closing bugs for 1.6 and 2.0, we were close to being in phenomenal shape here. We don't have a fraction of the internal resource to throw at that anymore, though, and volunteers haven't taken up the slack. So I figure the only reason Python stays in exceptionally good bug shape is that we also ignore the contributed patches <0.9 wink>. From mal@lemburg.com Tue Dec 11 10:04:55 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 11 Dec 2001 11:04:55 +0100 Subject: Feature Requests (RE: [Python-Dev] memory leaks in 2.2) References: Message-ID: <3C15DA47.FA5C39DA@lemburg.com> Tim Peters wrote: > > The SF Feature Request tracker didn't have any Categories defined. I've now > added the same Categories as are available in the Bug tracker: > > Build > Demos and Tools > Distutils > Documentation > Extension Modules > IDLE > Installation > Macintosh > Parser/Compiler > Python Interpreter Core > Python Library > Regular Expressions > Threads > Tkinter > Type/class unification > Unicode > Windows > XML Would be nice if we could also have Groups "Python 2.3", "Python 3k", etc. > Caution: it *could* be that when you change trackers, the Category is lost; > you may have to set it again after it's moved (this was certainly the case > when the FR tracker didn't have any Categories). Changing the data type pretty much reset all fields for me (including the assigned field which is particularly annoying) and the URL becomes invalid too. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From martin@v.loewis.de Tue Dec 11 10:12:00 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Tue, 11 Dec 2001 11:12:00 +0100 Subject: Feature Requests (RE: [Python-Dev] memory leaks in 2.2) In-Reply-To: <3C15DA47.FA5C39DA@lemburg.com> (mal@lemburg.com) References: <3C15DA47.FA5C39DA@lemburg.com> Message-ID: <200112111012.fBBAC0I01773@mira.informatik.hu-berlin.de> > Would be nice if we could also have Groups "Python 2.3", "Python 3k", > etc. I'm not sure. Feature requests are supposed to stay open until they are either implemented, withdrawn, or rejected for good. If you want to express that some requests are more important than others (so that they ought to go into 2.3, instead of P3k), you should give them a higher priority. Regards, Martin From mal@lemburg.com Tue Dec 11 11:00:14 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 11 Dec 2001 12:00:14 +0100 Subject: Feature Requests (RE: [Python-Dev] memory leaks in 2.2) References: <3C15DA47.FA5C39DA@lemburg.com> <200112111012.fBBAC0I01773@mira.informatik.hu-berlin.de> Message-ID: <3C15E73E.985DB93@lemburg.com> "Martin v. Loewis" wrote: > > > Would be nice if we could also have Groups "Python 2.3", "Python 3k", > > etc. > > I'm not sure. Feature requests are supposed to stay open until they > are either implemented, withdrawn, or rejected for good. If you want > to express that some requests are more important than others (so that > they ought to go into 2.3, instead of P3k), you should give them a > higher priority. If you look at it this way, yes. It's just that setting the Group to e.g. P3k would tell the submitter that the request is being considered, but won't get implemented until a few releases from now or that a specific request won't make it into e.g. Python 2.2 because of a feature freeze. Anyway, it's just an idea. BTW, I also think that we could use the feature request tracker to track all the PEP 42 requests and redirect PEP 42 to the tracker. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From fdrake@acm.org Tue Dec 11 22:16:01 2001 From: fdrake@acm.org (Fred L. Drake) Date: Tue, 11 Dec 2001 17:16:01 -0500 (EST) Subject: [Python-Dev] [development doc updates] Message-ID: <20011211221601.DB780286BC@cj42289-a.reston1.va.home.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Many various updates, including documentation for generators and extended call syntax. From skip@pobox.com (Skip Montanaro) Tue Dec 11 23:48:58 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Tue, 11 Dec 2001 17:48:58 -0600 Subject: [Python-Dev] new pubkey for SF Message-ID: <15382.39786.533044.951881@beluga.mojam.com> Folks, Where do I send a new SSH pubkey to regain non-anonymous access to the CVS repository? Thx, Skip From tim.one@home.com Wed Dec 12 00:31:57 2001 From: tim.one@home.com (Tim Peters) Date: Tue, 11 Dec 2001 19:31:57 -0500 Subject: [Python-Dev] new pubkey for SF In-Reply-To: <15382.39786.533044.951881@beluga.mojam.com> Message-ID: > Where do I send a new SSH pubkey to regain non-anonymous access to the > CVS repository? Log in to SF. From your "personal page", click on Account Options, then down to Shell Account Information, then click Edit Keys. It should be obvious from there. Beware the 2 (4? 8? 512?) hour synchronization delay. From bizopp2002us@yahoo.com Wed Dec 12 20:52:46 2001 From: bizopp2002us@yahoo.com (John McCuullen) Date: Wed, 12 Dec 2001 21:52:46 +0100 Subject: [Python-Dev] The Secret to Supercharge your MLM! Message-ID: <2191817-220011231220524660@yahoo.com> Discover "The Secret to Supercharge your MLM!"=20 A must read=2E=2E=2E=20 Hi there,=20 The secret is out!=20 Here's the mail we've all been waiting for! Read carefully and take immediate action on it!=20 I've discovered an amazing formula that will give your MLM an enormous enrollers explosion=2E You'll benefit hugely if you use it with YOUR primary MLM!=20 -THE ULTIMATE RECRUITMENT FORMULA-=20 So if you are already in MLM or about to get into one this will be the most important mail you will ever read=2E=20 Right now I know people who enroll dozens in their downlines WEEKLY using the formula I'm about to reveal=2E=20 Fred Stege, Founder of the company enrolled 127 people within 7 day's after launching! One of his enrollees from this batch has become top recruiter in his company within 90 days after using this incredible easy-to-use concept=2E=20 The NEW Formula is meant as a Front End Program to enhance "YOUR" MLM=2E I'm NOT pitching you in any MLM=2E ____________________________________________ In fact you may try it for absolutely FREE and you'll get an electrifying stack of tools straight away! ____________________________________________=20 We've discovered that the majority of Qualified MLM Leads (Prospects) all have 1 thing in common=2E=2E=2E=20 They all are looking for 3 things they desperately want and can't get in any one 100% generic System=2E=2E=2E=20 I just know that you'll agree with these 3 things and they are all to your advantage!=20 Let me reveal you the 3 most sought after topics first:=20 =B7 Cutting Edge MLM Information=2E =B7 A Legitimate Proven Formula=2E =B7 Pre Qualified Leads=2E (And how to get them for FREE)=20 Isn't that what everybody is looking for?=2E=2E Will these topics ever go out of style?=2E=2E Doesn't THAT Attract the Masses?=2E=2E=20 Does 'OLD SCHOOL MLM' attract the masses?=2E=2E=20 Discover "How you can EARN RESIDUAL INCOME While Recruiting on Auto Pilot whether they join your MLM or not!"=20 Welcome to 21st Century Massive Recruitment=2E=20 I encourage you to check the details and sign up immediately=2E That means NOW!=20 For more info please mail to mccuullen47@zonnet=2Enl and write your First Name + 'MORE INFO' in the subject line=2E You'll receive further instructions immediately=2E=20 Sincerely,=20 John McCuullen P=2ES=2E Don't forget to write your FIRST NAME + 'MORE INFO' In Subject line!=20 P=2EP=2ES=2E On the given site you'll find a secret that every PRO in our industry knows and uses=2E=2E=2E It's called: S-L-L-G: Self Liquidating Lead Generation=2E It will Create a Huge, Electrifying Surge in Your Cash Flow=2E=20 P=2EP=2EP=2ES=2E Find out the NEW 3 foot rule in MLM=2E "Why Old School MLM is nearly dead=2E"=20 From fredrik@pythonware.com Thu Dec 13 02:23:56 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 13 Dec 2001 03:23:56 +0100 Subject: [Python-Dev] what's the current plan for 2.2c1? Message-ID: <005101c1837d$3d0fc980$ced241d5@hagrid> the pep (and python.org) says december 12. just curious /F From fdrake@acm.org Thu Dec 13 19:08:41 2001 From: fdrake@acm.org (Fred L. Drake) Date: Thu, 13 Dec 2001 14:08:41 -0500 (EST) Subject: [Python-Dev] [development doc updates] Message-ID: <20011213190841.2EF95286BC@cj42289-a.reston1.va.home.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Updated the build instructions for extensions on Windows. From barry@zope.com Thu Dec 13 20:01:36 2001 From: barry@zope.com (Barry A. Warsaw) Date: Thu, 13 Dec 2001 15:01:36 -0500 Subject: [Python-Dev] Python 2.2rc1 - release branch created Message-ID: <15385.2336.521466.6255@anthem.wooz.org> I've created the Python 2.2rc1 release branch. The CVS tag is `r22rc1-branch'. There's also a tag `r22rc1-fork' -- that's the point on the trunk where the branch forks off. So far the only change in the branch is the version number. As usual, no one except Guido and myself (or our authorized bots) should make checkins to the branch. Checkins to the trunk are okay but discouraged, and generally will /not/ be merged into the release branch unless you notify us (a comment in the checkin message would suffice). I'll be cutting the release tomorrow morning local time (Friday December 14). After the release is done, the branch will be merged back into the trunk. Please note that as this is a release candidate, only essential or showstopper bugs should be fixed. Definitely, no new features should be added. According to PEP 251, Python 2.2 final is currently scheduled for 19-Dec-2001. If you're unsure whether to make a change or not, please contact us first. Enjoy, -Barry From fdrake@acm.org Thu Dec 13 20:02:45 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Thu, 13 Dec 2001 15:02:45 -0500 Subject: [Python-Dev] Doc/ tree closed for release Message-ID: <15385.2405.102160.574585@grendel.zope.com> The Doc/ tree is now closed so I can prepare for the release. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From fdrake@acm.org Fri Dec 14 04:25:14 2001 From: fdrake@acm.org (Fred L. Drake) Date: Thu, 13 Dec 2001 23:25:14 -0500 (EST) Subject: [Python-Dev] [development doc updates] Message-ID: <20011214042514.2204B286BC@beowolf.digicool.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Final 2.2c1 documentation. From barry@zope.com Fri Dec 14 20:06:13 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 14 Dec 2001 15:06:13 -0500 Subject: [Python-Dev] RELEASED Python 2.2 release candidate 1 is out! Message-ID: <15386.23477.137871.205510@anthem.wooz.org> Today we release Python 2.2c1, the first and hopefully only release candidate for Python 2.2, for your extravagation, eyestrain, and elucidation. Our thanks to everyone who helped test Python 2.2 during its alpha and beta phases, and to everyone who contributed patches, feedback, and suggestions! Barring any showstopper bugs, we plan on releasing Python 2.2 final in one week. Please continue to report any bugs you find to the bug tracker: http://sourceforge.net/bugs/?group_id=5470 Highlights of what's new for this release are outlined below. For a more complete list, please see: http://sf.net/project/shownotes.php?release_id=65583 Andrew Kuchling is writing a gentle introduction to the most important changes, titled "What's New in Python 2.2": http://www.amk.ca/python/2.2/ Guido's written his own introduction to the type/class unification at: http://www.python.org/2.2/descrintro.html Enjoy, -Barry -------------------- snip snip -------------------- What's New in Python 2.2c1? Release date: 14-Dec-2001 - The "delete attribute" method of descriptor objects is called __delete__, not __del__. - Some subtle issues with the super built-in were fixed. - hash() of an instance of a subclass of mutable type now properly raises TypeError. - New style objects now support deleting their __dict__. - -Qnew now works as documented in PEP 238. - complex() now only allows the first argument to be a string argument. - gc.get_referents was renamed to gc.get_referrers. - It is no longer necessary to use --with-suffix when building on a case-insensitive file system (such as Mac OS X HFS+). - In unix-Python on Mac OS X (and darwin) sys.platform is now "darwin", without any trailing digits. From barry@zope.com Fri Dec 14 20:19:56 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 14 Dec 2001 15:19:56 -0500 Subject: [Python-Dev] RELEASED Python 2.2 release candidate 1 is out! References: <15386.23477.137871.205510@anthem.wooz.org> Message-ID: <15386.24300.351531.84724@anthem.wooz.org> Sigh. A cut and paste error omitted the download link from the announcement. As usual, it is http://www.python.org/2.2/ Apologies, and thanks MAL! -Barry From guido@python.org Sat Dec 15 16:23:44 2001 From: guido@python.org (Guido van Rossum) Date: Sat, 15 Dec 2001 11:23:44 -0500 Subject: [Python-Dev] mpz module broken with GMP 4.0 Message-ID: <200112151623.LAA32158@cj20424-a.reston1.va.home.com> See SF bug #493631: http://sourceforge.net/tracker/index.php?func=detail&aid=493631&group_id=5470&atid=105470 The mpz modules doesn't work with GMP 4.0 (which is brand new -- http://www.swox.com/gmp/ lists Dec 1, 2001 as the release date). It looks like the mpz module is a mess. The setup.py file contains a reference to pympz.sourceforge.net, but that doesn't exist. There is, however, a SF project gmpy, which seems to be a Python wrapper for GMP 3.0 (does it work with GMP 4.0?). I'm cc'ing the project admins, maybe they can shed light on this. Does anybody know how to fix mpzmodule.c? Or should we just get rid of it? Should we at least fix the reference in setup.py to point to gmpy? --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Sat Dec 15 16:43:45 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Sat, 15 Dec 2001 11:43:45 -0500 Subject: [Python-Dev] regression tests with regrtest.py -r Message-ID: <15387.32193.854351.323371@grendel.zope.com> On a lark, I decided to try running the regression tests with -r, which tells regrtest.py to run the tests in random order. With this, I get the following warnings from test_long.py: ../Lib/test/test_long.py:266: OverflowWarning: integer addition special = [0, 1, 2, 3, sys.maxint-1, sys.maxint, sys.maxint+1] ../Lib/test/test_long.py:287: OverflowWarning: integer addition got = x + y ../Lib/test/test_long.py:291: OverflowWarning: integer subtraction got = x - y ../Lib/test/test_long.py:295: OverflowWarning: integer multiplication got = x * y ../Lib/test/test_long.py:313: OverflowWarning: integer exponentiation got = x ** y ../Lib/test/test_long.py:320: OverflowWarning: integer exponentiation got = pow(x, y, z) ../Lib/test/test_long.py:377: OverflowWarning: integer exponentiation value = 10 ** exp This was with the CVS trunk, which is still virtually identical to the 2.2c1 release. I won't have time to look at them this weekend, but this should be fixed. It may be possible there are other order-dependent behaviors, so it would be really good if people who have time can run the tests with -r on whatever platforms are available and report any other anomolies. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From guido@python.org Sat Dec 15 18:01:07 2001 From: guido@python.org (Guido van Rossum) Date: Sat, 15 Dec 2001 13:01:07 -0500 Subject: [Python-Dev] regression tests with regrtest.py -r In-Reply-To: Your message of "Sat, 15 Dec 2001 11:43:45 EST." <15387.32193.854351.323371@grendel.zope.com> References: <15387.32193.854351.323371@grendel.zope.com> Message-ID: <200112151801.NAA02181@cj20424-a.reston1.va.home.com> > On a lark, I decided to try running the regression tests with -r, Good idea! > which tells regrtest.py to run the tests in random order. With this, > I get the following warnings from test_long.py: > > ../Lib/test/test_long.py:266: OverflowWarning: integer addition > special = [0, 1, 2, 3, sys.maxint-1, sys.maxint, sys.maxint+1] [...] > > This was with the CVS trunk, which is still virtually identical to > the 2.2c1 release. > I won't have time to look at them this weekend, but this should be > fixed. It may be possible there are other order-dependent behaviors, > so it would be really good if people who have time can run the tests > with -r on whatever platforms are available and report any other > anomolies. I've found the cause: test_scope.py contains a call to warnings.resetwarnings(), which removes *all* warning filters, not just those made by the calling module. Because of the way OverflowWarning is suppressed, this removes the "preset" filter that suppresses OverflowWarning. Since an API redesign before the 2.2 release is out of the question, I'm fixing this by removing the resetwarnings() call from test_scope.py and making the filterwarnings() call there a little more specific. --Guido van Rossum (home page: http://www.python.org/~guido/) From aleax@aleax.it Sat Dec 15 18:23:26 2001 From: aleax@aleax.it (Alex Martelli) Date: Sat, 15 Dec 2001 19:23:26 +0100 Subject: [Python-Dev] Re: mpz module broken with GMP 4.0 In-Reply-To: <200112151623.LAA32158@cj20424-a.reston1.va.home.com> References: <200112151623.LAA32158@cj20424-a.reston1.va.home.com> Message-ID: <0112151923261C.01556@arthur> On Saturday 15 December 2001 17:23, Guido van Rossum wrote: > See SF bug #493631: > > http://sourceforge.net/tracker/index.php?func=detail&aid=493631&group_id= >5470&atid=105470 > > The mpz modules doesn't work with GMP 4.0 (which is brand new -- > http://www.swox.com/gmp/ lists Dec 1, 2001 as the release date). > > It looks like the mpz module is a mess. The setup.py file contains a > reference to pympz.sourceforge.net, but that doesn't exist. > > There is, however, a SF project gmpy, which seems to be a Python > wrapper for GMP 3.0 (does it work with GMP 4.0?). I'm cc'ing the > project admins, maybe they can shed light on this. Thanks for the indication -- I was not aware of the new GMP 4.0. A rapid test shows the current gmpy.c, release 0.8, breaks in exactly one place (structure gmpstate_t used to have a member named seed, now renamed to _mp_seed). Once that single line is fixed, gmpy 0.8 builds fine (on a Linux box) with the new GMP 4.0 and passes all of its 973 unit-tests (with Python 2.2b1). I think we should rapidly release a 0.8.1 with the one-line fix (arranged so it keeps building with GMP 3.1 while also building with 4.0, of course), and plan the modest gmpy extensions needed to exploit 4.0's novelties later (it's likely that little or no work is needed for that, since 4.0 seems to have mostly relaxed previous argument constraints, and gmpy generally doesn't duplicate GMP's own argument validity tests). Having gmpy fully exploit Python 2.2 is a longer-term objective (expose all types and make them inheritable). Alex From guido@python.org Sat Dec 15 18:31:45 2001 From: guido@python.org (Guido van Rossum) Date: Sat, 15 Dec 2001 13:31:45 -0500 Subject: [Python-Dev] Re: mpz module broken with GMP 4.0 In-Reply-To: Your message of "Sat, 15 Dec 2001 19:23:26 +0100." <0112151923261C.01556@arthur> References: <200112151623.LAA32158@cj20424-a.reston1.va.home.com> <0112151923261C.01556@arthur> Message-ID: <200112151831.NAA02998@cj20424-a.reston1.va.home.com> Alex, do you have a suggestion for what to do with mpzmodule.c in the 2.2 distribution? --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Sat Dec 15 18:34:26 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Sat, 15 Dec 2001 13:34:26 -0500 Subject: [Python-Dev] Re: mpz module broken with GMP 4.0 In-Reply-To: <0112151923261C.01556@arthur> References: <200112151623.LAA32158@cj20424-a.reston1.va.home.com> <0112151923261C.01556@arthur> Message-ID: <15387.38834.552687.60342@grendel.zope.com> Alex Martelli writes: > Thanks for the indication -- I was not aware of the new GMP 4.0. A rapid > test shows the current gmpy.c, release 0.8, breaks in exactly one place I'm adding a link to the gmpy website from the mpz module documentation. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tim.one@home.com Sat Dec 15 18:43:26 2001 From: tim.one@home.com (Tim Peters) Date: Sat, 15 Dec 2001 13:43:26 -0500 Subject: [Python-Dev] regression tests with regrtest.py -r In-Reply-To: <15387.32193.854351.323371@grendel.zope.com> Message-ID: [Fred L. Drake, Jr.] > ... > It may be possible there are other order-dependent behaviors, There are, but ... > so it would be really good if people who have time can run the tests > with -r on whatever platforms are available and report any other > anomolies. By far the most frequent cause of errors when using -r is bogus module objects left behind in sys.modules on non-Unix platforms that can't import some Unix-specific things. A later attempt to import the same module again doesn't fail, but since the module object is incomplete it fails for some other reason when a test attempts to use the module. I spent countless hours worming around this in the past, but lost heart for it; on my Win9X box, the failing test has usually scrolled off the (50-line) DOS box before regrtest completes, and so also the history of the test ordering that caused it. Heh -- -r failed in test___all__ while I was typing this, but I've no idea why. I'll pursue it until I get sick to death of it . From fdrake@acm.org Sat Dec 15 18:46:16 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Sat, 15 Dec 2001 13:46:16 -0500 Subject: [Python-Dev] regression tests with regrtest.py -r In-Reply-To: <200112151801.NAA02181@cj20424-a.reston1.va.home.com> References: <15387.32193.854351.323371@grendel.zope.com> <200112151801.NAA02181@cj20424-a.reston1.va.home.com> Message-ID: <15387.39544.377965.395044@grendel.zope.com> Guido van Rossum writes: > I've found the cause: test_scope.py contains a call to > warnings.resetwarnings(), which removes *all* warning filters, not > just those made by the calling module. Because of the way This is the danger of running the tests the way we do (all in one process), but that also makes us be more careful with the tests. I'll have to run with -r more often! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From mal@lemburg.com Sat Dec 15 19:03:21 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 15 Dec 2001 20:03:21 +0100 Subject: [Python-Dev] Re: mpz module broken with GMP 4.0 References: <200112151623.LAA32158@cj20424-a.reston1.va.home.com> <0112151923261C.01556@arthur> <15387.38834.552687.60342@grendel.zope.com> Message-ID: <3C1B9E79.1020202@lemburg.com> Fred L. Drake, Jr. wrote: > Alex Martelli writes: > > Thanks for the indication -- I was not aware of the new GMP 4.0. A rapid > > test shows the current gmpy.c, release 0.8, breaks in exactly one place > > I'm adding a link to the gmpy website from the mpz module > documentation. You might also want to add a link to mxNumber which is another wrapper for GMP. Haven't tried it with GMP 4.0 yet, but it works great with 3.1.1 -- even on Windows. http://www.egenix.com/files/python/mxNumber.html -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From tim.one@home.com Sat Dec 15 20:29:15 2001 From: tim.one@home.com (Tim Peters) Date: Sat, 15 Dec 2001 15:29:15 -0500 Subject: [Python-Dev] regression tests with regrtest.py -r In-Reply-To: Message-ID: Here's one cause of current -r failures on Windows: C:\Code\python\PCbuild>python ../lib/test/regrtest.py \ test_pty test___all__ test_pty test test_pty skipped -- No module named termios test___all__ test test___all__ failed -- pty has no __all__ attribute 1 test failed: test___all__ 1 test skipped: test_pty Those skips are all expected on win32. It's "the usual": importing pty fails on Windows, but test_pty then leaves behind a bogus pty module object in sys.modules, which fools test___all__. Note that pty.py already has this atrocity from a previous round of -r wrestling: """ # Absurd: import termios and then delete it. This is to force an attempt # to import pty to raise an ImportError on platforms that lack termios. # Without this explicit import of termios here, some other module may # import tty first, which in turn imports termios and dies with an # ImportError then. But since tty *does* exist across platforms, that # leaves a damaged module object for tty in sys.modules, and the import # of tty here then appears to work despite that the tty imported is junk. import termios del termios import tty """ test___all__ also tries to compensate for pty (among others) insanity: """ def check_all(modname): names = {} try: exec "import %s" % modname in names except ImportError: # Silent fail here seems the best route since some modules # may not be available in all environments. # Since an ImportError may leave a partial module object in # sys.modules, get rid of that first. Here's what happens if # you don't: importing pty fails on Windows because pty tries to # import FCNTL, which doesn't exist. That raises an ImportError, # caught here. It also leaves a partial pty module in sys.modules. # So when test_pty is called later, the import of pty succeeds, # but shouldn't. As a result, test_pty crashes with an # AttributeError instead of an ImportError, and regrtest interprets # the latter as a test failure (ImportError is treated as "test # skipped" -- which is what test_pty should say on Windows). try: del sys.modules[modname] except KeyError: pass return """ I've added another hack to regrtest.py to worm around the test_pty followed by test___all__ case (more generally, any test_xxx followed by test___all__ case, when test_xxx fails due to leaving behind a damaged module object for xxx). However, that still doesn't fix test_rlcompleter followed by test___all__ on Windows. In that case, test_rlcompleter leaves behind a damaged rlcompleter module, because rlcompleter.py does exist on Windows (it fails due to the lack of the readline module). leaving-behind-incomplete-module-objects-still-seems-like-a-bad-idea- to-me-ly y'rs - tim From fdrake@acm.org Sat Dec 15 20:36:42 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Sat, 15 Dec 2001 15:36:42 -0500 Subject: [Python-Dev] Re: mpz module broken with GMP 4.0 In-Reply-To: <3C1B9E79.1020202@lemburg.com> References: <200112151623.LAA32158@cj20424-a.reston1.va.home.com> <0112151923261C.01556@arthur> <15387.38834.552687.60342@grendel.zope.com> <3C1B9E79.1020202@lemburg.com> Message-ID: <15387.46170.620220.286507@grendel.zope.com> M.-A. Lemburg writes: > You might also want to add a link to mxNumber which is another > wrapper for GMP. Haven't tried it with GMP 4.0 yet, but it works > great with 3.1.1 -- even on Windows. I'll have this checked in shortly -- thanks! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From fdrake@beowolf.digicool.com Sat Dec 15 20:44:16 2001 From: fdrake@beowolf.digicool.com (Fred Drake) Date: Sat, 15 Dec 2001 15:44:16 -0500 (EST) Subject: [Python-Dev] [maintenance doc updates] Message-ID: <20011215204416.B421F286BC@beowolf.digicool.com> The development version of the documentation has been updated: http://python.sourceforge.net/maint-docs/ Miscellaneous minor updates/fixes. From fdrake@acm.org Sat Dec 15 20:46:40 2001 From: fdrake@acm.org (Fred L. Drake) Date: Sat, 15 Dec 2001 15:46:40 -0500 (EST) Subject: [Python-Dev] [development doc updates] Message-ID: <20011215204640.8FF5E286BC@beowolf.digicool.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Miscellaneous minor updates/fixes post 2.2c1. From aleax@aleax.it Sat Dec 15 21:17:12 2001 From: aleax@aleax.it (Alex Martelli) Date: Sat, 15 Dec 2001 22:17:12 +0100 Subject: [Python-Dev] Re: mpz module broken with GMP 4.0 In-Reply-To: <200112151831.NAA02998@cj20424-a.reston1.va.home.com> References: <200112151623.LAA32158@cj20424-a.reston1.va.home.com> <0112151923261C.01556@arthur> <200112151831.NAA02998@cj20424-a.reston1.va.home.com> Message-ID: <0112152217121F.01556@arthur> On Saturday 15 December 2001 19:31, Guido van Rossum wrote: > Alex, do you have a suggestion for what to do with mpzmodule.c in the > 2.2 distribution? Remove it? It appears to me that there's no maintenance of it, and either Lemburg's module, or gmpy, would be a better bet for Pythonistas needing GMP access, anyway. Alex From tim.one@home.com Sat Dec 15 21:20:32 2001 From: tim.one@home.com (Tim Peters) Date: Sat, 15 Dec 2001 16:20:32 -0500 Subject: [Python-Dev] regrtest.py mystery Message-ID: We do this after running every test: # Unload the newly imported modules (best effort finalization) for module in sys.modules.keys(): if module not in save_modules and module.startswith("test."): test_support.unload(module) Unfortunately, that doesn't nuke whatever damaged non-test module objects may have been left behind. If I change it to make a truly best effort: if module not in save_modules: test_support.unload(module) then about a dozen std tests fail in mysterious ways, but only when running more than one test. Like just running test_email test_string causes test_string to die like so: python ../lib/test/regrtest.py -v test_email test_string ... test test_string crashed -- exceptions.AttributeError: 'NoneType' object has no attribute 'get' Traceback (most recent call last): File "../lib/test/regrtest.py", line 305, in runtest the_module = __import__(test, globals(), locals(), []) File "../lib/test\test_string.py", line 34, in ? string_tests.run_method_tests(test) File "../lib/test\string_tests.py", line 233, in run_method_tests verify('hello world'.encode('zlib') == data) File "C:\CODE\PYTHON\lib\encodings\__init__.py", line 43, in search_function entry = _cache.get(encoding,_unknown) AttributeError: 'NoneType' object has no attribute 'get' It seems generally the case that these failures are due to various module-level names getting rebound to None. A simpler example: python ../lib/test/regrtest.py -v test_string test_string ... test test_string crashed -- exceptions.AttributeError: 'NoneType' object has no attribute 'compress' Traceback (most recent call last): File "../lib/test/regrtest.py", line 305, in runtest the_module = __import__(test, globals(), locals(), []) File "../lib/test\test_string.py", line 34, in ? string_tests.run_method_tests(test) File "../lib/test\string_tests.py", line 233, in run_method_tests verify('hello world'.encode('zlib') == data) File "C:\CODE\PYTHON\lib\encodings\zlib_codec.py", line 25, in zlib_encode output = zlib.compress(input) AttributeError: 'NoneType' object has no attribute 'compress' What's up with that? So far, they all seem to involve the encodings directory ... From martin@v.loewis.de Sat Dec 15 22:36:39 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Sat, 15 Dec 2001 23:36:39 +0100 Subject: [Python-Dev] Re: mpz module broken with GMP 4.0 In-Reply-To: <0112152217121F.01556@arthur> (message from Alex Martelli on Sat, 15 Dec 2001 22:17:12 +0100) References: <200112151623.LAA32158@cj20424-a.reston1.va.home.com> <0112151923261C.01556@arthur> <200112151831.NAA02998@cj20424-a.reston1.va.home.com> <0112152217121F.01556@arthur> Message-ID: <200112152236.fBFMadK05615@mira.informatik.hu-berlin.de> > Remove it? I think this is unacceptable for 2.2, now that 2.2c1 carries the module. It seems that it still works fine as before with older versions of GMP, and it also seems that there are a few users out there of the module. If users care enough, somebody will contribute a patch. If they don't care, we can deprecate it for 2.3, and remove it in 2.4 (following the guidelines for language evolution). Regards, Martin From martin@v.loewis.de Sat Dec 15 22:52:17 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Sat, 15 Dec 2001 23:52:17 +0100 Subject: [Python-Dev] regrtest.py mystery In-Reply-To: References: Message-ID: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> > What's up with that? So far, they all seem to involve the encodings > directory ... Python/codecs.c holds search functions in _PyCodec_SearchPath, among them encodings.__init__.search_function. It also imports "encodings" the first time somebody invokes .encode/.decode on some kind of string, but doesn't hold onto the module. When the last reference to encodings.__init__ goes away, Python will clear all globals - but codecs still holds a reference to search_function, so that won't go away. Invoking search_function later will cause the problem you see. One solution would be to bind all globals used inside encodings.__init__.search_function to the search_function, e.g. by means of a class: class GlobalSearchFunction: _cache = _unknown = class CodecRegistryError(...): def search_function(self,encoding): codecs.register(GlobalSearchFunction().search_function) Of course, if this gets cleaned from sys.modules, nobody will be able to catch CodecRegistryError anymore (since re-importing the module will produce a different class). Regards, Martin From fdrake@acm.org Sat Dec 15 23:53:57 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Sat, 15 Dec 2001 18:53:57 -0500 Subject: [Python-Dev] regrtest.py mystery In-Reply-To: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> References: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> Message-ID: <15387.58005.329451.71260@grendel.zope.com> Martin v. Loewis writes: > Python/codecs.c holds search functions in _PyCodec_SearchPath, among > them encodings.__init__.search_function. It also imports "encodings" > the first time somebody invokes .encode/.decode on some kind of > string, but doesn't hold onto the module. Perhaps it should hold onto the module as well? That would avoid it getting cleaned up before possibly dependent modules. For the test case problem, perhaps there should be a list of exceptional modules that don't get cleaned up by regrtest once they appear. encodings and warnings would probably both need to be on the list. This list should probably be in the sys module; it would also be useful for the RollbackImporter in the unittestgui.py script that comes with the separate PyUnit package. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From guido@python.org Sun Dec 16 01:01:35 2001 From: guido@python.org (Guido van Rossum) Date: Sat, 15 Dec 2001 20:01:35 -0500 Subject: [Python-Dev] Re: mpz module broken with GMP 4.0 In-Reply-To: Your message of "Sat, 15 Dec 2001 23:36:39 +0100." <200112152236.fBFMadK05615@mira.informatik.hu-berlin.de> References: <200112151623.LAA32158@cj20424-a.reston1.va.home.com> <0112151923261C.01556@arthur> <200112151831.NAA02998@cj20424-a.reston1.va.home.com> <0112152217121F.01556@arthur> <200112152236.fBFMadK05615@mira.informatik.hu-berlin.de> Message-ID: <200112160101.UAA03490@cj20424-a.reston1.va.home.com> > > Remove it? > > I think this is unacceptable for 2.2, now that 2.2c1 carries the > module. It seems that it still works fine as before with older > versions of GMP, and it also seems that there are a few users out > there of the module. > > If users care enough, somebody will contribute a patch. If they don't > care, we can deprecate it for 2.3, and remove it in 2.4 (following the > guidelines for language evolution). Let's mark it as deprecated in 2.2, and remove it in 2.3. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@home.com Sun Dec 16 05:23:43 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 16 Dec 2001 00:23:43 -0500 Subject: [Python-Dev] regrtest.py mystery In-Reply-To: <15387.58005.329451.71260@grendel.zope.com> Message-ID: [Fred] > Perhaps it should hold onto the module as well? That would avoid it > getting cleaned up before possibly dependent modules. > For the test case problem, perhaps there should be a list of > exceptional modules that don't get cleaned up by regrtest once they > appear. encodings and warnings would probably both need to be on the > list. This list should probably be in the sys module; it would also > be useful for the RollbackImporter in the unittestgui.py script that > comes with the separate PyUnit package. I expect the problem is intractable. Here's a revealing experiment: add this block to the start of test___all__.check_all(): if modname in sys.modules: del sys.modules[modname] That is, just prior to test___all__ trying to import a module, remove *only* that module from sys.modules if it's already there. Sounds pretty harmless, right? But on Windows, it causes 28 standard tests to fail (possibly more on Linux): 28 tests failed: test___all__ test_difflib test_email test_fileinput test_fnmatch test_future test_gettext test_glob test_global test_htmllib test_htmlparser test_inspect test_mimetypes test_mmap test_os test_pyclbr test_re test_regex test_repr test_scope test_sre test_strftime test_strop test_sundry test_tokenize test_urllib2 test_xmllib test_xmlrpc Trying to figure which modules "hurt" turns out to be a nightmare. Here's the minimal set that allows the test suite to pass again on Windows, using the default test order: if modname in sys.modules: if modname not in ( "copy_reg", "os", "sgmllib", "sre", "tempfile", "tokenize", "warnings", ): del sys.modules[modname] Am I gonna check that in? Fat chance . From tim.one@home.com Sun Dec 16 05:42:33 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 16 Dec 2001 00:42:33 -0500 Subject: [Python-Dev] regression tests with regrtest.py -r In-Reply-To: Message-ID: [Tim] > ordering that caused it. Heh -- -r failed in test___all__ while I was > typing this, but I've no idea why. I'll pursue it until I get > sick to death of it . In case anyone's waiting, stop -- I got sick to death of it, and won't be checking in anything related. (BTW, there is no test_rlcompleter, of course; it's the combo of test_sundry and test___all__ that screws up (both of those import rlcompleter)) From martin@v.loewis.de Sun Dec 16 09:38:34 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Sun, 16 Dec 2001 10:38:34 +0100 Subject: [Python-Dev] regrtest.py mystery In-Reply-To: <15387.58005.329451.71260@grendel.zope.com> (fdrake@acm.org) References: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> <15387.58005.329451.71260@grendel.zope.com> Message-ID: <200112160938.fBG9cYV01682@mira.informatik.hu-berlin.de> > > Python/codecs.c holds search functions in _PyCodec_SearchPath, among > > them encodings.__init__.search_function. It also imports "encodings" > > the first time somebody invokes .encode/.decode on some kind of > > string, but doesn't hold onto the module. > > Perhaps it should hold onto the module as well? That would avoid it > getting cleaned up before possibly dependent modules. Re-importing encodings would still do funny things, then: It would register itself once more, resulting in two search functions being registered. codecs would hold one of the encodings modules, but not the other. So if the second search function is ever invoked, you get the same error. Regards, Martin From skip@pobox.com (Skip Montanaro) Sun Dec 16 15:11:31 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Sun, 16 Dec 2001 09:11:31 -0600 Subject: [Python-Dev] regression tests with regrtest.py -r In-Reply-To: References: Message-ID: <15388.47523.871195.694764@12-248-41-177.client.attbi.com> Tim> In case anyone's waiting, stop -- I got sick to death of it, and Tim> won't be checking in anything related. (BTW, there is no Tim> test_rlcompleter, of course; it's the combo of test_sundry and Tim> test___all__ that screws up (both of those import rlcompleter)) A couple wild-ass-guess ideas: * Perhaps more test___all__ smarts are needed? You added a "known skips" list to regrtest.py. test___all__ could perhaps avoid importing any module that's not supposed to work on the current platform. * How about comparing sys.modules.keys() before and after a test and then remove any that were added that are Python modules? (It's not safe to remove extension modules, right?) calling-it-intractable-means-someone-will-solve-it-ly, y'rs, Skip From simon@simon-cozens.org Sun Dec 16 21:14:05 2001 From: simon@simon-cozens.org (Simon Cozens) Date: Sun, 16 Dec 2001 21:14:05 +0000 Subject: [Python-Dev] First cut of pyc2pasm released Message-ID: <20011216211405.GA16141@netthink.co.uk> I hope you do not mind another brief Parroty interlude. If you're sick of that kind of thing, I apologise, I'm not trying to poach your time, so excuse me. But! With the 0.0.3 release of Parrot list week, we've got sufficiently powerful that it's sensible to think about running other languages' bytecode on the Parrot VM. (Converting the bytecode alone saves the hassle of writing a compiler, which is the difficult bit.) So naturally, I started with Perl. I think that lasted for about half an hour before I ran away screaming. Don't get me wrong, I like Perl, I just wouldn't like to turn its op tree into anything sensible. So I thought I'd show the buggers. Python's bytecode is nice and straightforward, and dis.py turns it into relatively simple assembly. The operations that it needs, Parrot provides, even things like set_lineno and rot_four. It was only a matter of time before I could disassemble Python bytecode and put it together into Parrot assembly. http://archive.develooper.com/perl6-internals@perl.org/msg07040.html is about as far as I got. It works. It runs "Hello World" and things slightly more complex, and it groks simple variable manipulation. It wouldn't be much hassle to add control structures. As a slight disclaimer, the variable classes [1] it uses - PythonInteger and so on - don't actually exist, but you can fake it by changing the outputted assembler code to use the ones that Parrot provides. I'm going to work on providing Python types in the core Real Soon Now, tuit permitting. Anyway, there it is. Enjoy it. If you want to work on it, feel free; if you want to take it and rewrite it in Python, then I'd be honored. Whatever, it's out there, and we're serious about making Parrot work for Python. (Or is it the other way around..? ;) Simon [1] One of Parrot's internal data types is the Parrot Magic Cookie, something which is not a million miles distant from a PyObject. You define PMCs by filling in a standard structure of manipulator methods, just like in Objects/*object.c. -- Hubris is when you really do have it, enough so only the gods slap you down. Pretentiousness is when you don't have it, and everyone slaps you down. Arrogance is somewhere in between. - Thorfinn From mal@lemburg.com Mon Dec 17 10:56:14 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 17 Dec 2001 11:56:14 +0100 Subject: [Python-Dev] regrtest.py mystery References: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> <15387.58005.329451.71260@grendel.zope.com> <200112160938.fBG9cYV01682@mira.informatik.hu-berlin.de> Message-ID: <3C1DCF4E.193D27AB@lemburg.com> "Martin v. Loewis" wrote: > > > > Python/codecs.c holds search functions in _PyCodec_SearchPath, among > > > them encodings.__init__.search_function. It also imports "encodings" > > > the first time somebody invokes .encode/.decode on some kind of > > > string, but doesn't hold onto the module. > > > > Perhaps it should hold onto the module as well? That would avoid it > > getting cleaned up before possibly dependent modules. > > Re-importing encodings would still do funny things, then: It would > register itself once more, resulting in two search functions being > registered. codecs would hold one of the encodings modules, but not > the other. So if the second search function is ever invoked, you get > the same error. Rather than "fixing" the encodings package, I'd rather suggest to fix the unload procedure in the test suite. Some modules simply should not be unloaded because it is known that the interpreter needs them for internal purposes. In the case of the encodings package it is not only the search function that gets kept alive, it's also the codecs themselves which are cached by the implementation in _codecs.c. How about following Fred's suggestion to run each test in a separate process (possibly as an option to regrtest.py) ?! -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Mon Dec 17 14:08:31 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 17 Dec 2001 09:08:31 -0500 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Modules xxsubtype.c,2.9,2.10 In-Reply-To: Your message of "Mon, 17 Dec 2001 10:39:31 +0100." <3C1DBD53.2672A1C@lemburg.com> References: <3C1DBD53.2672A1C@lemburg.com> Message-ID: <200112171408.JAA16669@cj20424-a.reston1.va.home.com> [Tim] > > Log Message: > > David Abrahams tried to compile this as a separate DLL under MSVC, and > > got a barrage of compile errors that didn't make sense to the C++ brain: > > MSVC does not allow C (but does allow C++) initializers to contain > > data addresses supplied by other DLLs. So changed the initializers here > > to use dummy nulls, and changed module init to plug in the foreign > > addresses at runtime (manually simulating what C++ does by magic). Tested > > on Windows, and Guido tested on Linux (thanks!). BTW, the *point* is that > > people are going to use this module as a template for writing their own > > subtypes, and it's unusual for extension authors to build their extensions > > into Python directly (separate DLLs are the norm on Windows); so it's > > better if we give them a template that works . [MAL] > This is a common problem when compiling Windows extensions. Perhaps > we should provide some more generic way to help the extension > writers here ?! FWIW, I'm using this macro for doing the deferred > type init: > > #define PyType_Init(x) \ > { \ > x.ob_type = &PyType_Type; \ > } Hm, alternatively, we could put a statement in PyType_Ready() that initializes ob_type if it's NULL -- then we'd encourage people to call PyType_Ready(), which is a good idea anyway. --Guido van Rossum (home page: http://www.python.org/~guido/) From thomas.heller@ion-tof.com Mon Dec 17 14:23:06 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Mon, 17 Dec 2001 15:23:06 +0100 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Modules xxsubtype.c,2.9,2.10 References: <3C1DBD53.2672A1C@lemburg.com> <200112171408.JAA16669@cj20424-a.reston1.va.home.com> Message-ID: <021401c18706$6aacaf20$e000a8c0@thomasnotebook> > Hm, alternatively, we could put a statement in PyType_Ready() that > initializes ob_type if it's NULL -- then we'd encourage people to call > PyType_Ready(), which is a good idea anyway. > > --Guido van Rossum (home page: http://www.python.org/~guido/) +2 ;-) Thomas From fdrake@acm.org Mon Dec 17 14:26:49 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 17 Dec 2001 09:26:49 -0500 (EST) Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Modules xxsubtype.c,2.9,2.10 In-Reply-To: <200112171408.JAA16669@cj20424-a.reston1.va.home.com> References: <3C1DBD53.2672A1C@lemburg.com> <200112171408.JAA16669@cj20424-a.reston1.va.home.com> Message-ID: <15390.169.62909.589215@cj42289-a.reston1.va.home.com> Guido van Rossum writes: > Hm, alternatively, we could put a statement in PyType_Ready() that > initializes ob_type if it's NULL -- then we'd encourage people to call > PyType_Ready(), which is a good idea anyway. This would be very good. I'll second Thomas's +2. ;-) -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From fdrake@acm.org Mon Dec 17 14:32:00 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 17 Dec 2001 09:32:00 -0500 (EST) Subject: [Python-Dev] regrtest.py mystery In-Reply-To: <3C1DCF4E.193D27AB@lemburg.com> References: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> <15387.58005.329451.71260@grendel.zope.com> <200112160938.fBG9cYV01682@mira.informatik.hu-berlin.de> <3C1DCF4E.193D27AB@lemburg.com> Message-ID: <15390.480.513981.132090@cj42289-a.reston1.va.home.com> M.-A. Lemburg writes: > How about following Fred's suggestion to run each test in a separate > process (possibly as an option to regrtest.py) ?! I wasn't really suggesting it, but it would solve this particular problem. Note that the effectiveness of this to solve this particular problem depends on the set of modules regrtest itself imports; if it imports something and catches an ImportError, using a simple fork() to create the child is not sufficient. Using fork()/exec() would be extremely slow and should be avoided if we can. Frankly, I like that we find interactions between the tests, so I'm not convinced that there are any real problems with the test framework. I do use to a C unit test package that uses fork() to make sure individual tests can't core the framework, but that's not what we're protecting against here. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From mwh@python.net Mon Dec 17 14:36:59 2001 From: mwh@python.net (Michael Hudson) Date: 17 Dec 2001 14:36:59 +0000 Subject: [Python-Dev] regrtest.py mystery In-Reply-To: "Fred L. Drake, Jr."'s message of "Mon, 17 Dec 2001 09:32:00 -0500 (EST)" References: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> <15387.58005.329451.71260@grendel.zope.com> <200112160938.fBG9cYV01682@mira.informatik.hu-berlin.de> <3C1DCF4E.193D27AB@lemburg.com> <15390.480.513981.132090@cj42289-a.reston1.va.home.com> Message-ID: <2mbsgxlxpw.fsf@starship.python.net> "Fred L. Drake, Jr." writes: > M.-A. Lemburg writes: > > How about following Fred's suggestion to run each test in a separate > > process (possibly as an option to regrtest.py) ?! > > I wasn't really suggesting it, but it would solve this particular > problem. Note that the effectiveness of this to solve this particular > problem depends on the set of modules regrtest itself imports; if it > imports something and catches an ImportError, using a simple fork() to > create the child is not sufficient. Using fork()/exec() would be > extremely slow and should be avoided if we can. And I think you'd have fun getting anything like that to run on MacOS before OSX, and probably other oddball platforms (Windows at least has the os.spawn* functions). Making the test suite less portable is /unlikely/ to be a win, I'd have thought. Cheers, M. -- People think I'm a nice guy, and the fact is that I'm a scheming, conniving bastard who doesn't care for any hurt feelings or lost hours of work if it just results in what I consider to be a better system. -- Linus Torvalds From barry@zope.com Mon Dec 17 15:45:36 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 17 Dec 2001 10:45:36 -0500 Subject: [Python-Dev] regrtest.py mystery References: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> <15387.58005.329451.71260@grendel.zope.com> <200112160938.fBG9cYV01682@mira.informatik.hu-berlin.de> <3C1DCF4E.193D27AB@lemburg.com> Message-ID: <15390.4896.825987.759927@anthem.wooz.org> >>>>> "M" == M writes: M> Rather than "fixing" the encodings package, I'd rather suggest M> to fix the unload procedure in the test suite. Featurette request for Python 2.3: a builtin unload() which Does The Right Thing to unload a module, or raise an exception if the module cannot be unloaded. Lots and lots of applications would find this useful. -Barry From fdrake@acm.org Mon Dec 17 15:54:51 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 17 Dec 2001 10:54:51 -0500 (EST) Subject: [Python-Dev] regrtest.py mystery In-Reply-To: <15390.4896.825987.759927@anthem.wooz.org> References: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> <15387.58005.329451.71260@grendel.zope.com> <200112160938.fBG9cYV01682@mira.informatik.hu-berlin.de> <3C1DCF4E.193D27AB@lemburg.com> <15390.4896.825987.759927@anthem.wooz.org> Message-ID: <15390.5451.781522.718432@cj42289-a.reston1.va.home.com> Barry A. Warsaw writes: > Featurette request for Python 2.3: a builtin unload() which Does The > Right Thing to unload a module, or raise an exception if the module > cannot be unloaded. Lots and lots of applications would find this > useful. Useful or not, isn't the basic problem that we haven't been able to define The Right Thing? I'd certainly expect a module to be able to declare in some way that it could not be unloaded, or control how it gets unloaded. Perhaps allow Python modules to define an __unload__() function; when unload() is called on a module, the __unload__() is called; if that raises an exception, it gets propogated, otherwise the unload continues if the module still exists in sys.modules. Not clear that this would be useful for extensions. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From paul@pfdubois.com Mon Dec 17 16:09:59 2001 From: paul@pfdubois.com (Paul F. Dubois) Date: Mon, 17 Dec 2001 08:09:59 -0800 Subject: [Python-Dev] RE: [Numpy-discussion] Python 2.2 In-Reply-To: <006201c186e7$da0d32f0$df13000a@RDREWXP> Message-ID: <000001c18715$477816c0$0c01a8c0@freedom> I downloaded and installed 2.2c1 and built the Numeric installers. However, when I run them they all fail when the installation begins (after one clicks the final click to install) with an access violation. I removed any previous Numeric and it still happened. Building and installing with setup.py install works ok. -----Original Message----- From: numpy-discussion-admin@lists.sourceforge.net [mailto:numpy-discussion-admin@lists.sourceforge.net] On Behalf Of Ray Drew Sent: Monday, December 17, 2001 2:45 AM To: numpy-discussion@lists.sourceforge.net Subject: [Numpy-discussion] Python 2.2 Does anyone know when a release for Python 2.2 on Windows will be available? regards, Ray Drew _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From guido@python.org Mon Dec 17 16:12:54 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 17 Dec 2001 11:12:54 -0500 Subject: [Python-Dev] regrtest.py mystery In-Reply-To: Your message of "Mon, 17 Dec 2001 10:45:36 EST." <15390.4896.825987.759927@anthem.wooz.org> References: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> <15387.58005.329451.71260@grendel.zope.com> <200112160938.fBG9cYV01682@mira.informatik.hu-berlin.de> <3C1DCF4E.193D27AB@lemburg.com> <15390.4896.825987.759927@anthem.wooz.org> Message-ID: <200112171612.LAA01013@cj20424-a.reston1.va.home.com> > Featurette request for Python 2.3: a builtin unload() which Does The > Right Thing to unload a module, or raise an exception if the module > cannot be unloaded. Lots and lots of applications would find this > useful. I haven't been following this discussion. Can you give a clear definition of The Right Thing? --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Mon Dec 17 16:18:42 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 17 Dec 2001 11:18:42 -0500 Subject: [Python-Dev] regrtest.py mystery References: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> <15387.58005.329451.71260@grendel.zope.com> <200112160938.fBG9cYV01682@mira.informatik.hu-berlin.de> <3C1DCF4E.193D27AB@lemburg.com> <15390.4896.825987.759927@anthem.wooz.org> <15390.5451.781522.718432@cj42289-a.reston1.va.home.com> Message-ID: <15390.6882.514689.285195@anthem.wooz.org> >>>>> "Fred" == Fred L Drake, Jr writes: Fred> Useful or not, isn't the basic problem that we haven't Fred> been able to define The Right Thing? Probably so. Fred> I'd certainly expect a module to be able to declare in some Fred> way that it could not be unloaded, or control how it gets Fred> unloaded. Perhaps allow Python modules to define an Fred> __unload__() function; when unload() is called on a module, Fred> the __unload__() is called; if that raises an exception, it Fred> gets propogated, otherwise the unload continues if the Fred> module still exists in sys.modules. Not clear that this Fred> would be useful for extensions. The "unload protocol" would be I think. I.e. extension modules ought to be unloadable if they can cooperate. From the Linux dl*() manpages: dlclose decrements the reference count on the dynamic library handle handle. If the reference count drops to zero and no other loaded libraries use symbols in it, then the dynamic library is unloaded. If the dynamic library exports a routine named _fini, then that routine is called just before the library is unloaded. season-to-tasste-on-your-favorite-OS-bread-l y'rs, -Barry From guido@python.org Mon Dec 17 16:16:27 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 17 Dec 2001 11:16:27 -0500 Subject: [Python-Dev] RE: [Numpy-discussion] Python 2.2 In-Reply-To: Your message of "Mon, 17 Dec 2001 08:09:59 PST." <000001c18715$477816c0$0c01a8c0@freedom> References: <000001c18715$477816c0$0c01a8c0@freedom> Message-ID: <200112171616.LAA04194@cj20424-a.reston1.va.home.com> > I downloaded and installed 2.2c1 and built the Numeric installers. > However, when I run them they all fail when the installation begins > (after one clicks the final click to install) with an access violation. > I removed any previous Numeric and it still happened. > > Building and installing with setup.py install works ok. I don't know anything about your installers, but could it be that you were trying to install without Administrator permissions? That used to crash the previous Python installer too. (The new one doesn't, but it's a commercial product so we can't share it.) --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Mon Dec 17 16:21:41 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 17 Dec 2001 11:21:41 -0500 Subject: [Python-Dev] regrtest.py mystery References: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> <15387.58005.329451.71260@grendel.zope.com> <200112160938.fBG9cYV01682@mira.informatik.hu-berlin.de> <3C1DCF4E.193D27AB@lemburg.com> <15390.4896.825987.759927@anthem.wooz.org> <200112171612.LAA01013@cj20424-a.reston1.va.home.com> Message-ID: <15390.7061.122528.163470@anthem.wooz.org> >>>>> "GvR" == Guido van Rossum writes: GvR> I haven't been following this discussion. Can you give a GvR> clear definition of The Right Thing? I think we need to define the goals of unloading. From mwh@python.net Mon Dec 17 16:25:54 2001 From: mwh@python.net (Michael Hudson) Date: 17 Dec 2001 16:25:54 +0000 Subject: [Python-Dev] regrtest.py mystery In-Reply-To: barry@zope.com's message of "Mon, 17 Dec 2001 11:21:41 -0500" References: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> <15387.58005.329451.71260@grendel.zope.com> <200112160938.fBG9cYV01682@mira.informatik.hu-berlin.de> <3C1DCF4E.193D27AB@lemburg.com> <15390.4896.825987.759927@anthem.wooz.org> <200112171612.LAA01013@cj20424-a.reston1.va.home.com> <15390.7061.122528.163470@anthem.wooz.org> Message-ID: <2mheqpke3x.fsf@starship.python.net> barry@zope.com (Barry A. Warsaw) writes: > >>>>> "GvR" == Guido van Rossum writes: > > GvR> I haven't been following this discussion. Can you give a > GvR> clear definition of The Right Thing? > > I think we need to define the goals of unloading. Doesn't that make it into a solution in search of a problem? When is unloading useful? I'm not being facetious, I've just never been hit by the need for it. Cheers, M. -- Q: Isn't it okay to just read Slashdot for the links? A: No. Reading Slashdot for the links is like having "just one hit" off the crack pipe. -- http://www.cs.washington.edu/homes/klee/misc/slashdot.html#faq From guido@python.org Mon Dec 17 16:31:17 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 17 Dec 2001 11:31:17 -0500 Subject: [Python-Dev] regrtest.py mystery In-Reply-To: Your message of "Mon, 17 Dec 2001 11:21:41 EST." <15390.7061.122528.163470@anthem.wooz.org> References: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> <15387.58005.329451.71260@grendel.zope.com> <200112160938.fBG9cYV01682@mira.informatik.hu-berlin.de> <3C1DCF4E.193D27AB@lemburg.com> <15390.4896.825987.759927@anthem.wooz.org> <200112171612.LAA01013@cj20424-a.reston1.va.home.com> <15390.7061.122528.163470@anthem.wooz.org> Message-ID: <200112171631.LAA15524@cj20424-a.reston1.va.home.com> > >>>>> "GvR" == Guido van Rossum writes: > > GvR> I haven't been following this discussion. Can you give a > GvR> clear definition of The Right Thing? > > I think we need to define the goals of unloading. Then how can you call it a featurette? Who knows if it's even a tractable problem? --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Mon Dec 17 16:46:02 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 17 Dec 2001 11:46:02 -0500 Subject: [Python-Dev] regrtest.py mystery References: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> <15387.58005.329451.71260@grendel.zope.com> <200112160938.fBG9cYV01682@mira.informatik.hu-berlin.de> <3C1DCF4E.193D27AB@lemburg.com> <15390.4896.825987.759927@anthem.wooz.org> <200112171612.LAA01013@cj20424-a.reston1.va.home.com> <15390.7061.122528.163470@anthem.wooz.org> <200112171631.LAA15524@cj20424-a.reston1.va.home.com> Message-ID: <15390.8522.971549.671038@anthem.wooz.org> >>>>> "GvR" == Guido van Rossum writes: GvR> Then how can you call it a featurette? Who knows if it's GvR> even a tractable problem? Okay, so it's a PEPable feature. -Barry From thomas.heller@ion-tof.com Mon Dec 17 16:49:54 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Mon, 17 Dec 2001 17:49:54 +0100 Subject: [Python-Dev] RE: [Numpy-discussion] Python 2.2 References: <000001c18715$477816c0$0c01a8c0@freedom> Message-ID: <028501c1871a$ec399260$e000a8c0@thomasnotebook> > I downloaded and installed 2.2c1 and built the Numeric installers. > However, when I run them they all fail when the installation begins > (after one clicks the final click to install) with an access violation. > I removed any previous Numeric and it still happened. > > Building and installing with setup.py install works ok. > Are you talking about bdist_wininst installers? I just could reproduce a problem with them (after renaming away all my zip.exe programs in the PATH). Thomas From guido@python.org Mon Dec 17 16:52:21 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 17 Dec 2001 11:52:21 -0500 Subject: [Python-Dev] regrtest.py mystery In-Reply-To: Your message of "Mon, 17 Dec 2001 11:46:02 EST." <15390.8522.971549.671038@anthem.wooz.org> References: <200112152252.fBFMqH205644@mira.informatik.hu-berlin.de> <15387.58005.329451.71260@grendel.zope.com> <200112160938.fBG9cYV01682@mira.informatik.hu-berlin.de> <3C1DCF4E.193D27AB@lemburg.com> <15390.4896.825987.759927@anthem.wooz.org> <200112171612.LAA01013@cj20424-a.reston1.va.home.com> <15390.7061.122528.163470@anthem.wooz.org> <200112171631.LAA15524@cj20424-a.reston1.va.home.com> <15390.8522.971549.671038@anthem.wooz.org> Message-ID: <200112171652.LAA15707@cj20424-a.reston1.va.home.com> > Okay, so it's a PEPable feature. Not until you've thought about what it should do. I bet you lunch that you can't come up with a working definition. --Guido van Rossum (home page: http://www.python.org/~guido/) From paul@pfdubois.com Mon Dec 17 17:24:14 2001 From: paul@pfdubois.com (Paul F. Dubois) Date: Mon, 17 Dec 2001 09:24:14 -0800 Subject: [Python-Dev] RE: [Numpy-discussion] Python 2.2 In-Reply-To: <200112171616.LAA04194@cj20424-a.reston1.va.home.com> Message-ID: <000201c1871f$a6f514e0$0c01a8c0@freedom> I am on Windows Me, so there is no concept of administrator. -----Original Message----- From: numpy-discussion-admin@lists.sourceforge.net [mailto:numpy-discussion-admin@lists.sourceforge.net] On Behalf Of Guido van Rossum Sent: Monday, December 17, 2001 8:16 AM To: Paul F. Dubois Cc: 'Ray Drew'; numpy-discussion@lists.sourceforge.net; python-dev@python.org Subject: Re: [Python-Dev] RE: [Numpy-discussion] Python 2.2 > I downloaded and installed 2.2c1 and built the Numeric installers. > However, when I run them they all fail when the installation begins > (after one clicks the final click to install) with an access > violation. I removed any previous Numeric and it still happened. > > Building and installing with setup.py install works ok. I don't know anything about your installers, but could it be that you were trying to install without Administrator permissions? That used to crash the previous Python installer too. (The new one doesn't, but it's a commercial product so we can't share it.) --Guido van Rossum (home page: http://www.python.org/~guido/) _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From jeremy@zope.com Mon Dec 17 18:24:26 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Mon, 17 Dec 2001 13:24:26 -0500 (EST) Subject: [Python-Dev] compiler package udpate Message-ID: <15390.14426.296516.102494@slothrop.digicool.com> I'd like to backport changes from the 2.2 compiler package (Lib/compiler) to the 2.1.2 version (Tools/compiler/compiler). The changes are bug fixes, but include some non-trivial changes because the old code has some non-trivial bugs. The chief problem is that it underestimates the stack depth, which leads to core dumps. Any objections to me making these changes today? Jeremy From tim.one@home.com Mon Dec 17 19:38:48 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 17 Dec 2001 14:38:48 -0500 Subject: [Python-Dev] regrtest.py mystery In-Reply-To: <3C1DCF4E.193D27AB@lemburg.com> Message-ID: Without diving into the unloading pit, the endless problems with -r on non-Unix boxes would go away if a module that failed to import didn't leave behind a damaged module object in sys.modules ("damaged" == it may not fulfill some or any of the module's contracts, and it's impossible to guess which ones it won't, or to guess how the unfufilled ones may fail). I'm tempted to add a pile of Windows-specific modules that import each other in incestuous ways, so everyone can enjoy this chronic pain . "A fix" seems to amount to treating import M as if it were try: import M except: if M in sys.modules: del sys.modules[M] raise How would that hurt? From ping@lfw.org Mon Dec 17 21:05:12 2001 From: ping@lfw.org (Ka-Ping Yee) Date: Mon, 17 Dec 2001 15:05:12 -0600 (CST) Subject: [Python-Dev] regrtest.py mystery In-Reply-To: Message-ID: On Mon, 17 Dec 2001, Tim Peters wrote: > "A fix" seems to amount to treating > > import M > > as if it were > > try: > import M > except: > if M in sys.modules: > del sys.modules[M] > raise I remember suggesting exactly that some time ago (i think it was motivated at the time by the extreme pain that broken modules were causing for webserver-mode pydoc in its attempt to update loaded modules if the source files had changed on disk). Guido rejected it because you can't guarantee that the refcount on M is 1 at the point where you attempt to 'del sys.modules[M]' above. (For example, some other module imported by M could have imported M again, and so hold a reference to it while M is running its startup commands. This is why the entry is added to sys.modules before the body of M starts to run.) He deemed the situation where M is loaded-but-missing-from-sys.modules to be even worse than for M to be left loaded-but-broken therein. If you allow M to stay in sys.modules, then you can at least maintain the guarantee that there is a one-to-one association between loaded module names and loaded module objects. If you remove M from sys.modules but it lingers in memory, referenced elsewhere, you lose even that -- there can be many modules loaded all with the same name and it's a nightmare. The argument is compelling and i am forced to agree, but i still think that we should look for a better solution. I'll ponder this and post any ideas i come up with. -- ?!ng From mal@lemburg.com Mon Dec 17 21:25:56 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 17 Dec 2001 22:25:56 +0100 Subject: [Python-Dev] Unloading modules (regrtest.py mystery) References: Message-ID: <3C1E62E4.C4E9A7A0@lemburg.com> Ka-Ping Yee wrote: > > On Mon, 17 Dec 2001, Tim Peters wrote: > > "A fix" seems to amount to treating > > > > import M > > > > as if it were > > > > try: > > import M > > except: > > if M in sys.modules: > > del sys.modules[M] > > raise > > I remember suggesting exactly that some time ago (i think it was > motivated at the time by the extreme pain that broken modules were > causing for webserver-mode pydoc in its attempt to update loaded > modules if the source files had changed on disk). > > Guido rejected it because you can't guarantee that the refcount on M > is 1 at the point where you attempt to 'del sys.modules[M]' above. > (For example, some other module imported by M could have imported M > again, and so hold a reference to it while M is running its startup > commands. This is why the entry is added to sys.modules before the > body of M starts to run.) > > He deemed the situation where M is loaded-but-missing-from-sys.modules > to be even worse than for M to be left loaded-but-broken therein. If > you allow M to stay in sys.modules, then you can at least maintain the > guarantee that there is a one-to-one association between loaded module > names and loaded module objects. If you remove M from sys.modules but > it lingers in memory, referenced elsewhere, you lose even that -- there > can be many modules loaded all with the same name and it's a nightmare. > > The argument is compelling and i am forced to agree, but i still think > that we should look for a better solution. I'll ponder this and post > any ideas i come up with. How about only deleting it from sys.modules iff the ref count is 1 ?! Note that you cannot currently remove extension modules that have failed to import correctly. Would be nice if we could come up with a reasonable way of unloading modules, but I guess this requires some hard PEP work... -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From sdm7g@Virginia.EDU Mon Dec 17 21:40:01 2001 From: sdm7g@Virginia.EDU (Steven Majewski) Date: Mon, 17 Dec 2001 16:40:01 -0500 (EST) Subject: [Python-Dev] regrtest.py mystery In-Reply-To: <2mheqpke3x.fsf@starship.python.net> Message-ID: On 17 Dec 2001, Michael Hudson wrote: > barry@zope.com (Barry A. Warsaw) writes: > > > >>>>> "GvR" == Guido van Rossum writes: > > > > GvR> I haven't been following this discussion. Can you give a > > GvR> clear definition of The Right Thing? > > > > I think we need to define the goals of unloading. > > Doesn't that make it into a solution in search of a problem? > > When is unloading useful? I'm not being facetious, I've just never > been hit by the need for it. Shared libraries in XlispStat support unloading. But all of the times I've ever used it, I've used it as a 'reload' : usually while debugging shared libs -- it's handy to keep the interpreter window running while you fix and recompile a bug and load it again. The only case I can thing of where I might use it in production code would also be for a reload, where you might have several alternate 'plugin' modules. When you do an unload, references to objects that were in the shared library are replaced by references to a stub that generates an error/exception. ( That stub routine is part of the builtin dynamic load "module" . ) It doesn't involve traceing all the references -- the pointers are unchanged, but the object is changed. I have no idea what 'the Right Thing' for Python would be here -- I'm not quite clear on what you're trying to fix. I'm just volunteering an example. -- Steve Majewski From sdm7g@Virginia.EDU Mon Dec 17 22:19:13 2001 From: sdm7g@Virginia.EDU (Steven Majewski) Date: Mon, 17 Dec 2001 17:19:13 -0500 (EST) Subject: [Python-Dev] regrtest.py mystery In-Reply-To: Message-ID: On Mon, 17 Dec 2001, Steven Majewski wrote: > I have no idea what 'the Right Thing' for Python would be here -- > I'm not quite clear on what you're trying to fix. I'm just > volunteering an example. However, I will add that if I *did* need an unload for anything, I would think the XlispStat semantics: unloading the module immediately and making any dangling references into exceptions, or replacing them with references to a reloaded module, would seem to be more useful than not really unloading it until all of the references are decremented (if ever) and then unloading. I can't think of a case where I would want to unload a module, where I would not want a dangling reference to something in that module to generate an exception. -- Steve From guido@python.org Mon Dec 17 22:24:31 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 17 Dec 2001 17:24:31 -0500 Subject: [Python-Dev] regrtest.py mystery In-Reply-To: Your message of "Mon, 17 Dec 2001 17:19:13 EST." References: Message-ID: <200112172224.RAA19426@cj20424-a.reston1.va.home.com> > However, I will add that if I *did* need an unload for anything, > I would think the XlispStat semantics: unloading the module > immediately and making any dangling references into exceptions, > or replacing them with references to a reloaded module, > would seem to be more useful than not really unloading it until > all of the references are decremented (if ever) and then > unloading. > > I can't think of a case where I would want to unload a module, > where I would not want a dangling reference to something in that > module to generate an exception. Python doesn't let you replace a reference to an object with something else -- except in special cases (e.g. with explicit weak refs) there's no way to know where references to an object might exist. So that's not a useful idea, alas. --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Mon Dec 17 22:37:53 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 17 Dec 2001 17:37:53 -0500 (EST) Subject: [Python-Dev] regrtest.py mystery In-Reply-To: <200112172224.RAA19426@cj20424-a.reston1.va.home.com> References: <200112172224.RAA19426@cj20424-a.reston1.va.home.com> Message-ID: <15390.29633.527008.217751@cj42289-a.reston1.va.home.com> Guido van Rossum writes: > So that's not a useful idea, alas. Well, it may be a useful idea, but difficult & impractical to implement in Python. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From greg@cosc.canterbury.ac.nz Mon Dec 17 22:47:06 2001 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Dec 2001 11:47:06 +1300 (NZDT) Subject: [Python-Dev] regrtest.py mystery In-Reply-To: Message-ID: <200112172247.LAA17572@s454.cosc.canterbury.ac.nz> Ka-Ping Yee : > He deemed the situation where M is loaded-but-missing-from-sys.modules > to be even worse than for M to be left loaded-but-broken therein. Maybe modules should have a this-module-is-broken flag which is set when the module fails to import properly, and which causes an exception on any further attempt to reference a name in it. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From fredrik@pythonware.com Mon Dec 17 23:14:43 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Tue, 18 Dec 2001 00:14:43 +0100 Subject: [Python-Dev] regrtest.py mystery References: <200112172247.LAA17572@s454.cosc.canterbury.ac.nz> Message-ID: <164201c18750$9ef89830$ced241d5@hagrid> greg wrote: > > > He deemed the situation where M is loaded-but-missing-from-sys.modules > > to be even worse than for M to be left loaded-but-broken therein. > > Maybe modules should have a this-module-is-broken flag which > is set when the module fails to import properly, and which > causes an exception on any further attempt to reference a > name in it. how do you detect that a module is successfully imported? the only way I can think of is "execution reached the end of the module", which would probably break considerable amounts of code that rely on "temporarily recursive" imports. From greg@cosc.canterbury.ac.nz Mon Dec 17 23:23:53 2001 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Dec 2001 12:23:53 +1300 (NZDT) Subject: [Python-Dev] regrtest.py mystery In-Reply-To: <164201c18750$9ef89830$ced241d5@hagrid> Message-ID: <200112172323.MAA17577@s454.cosc.canterbury.ac.nz> Fredrik Lundh : > how do you detect that a module is successfully imported? > > the only way I can think of is "execution reached the end of > the module", which would probably break considerable amounts > of code that rely on "temporarily recursive" imports. If an exception propagates back through an import statement, the module that it's importing is marked as broken. This may result in a single error causing more than one module to be marked as broken, but I think that's the right thing to do anyway. E.g. if you import module A which imports module B which fails, you can't be sure that either A or B is fit to use, so both should be marked as broken. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tim.one@home.com Mon Dec 17 23:36:22 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 17 Dec 2001 18:36:22 -0500 Subject: [Python-Dev] regrtest.py mystery In-Reply-To: <164201c18750$9ef89830$ced241d5@hagrid> Message-ID: [/F] > how do you detect that a module is successfully imported? > > the only way I can think of is "execution reached the end of > the module", which would probably break considerable amounts > of code that rely on "temporarily recursive" imports. Presumably Greg's flag would be initialized to "not broken", and set to "broken" when and only when an uncaught exception is raised. So in import A A imports B which imports A back to B B completes back to A A blows up before completing A's "broken" flag isn't set until the end, so B's recursive import of A isn't harmed. From sdm7g@Virginia.EDU Mon Dec 17 23:43:48 2001 From: sdm7g@Virginia.EDU (Steven Majewski) Date: Mon, 17 Dec 2001 18:43:48 -0500 (EST) Subject: [Python-Dev] regrtest.py mystery In-Reply-To: <200112172224.RAA19426@cj20424-a.reston1.va.home.com> Message-ID: On Mon, 17 Dec 2001, Guido van Rossum wrote: > Python doesn't let you replace a reference to an object with something > else -- except in special cases (e.g. with explicit weak refs) there's > no way to know where references to an object might exist. I was too fuzzy in my choice of words. What I meant was not finding and replacing the references, but replacing the object to which all of the 'dangling references' point. But that won't work either: Python objects share the same head, but total size of the object structs are too variable, and I don't know it's possible to make the minimal size object show the required behaviour. But even if that is possible, another problem with implementing it in a Python extension module is that Python objects are not (except for maybe functions and things that drag along a reference to global module namespace) "owned" by their modules in anything like the association between Lisp symbols and their packages: python modules can import other modules or can dynamically add things to their namespaces. We can't just iterate through a module dict and 'zero' out all of the objects there. > So that's not a useful idea, alas. If the problem was important enough, I think a way could be found around those problems above, but I'm still not sure that module unloading isn't a solution looking for a problem! -- Steve From martin@v.loewis.de Mon Dec 17 23:47:04 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Tue, 18 Dec 2001 00:47:04 +0100 Subject: [Python-Dev] regrtest.py mystery In-Reply-To: <164201c18750$9ef89830$ced241d5@hagrid> (fredrik@pythonware.com) References: <200112172247.LAA17572@s454.cosc.canterbury.ac.nz> <164201c18750$9ef89830$ced241d5@hagrid> Message-ID: <200112172347.fBHNl4m11257@mira.informatik.hu-berlin.de> > the only way I can think of is "execution reached the end of > the module", which would probably break considerable amounts > of code that rely on "temporarily recursive" imports. I would have said "does not throw ImportError". Regards, Martin From jeremy@zope.com Tue Dec 18 00:06:52 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Mon, 17 Dec 2001 19:06:52 -0500 (EST) Subject: [Python-Dev] Re: compiler package udpate In-Reply-To: <200112172209.fBHM9Xx02804@mbuna.arbhome.com.au> References: <15390.14426.296516.102494@slothrop.digicool.com> <200112172209.fBHM9Xx02804@mbuna.arbhome.com.au> Message-ID: <15390.34972.767112.259073@slothrop.digicool.com> I've made some good progress on it-- backported all the obvious stuff. I need to downgrade some of my testing tools so that they work with the old interface and re-run them. Once I've finished with these tests, I'll be satisfied. Jeremy From greg@cosc.canterbury.ac.nz Tue Dec 18 00:13:57 2001 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Dec 2001 13:13:57 +1300 (NZDT) Subject: Broken module detection (Re: [Python-Dev] regrtest.py mystery) In-Reply-To: <200112172347.fBHNl4m11257@mira.informatik.hu-berlin.de> Message-ID: <200112180013.NAA17623@s454.cosc.canterbury.ac.nz> "Martin v. Loewis" : > I would have said "does not throw ImportError". Or any other error. If a module fails to complete all its initialisation code for any reason, it can't be trusted. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido@python.org Tue Dec 18 00:15:59 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 17 Dec 2001 19:15:59 -0500 Subject: [Python-Dev] regrtest.py mystery In-Reply-To: Your message of "Mon, 17 Dec 2001 18:43:48 EST." References: Message-ID: <200112180015.TAA19857@cj20424-a.reston1.va.home.com> > > Python doesn't let you replace a reference to an object with something > > else -- except in special cases (e.g. with explicit weak refs) there's > > no way to know where references to an object might exist. > > I was too fuzzy in my choice of words. What I meant was not > finding and replacing the references, but replacing the object to > which all of the 'dangling references' point. > > But that won't work either: Python objects share the same head, but > total size of the object structs are too variable, and I don't know > it's possible to make the minimal size object show the required > behaviour. In general it's impossible. You can't even replace the type pointer with a pointer to a dummy type, because it would leak any objects referenced from the original object, and it would break if the original object lived in a special free list. > If the problem was important enough, I think a way could be found > around those problems above, but I'm still not sure that module > unloading isn't a solution looking for a problem! Well, there are a bunch of people who want to undo a partial import, and that boils down to about the same thing. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@home.com Tue Dec 18 02:41:23 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 17 Dec 2001 21:41:23 -0500 Subject: [Python-Dev] regrtest.py mystery In-Reply-To: Message-ID: [Ka-Ping Yee] Hi, Ping! If we don't hear from you again, Merry Christmas too . > I remember suggesting exactly that some time ago (i think it was > motivated at the time by the extreme pain that broken modules were > causing for webserver-mode pydoc in its attempt to update loaded > modules if the source files had changed on disk). > > Guido rejected it because you can't guarantee that the refcount on M > is 1 at the point where you attempt to 'del sys.modules[M]' above. If somebody wants to hold on to a reference to a damaged module object, that's OK by me. That's an unusual case, though, so shouldn't wag the dog. > (For example, some other module imported by M could have imported M > again, and so hold a reference to it while M is running its startup > commands. This is why the entry is added to sys.modules before the > body of M starts to run.) Understood, but it's rare. > He deemed the situation where M is loaded-but-missing-from-sys.modules > to be even worse than for M to be left loaded-but-broken therein. Why? The module is broken no mater what. I'm much keener on seeing that the *next* attempt to import the module doesn't falsely appear to succeed. > If you allow M to stay in sys.modules, then you can at least maintain the > guarantee that there is a one-to-one association between loaded module > names and loaded module objects. If you remove M from sys.modules but > it lingers in memory, referenced elsewhere, you lose even that -- there > can be many modules loaded all with the same name and it's a nightmare. Whether there's one copy or a million, it's broken. Given that, I don't see value in maintaining the one_name<->one_module invariant: of what practical use is it if the module is an insane state regardless? If there is some value to it I'm missing, it would be easy to give it a unique random (re)name . > The argument is compelling and i am forced to agree, but i still think > that we should look for a better solution. I'll ponder this and post > any ideas i come up with. Greg's "broken module" flag would do it for me too, if an attempt to import a known-to-be-broken module raised ImportError (in addition to Greg's suggestion that attribute references complain). That scheme has holes too, but it gets increasingly harder to fall into them. The current scheme has so many pits it's proved impossible not to fall into them when using regrtest -r, just doing utterly vanilla non-recursive imports. compelling-my-foot-ly y'rs - tim From tim.one@home.com Tue Dec 18 02:42:51 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 17 Dec 2001 21:42:51 -0500 Subject: [Python-Dev] Unloading modules (regrtest.py mystery) In-Reply-To: <3C1E62E4.C4E9A7A0@lemburg.com> Message-ID: [MAL] > ... > Would be nice if we could come up with a reasonable way of unloading > modules, but I guess this requires some hard PEP work... Yes, I believe it would. There are different kinds of unloading too: a module may well want to do different things depending on whether, e.g., it's really a reload, or it's because Py_Finalize was called ... hmm! That reminds me: Guido sent me this msg in March; it's still sitting in my inbox waiting for a rainy day: """ OK, so we need to define a way for modules to specify a function to be called at cleanup time. I propose that we bring this out in the open, at least in python-dev. Note that there are three or four potential cleanup times: - when the module object is deleted from sys.modules - when reload() is used on an extension - when an interpreter created with PyInterpreter_New() is finalized - when Py_Finalize() is called - when Python exits - when the Python DLL is unloaded (Windows only) """ There. Somebody double the word count and call it a PEP . Note that this is orthogonal to regrtest -r's woes. From mal@lemburg.com Tue Dec 18 09:22:25 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 18 Dec 2001 10:22:25 +0100 Subject: [Python-Dev] Unloading modules (regrtest.py mystery) References: Message-ID: <3C1F0AD1.8D580A4@lemburg.com> Tim Peters wrote: > > [MAL] > > ... > > Would be nice if we could come up with a reasonable way of unloading > > modules, but I guess this requires some hard PEP work... > > Yes, I believe it would. There are different kinds of unloading too: a > module may well want to do different things depending on whether, e.g., it's > really a reload, or it's because Py_Finalize was called ... hmm! That > reminds me: Guido sent me this msg in March; it's still sitting in my inbox > waiting for a rainy day: > > """ > OK, so we need to define a way for modules to specify a function to be > called at cleanup time. I propose that we bring this out in the open, > at least in python-dev. Note that there are three or four potential > cleanup times: > > - when the module object is deleted from sys.modules > > - when reload() is used on an extension > > - when an interpreter created with PyInterpreter_New() is finalized > > - when Py_Finalize() is called I think we already have this one: I usually register a Py_AtExit (or whatever the name is) function with Python to implement this. > - when Python exits > > - when the Python DLL is unloaded (Windows only) Why Windows only ? I believe we could support this on other platforms as well (e.g. Linux). > """ > > There. Somebody double the word count and call it a PEP . Since Barry brought this up, I guess we already have a volunteer ;-) > Note that this is orthogonal to regrtest -r's woes. True and it won't help the suite with the encodings package either (how would you know what to cleanup given the fact that the codec search functions decide how encoding names are mapped to codecs ?). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From niemeyer@conectiva.com Tue Dec 18 11:50:04 2001 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Tue, 18 Dec 2001 09:50:04 -0200 Subject: [Python-Dev] Re: mpz module broken with GMP 4.0 In-Reply-To: <0112151923261C.01556@arthur> References: <200112151623.LAA32158@cj20424-a.reston1.va.home.com> <0112151923261C.01556@arthur> Message-ID: <20011218095004.B1599@ibook.distro.conectiva> --0eh6TmSyL6TZE2Uz Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable > Thanks for the indication -- I was not aware of the new GMP 4.0. A rapid= =20 > test shows the current gmpy.c, release 0.8, breaks in exactly one place= =20 > (structure gmpstate_t used to have a member named seed, now renamed to=20 > _mp_seed). Once that single line is fixed, gmpy 0.8 builds fine (on a=20 Have you commited to the CVS? It'd be nice to have it compiling with both, the new and the old gmp. > Linux box) with the new GMP 4.0 and passes all of its 973 unit-tests (wit= h=20 > Python 2.2b1). I think we should rapidly release a 0.8.1 with the one-li= ne=20 > fix (arranged so it keeps building with GMP 3.1 while also building with= =20 > 4.0, of course), and plan the modest gmpy extensions needed to exploit=20 Ohh.. that's it! Nice.. :-) I'll commit the multiprecision pi code, so the next version already includes it. Thanks!! --=20 Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] --0eh6TmSyL6TZE2Uz Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE8Hy1sIlOymmZkOgwRAs8dAJ9UgFEYGeLQY/gjz1OMM4FGCNcPFQCg0uuv Uvq90IZdQTWRWUXTJjjgtn0= =KsOJ -----END PGP SIGNATURE----- --0eh6TmSyL6TZE2Uz-- From barry@zope.com Tue Dec 18 14:40:55 2001 From: barry@zope.com (Barry A. Warsaw) Date: Tue, 18 Dec 2001 09:40:55 -0500 Subject: [Python-Dev] Unloading modules (regrtest.py mystery) References: <3C1F0AD1.8D580A4@lemburg.com> Message-ID: <15391.21879.762287.430662@anthem.wooz.org> >>>>> "M" == M writes: >> """ There. Somebody double the word count and call it a PEP >> . M> Since Barry brought this up, I guess we already have a M> volunteer ;-) Sure MAL, I'll assign you a PEP number! -Barry From fdrake@acm.org Tue Dec 18 17:26:36 2001 From: fdrake@acm.org (Fred L. Drake) Date: Tue, 18 Dec 2001 12:26:36 -0500 (EST) Subject: [Python-Dev] [development doc updates] Message-ID: <20011218172636.9E823286BC@cj42289-a.reston1.va.home.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Added documentation for pydoc. Misc. other fixes. From thomas.heller@ion-tof.com Tue Dec 18 21:25:06 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 18 Dec 2001 22:25:06 +0100 Subject: Need help with bdist_wininst, was Re: [Python-Dev] RE: [Numpy-discussion] Python 2.2 References: <000001c18715$477816c0$0c01a8c0@freedom> <028501c1871a$ec399260$e000a8c0@thomasnotebook> Message-ID: <09bc01c1880a$8f0a4530$e000a8c0@thomasnotebook> I've just fixed the bdist_wininst command in CVS. There is not much time left before the release of Python 2.2 (how much exactly?). To make sure that the current version works correctly, I would kindly request some testers to build windows installers from some of their package distributions, and check them to make sure they work correctly. The full story is in bug [#483982]: Python 2.2b2 bdist_wininst crashes: http://sourceforge.net/tracker/index.php?func=detail&aid=483982&group_id=5470&atid=105470 To test the new version, you only have to replace the file Lib/distutils/command/bdist_wininst.py in your Python 2.2 distribution with the new one from CVS (rev 1.27): http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Lib/distutils/command/bdist_wininst.py Thanks, Thomas From mal@lemburg.com Tue Dec 18 21:47:24 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 18 Dec 2001 22:47:24 +0100 Subject: Need help with bdist_wininst, was Re: [Python-Dev] RE: [Numpy-discussion] Python 2.2 References: <000001c18715$477816c0$0c01a8c0@freedom> <028501c1871a$ec399260$e000a8c0@thomasnotebook> <09bc01c1880a$8f0a4530$e000a8c0@thomasnotebook> Message-ID: <3C1FB96C.E3EE8DCB@lemburg.com> Thomas Heller wrote: > > I've just fixed the bdist_wininst command in CVS. > There is not much time left before the release of Python 2.2 (how much exactly?). > > To make sure that the current version works correctly, I would > kindly request some testers to build windows installers from some > of their package distributions, and check them to make sure they work > correctly. > > The full story is in bug [#483982]: Python 2.2b2 bdist_wininst crashes: > http://sourceforge.net/tracker/index.php?func=detail&aid=483982&group_id=5470&atid=105470 > > To test the new version, you only have to replace the file > Lib/distutils/command/bdist_wininst.py in your Python 2.2 distribution > with the new one from CVS (rev 1.27): > > http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Lib/distutils/command/bdist_wininst.py FYI, I just tested it with egenix-mx-base and egenix-mx-commercial and both seem to work just fine. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From tim@zope.com Tue Dec 18 22:17:22 2001 From: tim@zope.com (Tim Peters) Date: Tue, 18 Dec 2001 17:17:22 -0500 Subject: Need help with bdist_wininst, was Re: [Python-Dev] RE: [Numpy-discussion] Python 2.2 In-Reply-To: <09bc01c1880a$8f0a4530$e000a8c0@thomasnotebook> Message-ID: [Thomas Heller] > I've just fixed the bdist_wininst command in CVS. Thank you! > There is not much time left before the release of Python 2.2 (how > much exactly?). 2.2 final should be released this Friday. From barry@zope.com Tue Dec 18 22:19:03 2001 From: barry@zope.com (Barry A. Warsaw) Date: Tue, 18 Dec 2001 17:19:03 -0500 Subject: Need help with bdist_wininst, was Re: [Python-Dev] RE: [Numpy-discussion] Python 2.2 References: <000001c18715$477816c0$0c01a8c0@freedom> <028501c1871a$ec399260$e000a8c0@thomasnotebook> <09bc01c1880a$8f0a4530$e000a8c0@thomasnotebook> Message-ID: <15391.49367.712590.185830@anthem.wooz.org> >>>>> "TH" == Thomas Heller writes: TH> I've just fixed the bdist_wininst command in CVS. There is TH> not much time left before the release of Python 2.2 (how much TH> exactly?). About 3 days. :) -Barry From paul@pfdubois.com Wed Dec 19 00:06:57 2001 From: paul@pfdubois.com (Paul F. Dubois) Date: Tue, 18 Dec 2001 16:06:57 -0800 Subject: Need help with bdist_wininst, was Re: [Python-Dev] RE: [Numpy-discussion] Python 2.2 In-Reply-To: <09bc01c1880a$8f0a4530$e000a8c0@thomasnotebook> Message-ID: <000001c18821$20455ec0$0c01a8c0@freedom> Our hero! Numeric installs ok now. -----Original Message----- From: numpy-discussion-admin@lists.sourceforge.net [mailto:numpy-discussion-admin@lists.sourceforge.net] On Behalf Of Thomas Heller Sent: Tuesday, December 18, 2001 1:25 PM To: Paul F. Dubois; numpy-discussion@lists.sourceforge.net Cc: python-dev@python.org Subject: Need help with bdist_wininst, was Re: [Python-Dev] RE: [Numpy-discussion] Python 2.2 I've just fixed the bdist_wininst command in CVS. There is not much time left before the release of Python 2.2 (how much exactly?). To make sure that the current version works correctly, I would kindly request some testers to build windows installers from some of their package distributions, and check them to make sure they work correctly. The full story is in bug [#483982]: Python 2.2b2 bdist_wininst crashes: http://sourceforge.net/tracker/index.php?func=detail&aid=483982&group_id =5470&atid=105470 To test the new version, you only have to replace the file Lib/distutils/command/bdist_wininst.py in your Python 2.2 distribution with the new one from CVS (rev 1.27): http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Li b/distutils/command/bdist_wininst.py Thanks, Thomas _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From mhammond@skippinet.com.au Wed Dec 19 01:57:31 2001 From: mhammond@skippinet.com.au (Mark Hammond) Date: Wed, 19 Dec 2001 12:57:31 +1100 Subject: [Python-Dev] No longer at ActiveState. Message-ID: FYI, I am no longer emplyed by ActiveState (Merry Christmas! :) ActiveState have pretty much stopped all Komodo and Python work for the forseeable future. This left me without an immediate project and therefore without an immediate job. It was very dissapointing to hear this just after slaving to get the 1.2 release of Komodo out the door and just a few days before Christmas. So, please don't use my ActiveState.com email address. Further, I will probably be moving back to being self-employed (in the short term anyway), so if anyone hears of suitable opportunities, then feel free to pass them my way :) On the bright side, it will free my time to finish up some pending Python work I have been meaning to do. Happy holidays, Mark. From neal@metaslash.com Wed Dec 19 02:45:03 2001 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 18 Dec 2001 21:45:03 -0500 Subject: [Python-Dev] test coverage Message-ID: <3C1FFF2F.A69CDCE@metaslash.com> I've been doing a bunch of testing using the regression tests and have tried to improve the test coverage. I've made some progress. Overall, the regression tests do a pretty good job. Below are all the functions that are not currently covered by the regression tests in Objects/ Parser/ and Python/ (Modules/ has too many at this point). This list may not be perfect, but should be pretty close (some tests failed: test_new, test_funcattrs, and test_symtable). I don't know how many of these can be reached from within the python (interpreter), some may only be callable from C code which embeds python. After improving test coverage, I'll go back and run purify to see if there are any new memory leaks. Sorry, the list is kinda long. Let me know if there's a better way to distribute this info. Neal -- Objects/abstract.c PyObject_CallMethodObjArgs PyObject_CallFunctionObjArgs PyObject_Type PyObject_Length PyObject_AsWriteBuffer PyNumber_InPlaceTrueDivide PySequence_Length PySequence_In PySequence_InPlaceConcat PySequence_InPlaceRepeat PyMapping_Length PyMapping_HasKeyString PyMapping_HasKey PyMapping_GetItemString PyMapping_SetItemString null_error sliceobj_from_intint objargs_mktuple Objects/bufferobject.c PyBuffer_FromReadWriteObject PyBuffer_FromReadWriteMemory PyBuffer_New buffer_compare buffer_getwritebuf Objects/cellobject.c cell_compare cell_repr Objects/classobject.c instance_ass_item PyMethod_Function PyMethod_Self PyMethod_Class set_slot set_attr_slots set_dict set_bases set_name instancemethod_hash Objects/cobject.c PyCObject_FromVoidPtrAndDesc PyCObject_GetDesc PyCObject_Import Objects/complexobject.c PyComplex_FromDoubles complex_print complex_div Objects/descrobject.c member_repr getset_repr wrapper_repr method_get_doc wrapper_get_doc proxy_len proxy_has_key proxy_get proxy_values proxy_items proxy_copy proxy_getiter proxy_str proxy_traverse wrapper_name Objects/dictobject.c PyDict_Values PyDict_Items dictiter_next Objects/fileobject.c PyFile_FromString PyFile_Name Objects/floatobject.c PyFloat_AsString PyFloat_AsStringEx float_print Objects/funcobject.c PyFunction_GetCode PyFunction_GetGlobals PyFunction_GetDefaults PyFunction_GetClosure PyClassMethod_New func_set_code func_set_defaults Objects/iterobject.c iter_traverse iter_next calliter_traverse calliter_next Objects/listobject.c list_print immutable_list_op immutable_list_ass Objects/longobject.c PyLong_AsVoidPtr Objects/methodobject.c listmethodchain PyCFunction_GetFunction PyCFunction_GetSelf PyCFunction_GetFlags meth_traverse meth_get__self__ meth_compare Objects/object.c _PyObject_Dump PyObject_Not PyObject_Malloc PyObject_Realloc PyObject_Free none_dealloc NotImplemented_repr Objects/rangeobject.c range_compare range_slice range_tolist range_getattr Objects/sliceobject.c PySlice_GetIndices slice_compare Objects/stringobject.c _Py_ReleaseInternedStrings PyString_Decode PyString_Encode PyString_AsEncodedString PyString_AsDecodedString string_getbuffer string_buffer_getwritebuf Objects/structseq.c structseq_repr structseq_concat structseq_repeat structseq_contains structseq_hash Objects/tupleobject.c tupleprint Objects/typeobject.c slot_nb_coerce type_set_module type_subclasses wrap_binaryfunc_r wrap_coercefunc wrap_ternaryfunc_r wrap_sq_delitem wrap_delslice wrap_delitem wrap_hashfunc slot_sq_length slot_sq_ass_item slot_tp_iter slot_tp_iternext super_repr Objects/unicodeobject.c PyUnicodeUCS2_Replace PyUnicodeUCS2_AsUnicode PyUnicodeUCS2_GetSize PyUnicodeUCS2_FromWideChar PyUnicodeUCS2_AsWideChar PyUnicodeUCS2_SetDefaultEncoding PyUnicodeUCS2_Encode PyUnicodeUCS2_AsUTF16String PyUnicodeUCS2_AsUnicodeEscapeString PyUnicodeUCS2_AsRawUnicodeEscapeString PyUnicodeUCS2_AsCharmapString PyUnicodeUCS2_Translate PyUnicodeUCS2_Tailmatch PyUnicodeUCS2_Find utf8_decoding_error utf16_decoding_error latin1_encoding_error charmap_encoding_error unicode_index unicode_rindex unicode_buffer_getreadbuf unicode_buffer_getwritebuf Objects/weakrefobject.c PyWeakref_GetObject gc_clear weakref_repr proxy_print proxy_repr proxy_compare proxy_nonzero proxy_slice proxy_ass_slice proxy_contains proxy_length proxy_setitem Parser/bitset.c _Py_delbitset Parser/grammar1.c PyGrammar_LabelRepr Parser/listnode.c list1node PyNode_ListTree listnode Parser/myreadline.c PyOS_StdioReadline PyOS_Readline my_fgets Parser/parsetok.c PyParser_ParseString Parser/pgen.c dumpstate dumpnfa printssdfa Parser/pgenmain.c PySys_WriteStderr Py_FatalError PyOS_Readline Parser/tokenizer.c indenterror Python/ceval.c call_exc_trace PyEval_CallObject Py_SetRecursionLimit Py_GetRecursionLimit PyEval_AcquireLock PyEval_ReleaseThread loop_subscript gen_traverse gen_next Python/codecs.c PyCodec_StreamReader PyCodec_StreamWriter build_stream_codec Python/compile.c PyCode_Addr2Line PyNode_CompileSymtable code_repr do_pad dump Python/errors.c PyErr_WriteUnraisable PyErr_BadArgument PyErr_NoMemory PyErr_BadInternalCall _PyErr_BadInternalCall Python/frozenmain.c Py_FrozenMain Python/getargs.c PyArg_VaParse Python/import.c PyImport_ExtendInittab PyImport_ExecCodeModule PyImport_AppendInittab get_frozen_object imp_init_builtin imp_init_frozen imp_get_frozen_object imp_is_builtin imp_is_frozen imp_load_compiled imp_load_dynamic imp_load_module imp_load_package Python/marshal.c PyMarshal_ReadShortFromFile PyMarshal_ReadObjectFromFile w_more marshal_dump marshal_load Python/modsupport.c PyEval_CallFunction _ustrlen Python/pyfpe.c PyFPE_dummy Python/pystate.c PyInterpreterState_Head PyInterpreterState_Next PyInterpreterState_ThreadHead PyThreadState_Next Python/pythonrun.c Py_NewInterpreter PyRun_InteractiveLoopFlags PyRun_InteractiveOneFlags parse_syntax_error print_error_text PyErr_Display Py_FatalError Py_SetPythonHome Py_EndInterpreter PyRun_AnyFile PyRun_AnyFileEx PyRun_AnyFileFlags PyRun_SimpleString PyRun_SimpleStringFlags PyRun_SimpleFile PyRun_SimpleFileEx PyRun_InteractiveOne PyRun_InteractiveLoop PyRun_File PyRun_FileFlags Py_CompileString Py_SymtableString PyErr_Print Py_AtExit run_pyc_file add_flag Python/structmember.c PyMember_Set listmembers Python/symtable.c ste_repr Python/sysmodule.c mywrite PySys_WriteStdout PySys_WriteStderr PySys_GetFile PySys_AddWarnOption sys_excepthook sys_getdefaultencoding sys_setdefaultencoding sys_setcheckinterval sys_setrecursionlimit sys_getrecursionlimit sys_setdlopenflags sys_getdlopenflags Python/thread.c PyThread_allocate_sema PyThread_free_sema PyThread_down_sema PyThread_up_sema PyThread__exit_thread Python/traceback.c tb_displayline PyTraceBack_Print tb_clear tb_printinternal From guido@python.org Wed Dec 19 02:51:25 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 18 Dec 2001 21:51:25 -0500 Subject: [Python-Dev] No longer at ActiveState. In-Reply-To: Your message of "Wed, 19 Dec 2001 12:57:31 +1100." References: Message-ID: <200112190251.VAA23676@cj20424-a.reston1.va.home.com> > FYI, I am no longer emplyed by ActiveState (Merry Christmas! :) Au, that hurts. :( > ActiveState have pretty much stopped all Komodo and Python work for > the forseeable future. This left me without an immediate project > and therefore without an immediate job. It was very dissapointing > to hear this just after slaving to get the 1.2 release of Komodo out > the door and just a few days before Christmas. I'll say. I'm guessing we might as well drop ActiveState from the list of PSF sponsors now. > So, please don't use my ActiveState.com email address. Further, I > will probably be moving back to being self-employed (in the short > term anyway), so if anyone hears of suitable opportunities, then > feel free to pass them my way :) Hmm... The economy is going through a rough time on this side of the pond / border as well. :( > On the bright side, it will free my time to finish up some pending > Python work I have been meaning to do. Yay! Will we see you at the next (last) Python conference? --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz@rahul.net Wed Dec 19 03:38:52 2001 From: aahz@rahul.net (Aahz Maruch) Date: Tue, 18 Dec 2001 19:38:52 -0800 (PST) Subject: [Python-Dev] IPC10 In-Reply-To: <200112190251.VAA23676@cj20424-a.reston1.va.home.com> from "Guido van Rossum" at Dec 18, 2001 09:51:25 PM Message-ID: <20011219033853.67370E8C3@waltz.rahul.net> Guido van Rossum wrote: > > Will we see you at the next (last) Python conference? Am I misreading this, or is IPC10 going to be the final Python conference? (I'm still way behind on c.l.py, but I figure something like this ought to have been posted to c.l.py.announce if true.) -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many. From tim.one@home.com Wed Dec 19 06:13:44 2001 From: tim.one@home.com (Tim Peters) Date: Wed, 19 Dec 2001 01:13:44 -0500 Subject: [Python-Dev] No longer at ActiveState. In-Reply-To: Message-ID: [Mark Hammond] > FYI, I am no longer emplyed by ActiveState (Merry Christmas! :) Indeed! So you'll be using the PythonLabs installer again? Pay close attention to its "Installation Finished" screen -- I changed that months ago in anticipation of your coming home to us . > ActiveState have pretty much stopped all Komodo and Python work for the > forseeable future. This left me without an immediate project and > therefore without an immediate job. Bummer. It's times like this I'm glad Aussies don't have feelings. > It was very dissapointing to hear this just after slaving to get the 1.2 > release of Komodo out the door and just a few days before Christmas. Yup. If it's any consolation, BeOpen.com stopped paying us the week before Python 2.0 was scheduled for release. We all wanted to go on a month-long bender, but Guido didn't let us leave his office until the release was finished. See? You *could* have been working for Guido, and *then* what a living hell your life would have been ... > So, please don't use my ActiveState.com email address. Further, I will > probably be moving back to being self-employed (in the short term anyway), > so if anyone hears of suitable opportunities, then feel free to > pass them my way :) Will do. > On the bright side, it will free my time to finish up some pending Python > work I have been meaning to do. If you find a way to move Armin Rigo's Psyco along while getting paid for it, I'll join you. Else I'll just reassign every bug and patch with the string "win" in it to you -- heh heh. Have a relaxed holiday, Mark! We'll still be here when you get out of detox. or-at-least-guido-will-be-and-that's-enough-ly y'rs - tim From tim.one@home.com Wed Dec 19 06:20:09 2001 From: tim.one@home.com (Tim Peters) Date: Wed, 19 Dec 2001 01:20:09 -0500 Subject: [Python-Dev] IPC10 In-Reply-To: <20011219033853.67370E8C3@waltz.rahul.net> Message-ID: [Guido] > Will we see you at the next (last) Python conference? [Aahz] > Am I misreading this, or is IPC10 going to be the final Python > conference? I expect Guido was confusing tenses due to time machine travel to fix the worst 2.2 bugs before they got reported -- they were (or would have been -- or will be had they not been fixed beforehand in the past now) really embarrassing. Whether there will be another Python conference is always a matter for keen speculation. The organizers claim to lose money on it, despite the craters it leaves in attendees' wallets. Ask again about a year from now ... From thomas.heller@ion-tof.com Wed Dec 19 07:48:23 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Wed, 19 Dec 2001 08:48:23 +0100 Subject: Need help with bdist_wininst, was Re: [Python-Dev] RE: [Numpy-discussion] Python 2.2 References: <000001c18715$477816c0$0c01a8c0@freedom> <028501c1871a$ec399260$e000a8c0@thomasnotebook> <09bc01c1880a$8f0a4530$e000a8c0@thomasnotebook> Message-ID: <0c8101c18861$a101a790$e000a8c0@thomasnotebook> > I've just fixed the bdist_wininst command in CVS. > There is not much time left before the release of Python 2.2 (how much exactly?). > > To make sure that the current version works correctly, I would > kindly request some testers to build windows installers from some > of their package distributions, and check them to make sure they work > correctly. Thanks for the tests, Paul and Marc-Andre. Thomas From fdrake@acm.org Wed Dec 19 16:52:38 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Wed, 19 Dec 2001 11:52:38 -0500 (EST) Subject: [Python-Dev] test coverage In-Reply-To: <3C1FFF2F.A69CDCE@metaslash.com> References: <3C1FFF2F.A69CDCE@metaslash.com> Message-ID: <15392.50646.688242.432369@cj42289-a.reston1.va.home.com> Neal Norwitz writes: > I don't know how many of these can be reached from within > the python (interpreter), some may only be callable from C code > which embeds python. I've added a few tests for the weakref support that should at least improve the coverage there. I don't think I'll have any more time to help in this effort before 2.2 goes out, but I think this is really important. > After improving test coverage, I'll go back and run purify > to see if there are any new memory leaks. > > Sorry, the list is kinda long. Let me know if there's a better way > to distribute this info. I can live with this distribution method; if it gets us to actually improve coverage, we all get to see the benefit of a shorter list. ;-) -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From mhammond@skippinet.com.au Thu Dec 20 06:06:04 2001 From: mhammond@skippinet.com.au (Mark Hammond) Date: Thu, 20 Dec 2001 17:06:04 +1100 Subject: [Python-Dev] No longer at ActiveState. In-Reply-To: <200112190251.VAA23676@cj20424-a.reston1.va.home.com> Message-ID: > > ActiveState have pretty much stopped all Komodo and Python work for > > the forseeable future. This left me without an immediate project > > and therefore without an immediate job. It was very dissapointing > > to hear this just after slaving to get the 1.2 release of Komodo out > > the door and just a few days before Christmas. > > I'll say. I'm guessing we might as well drop ActiveState from the > list of PSF sponsors now. I obviously can't speak for ActiveState ;-) I really have no idea exactly what "pretty much stopped ... Python work" means in reality, nor what the longer term plans are. > > On the bright side, it will free my time to finish up some pending > > Python work I have been meaning to do. > > Yay! Indeed :) > Will we see you at the next (last) Python conference? Unfortunately not :( It would be irresponsible of me to commit to that much money without knowing where my next dollar was coming from :) And if you had ever met my girlfriend, you would know that I am very aware exactly where my next dollar is *going* :) Mark. From barry@zope.com Thu Dec 20 06:18:02 2001 From: barry@zope.com (Barry A. Warsaw) Date: Thu, 20 Dec 2001 01:18:02 -0500 Subject: [Python-Dev] No longer at ActiveState. References: <200112190251.VAA23676@cj20424-a.reston1.va.home.com> Message-ID: <15393.33434.171378.132917@anthem.wooz.org> >>>>> "MH" == Mark Hammond writes: MH> Unfortunately not :( It would be irresponsible of me to commit MH> to that much money without knowing where my next dollar was MH> coming from :) And if you had ever met my girlfriend, you MH> would know that I am very aware exactly where my next dollar MH> is *going* :) LOL, COL. We'll raise a pint for ya Mark. purple-ly y'rs, -Barry From barry@zope.com Thu Dec 20 19:28:32 2001 From: barry@zope.com (Barry A. Warsaw) Date: Thu, 20 Dec 2001 14:28:32 -0500 Subject: [Python-Dev] Python 2.2 final - release branch created Message-ID: <15394.15328.321075.820271@anthem.wooz.org> I've created the Python 2.2 final release branch. The CVS tag is different than with the alpha and beta release branches. Here, it is called `release22-branch'. There's also a tag `release22-fork' -- that's the point on the trunk where the branch forks off. So far the only change in the branch is the version number. As usual, no one except Guido and myself (or our authorized bots) should make checkins to the branch. Checkins to the trunk and branch are discouraged unless they fix showstopper bugs in Python 2.2. Nothing will be merged from the trunk to the branch unless you notify us, e.g. with a comment in the checkin message. I'm planning to cut the release tomorrow morning local time, Friday December 21. Enjoy, -Barry From skip@pobox.com (Skip Montanaro) Thu Dec 20 21:02:01 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Thu, 20 Dec 2001 15:02:01 -0600 Subject: [Python-Dev] Can we checkin on the main trunk? Message-ID: <15394.20937.842175.101270@beluga.mojam.com> I have a trivial change for asynchat to shut up pychecker: /usr/local/lib/python2.2/asynchat.py:83: Local variable (why) not used /usr/local/lib/python2.2/asynchat.py:217: Local variable (why) not used (delete the "why" variable in both cases). Can I check in on the main trunk and have it go into 2.3 without disturbing the 2.2 release? I'm never sure during these alpha/beta/release candidate situations what is and isn't allowed (other than that Barry's new branch is clearly off-limits). This is such a trivial change that it seems like overkill to me to submit a bug report or a patch, have it assigned to someone who two weeks later says, "sure, looks fine to me - check it in". (Hint: For those of us with rotting neurons, whenever development enters a release phase such as we are in right now, it would be helpful to remind us what we can do as well as what we can't.) Skip From jack@oratrix.nl Thu Dec 20 23:20:48 2001 From: jack@oratrix.nl (Jack Jansen) Date: Fri, 21 Dec 2001 00:20:48 +0100 Subject: [Python-Dev] test_cpickle failing in MacPython - ideas needed Message-ID: <20011220232048.430FDE8453@oratrix.oratrix.nl> Folks, test_cpickle suddenly started failing (often with a hang) in MacPython (CVS tree, main branch). As I'm going to be offline all day tomorrow and I'd like to get MacPython out the door simultaneously with unix and windows-Python: does anyone know what could have changed over the last couple of days to cause this? -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | ++++ see http://www.xs4all.nl/~tank/ ++++ From guido@python.org Thu Dec 20 23:19:42 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 20 Dec 2001 18:19:42 -0500 Subject: [Python-Dev] Can we checkin on the main trunk? In-Reply-To: Your message of "Thu, 20 Dec 2001 15:02:01 CST." <15394.20937.842175.101270@beluga.mojam.com> References: <15394.20937.842175.101270@beluga.mojam.com> Message-ID: <200112202319.fBKNJjR24505@odiug.zope.com> > I have a trivial change for asynchat to shut up pychecker: > > /usr/local/lib/python2.2/asynchat.py:83: Local variable (why) not used > /usr/local/lib/python2.2/asynchat.py:217: Local variable (why) not used > > (delete the "why" variable in both cases). > > Can I check in on the main trunk and have it go into 2.3 without disturbing > the 2.2 release? Yes, the release branch has been made. The trunk is all yours. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim@zope.com Fri Dec 21 00:19:22 2001 From: tim@zope.com (Tim Peters) Date: Thu, 20 Dec 2001 19:19:22 -0500 Subject: [Python-Dev] test_cpickle failing in MacPython - ideas needed In-Reply-To: <20011220232048.430FDE8453@oratrix.oratrix.nl> Message-ID: [Jack Jansen] > test_cpickle suddenly started failing (often with a hang) in MacPython > (CVS tree, main branch). Ouch! > As I'm going to be offline all day tomorrow and I'd like to get MacPython > out the door simultaneously with unix and windows-Python: does anyone > know what could have changed over the last couple of days to cause this? Two very recent changes: 1. Low probability. Guido changed the way pickle works for instances of new-style classes with custom metatypes. 2. High probability. Since 2001/10/15, test_pickle and test_cpickle actually didn't run any tests at all. Guido discovered this when working on #1 and not seeing an expected error. When they were converted to unittests, an inappropriate piece of if __name__ == '__main__':' boilerplate prevented them from running any tests unless run directly (when run via regrtest.py, they didn't do anything). Fred fixed that about 24 hours ago. Unfortunately, #2 means "the real" problem may have been introduced anytime within the last 2 months. Please try to whittle the failure down -- I'll be happy to help, but there are no failures here. It would also help to know once again *exactly* how you run the test suite (previous problems unique to you have been pinned on something unique about the way you run tests, no matter how much you protest in advance that it isn't possible ). From tim.one@home.com Fri Dec 21 02:10:47 2001 From: tim.one@home.com (Tim Peters) Date: Thu, 20 Dec 2001 21:10:47 -0500 Subject: [Python-Dev] Can we checkin on the main trunk? In-Reply-To: <15394.20937.842175.101270@beluga.mojam.com> Message-ID: [Skip Montanaro] > ... > (Hint: For those of us with rotting neurons, whenever development enters a > release phase such as we are in right now, it would be helpful to remind us > what we can do as well as what we can't.) The things you can do are those you don't get publicly humiliated for doing. Sure, it's a two-pass algorithm, and even then it makes the set of things you can do at best recursively enumerable (there's no bound on how long it may take before you're attacked, so you can never know for sure that you're safe, only that if you're damned you'll *eventually* find out) -- but in the absence of a PEP, that's the best we can do for you . To a good first approximation, don't check in anything on the release *branch* without Barry's explicit advance blessing, and I expect you'll live to be appreciated in 2002. From tim.one@home.com Fri Dec 21 02:48:11 2001 From: tim.one@home.com (Tim Peters) Date: Thu, 20 Dec 2001 21:48:11 -0500 Subject: [Python-Dev] For Jack Jansen Message-ID: Mail to you from me bounces, Jack: Reporting-MTA: dns; femail4.home.com Arrival-Date: Thu, 20 Dec 2001 18:43:54 -0800 Received-From-MTA: dns; CJ569191B (65.1.136.53) Final-Recipient: RFC822; Action: failed Status: 5.1.1 Remote-MTA: dns; hera.cwi.nl (192.16.191.8) Diagnostic-Code: smtp; 550 ... Blocked relay femail4.sdc1.sfba.home.com I'm not sure CWI wants to cut off the 4 million remaining @Home customers via one dumb spam trick -- although I'm sure it's an effective one . From akuchlin@mems-exchange.org Fri Dec 21 04:53:04 2001 From: akuchlin@mems-exchange.org (akuchlin@mems-exchange.org) Date: Thu, 20 Dec 2001 23:53:04 -0500 Subject: [Python-Dev] 2.2 "What's New" article finished Message-ID: <20011220235304.A18909@mozart.mems-exchange.org> I've made a few last updates and called it complete for 2.2final. (Bug reports and suggestions are still welcome, though.) Note for the PythonLabs gang: if you want to make a copy on python.org or zope.com (for press release purposes, consistent page design, or whatever), feel free to do so; the LaTeX source is in the pyhowto CVS tree at sourceforge.net. Oh, yeah, the URL: still http://www.amk.ca/python/2.2/ . --amk (www.amk.ca) The Sony Walkman will become obsolete thanks to the Sony rnman. The instructions will be on a separate device, the Sony manman. -- Kibo, in the Happynet Manifesto From anthony@interlink.com.au Fri Dec 21 05:57:14 2001 From: anthony@interlink.com.au (Anthony Baxter) Date: Fri, 21 Dec 2001 16:57:14 +1100 Subject: [Python-Dev] 2.1.2 status. Message-ID: <200112210557.fBL5vE908871@mbuna.arbhome.com.au> I'm churning through the list of "things that I want to get fixed" for 2.1.2. I've had to abandon my initial plan of examining pretty much all patches for suitability - the backlog since 2.1.1 is just too huge. If there's something you know of that you think really should be in there, and you're not sure if it is actually there yet, please let me know ASAP! Anthony. From barry@zope.com Fri Dec 21 06:05:23 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 21 Dec 2001 01:05:23 -0500 Subject: [Python-Dev] Can we checkin on the main trunk? References: <15394.20937.842175.101270@beluga.mojam.com> Message-ID: <15394.53539.912400.804116@anthem.wooz.org> >>>>> "SM" == Skip Montanaro writes: SM> Can I check in on the main trunk and have it go into 2.3 SM> without disturbing the 2.2 release? I'm never sure during SM> these alpha/beta/release candidate situations what is and SM> isn't allowed (other than that Barry's new branch is clearly SM> off-limits). Trunk checkins are always safe. We create a release branch specifically so that no strict freeze on checkins need be imposed on CVS committers. The release branch is always off-limits to everyone except those of us who are making the releases (or our designated bots), as per PEP 101. This way we can ensure a clean, controlled release branch but not hold up ongoing Python development and maintenance on the trunk. BTW, this policy is fairly new with Python 2.2 and, speaking as Release Manager, I think it has worked exceedingly well. We actually got the idea from the way Zope's CVS is managed, though we've tweaked the process a bit to suite Python's differing needs. aIMO, the most important adjustment was in finding the right balance between the need for a branch and the reality that the inevitable merge from branch back to trunk can be a royal PITA (ask Tim some time about is descr-branch merge experience. ;) Currently we try to branch at about noon (local time 'natch) the day before the release. This seems like a pretty good balance between the conflicting goals of release isolation and a pain-free trunk-merge. I know my announcement earlier today discouraged trunk checkins until we spin the Python 2.2 final release. I think that makes sense given that this is a "final" release, so there are a few special considerations. I'm willing to accept that my discouraging trunk checkins is just paranoia on my part ;). -Barry From davida@ActiveState.com Fri Dec 21 18:06:25 2001 From: davida@ActiveState.com (David Ascher) Date: Fri, 21 Dec 2001 10:06:25 -0800 (PST) Subject: [Python-Dev] ActiveState & Python Message-ID: As Mark mentioned in his email to python-dev, he is no longer employed by ActiveState. While I'm dealing with my grief about that unfortunate event in private, I think it's worth clarifying a few things in public: 1) Over the last two years, Mark has become my personal god of programming. ActiveState as a whole has benefited tremendously from having Mark on staff, I have learned a tremendous amount from him, and the entire ActiveState team hope to be able to bring him back on board as soon as possible. 2) The reason for Mark's departure is simple -- ActiveState has had to make some painful cuts in staff in order to ensure our continued survival in these difficult economic times. We hope these cuts will be short-lived and that we will be able to bring back the excellent personel that had to be let go soon. 3) ActiveState is still committed to supporting its Python initiatives and current product line. For example, a release of ActivePython based on Python 2.2 is forthcoming. As regards Komodo, we do have plans for future releases. The features we are looking at adding in future releases include: * Integrated version control support (right now it's done through the "external command" interface) * Perl autocompletion (not sure that python-dev'ers care about that one =) * Improved web services consumption (1.2 has support for this already) * Macro support * Key-mappings (like emacs and vi key modes) As usual, we look forward to feedback from customers as to what the next version of Komodo should look like. If you have any questions about any of the above, feel free to contact me. I'm on vacation but am reading email periodically. --David Ascher Friend of Mark's Director, Programming Tools ActiveState From guido@python.org Fri Dec 21 19:54:36 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 21 Dec 2001 14:54:36 -0500 Subject: [Python-Dev] RELEASED - Python 2.2 final Message-ID: <200112211954.OAA16972@cj20424-a.reston1.va.home.com> On December 21, just in time to be placed under the Christmas tree, we're issuing the final release of Python 2.2. We're proud of this release, and expect that you'll like it. Please check it out here: http://www.python.org/2.2/ Our thanks to everyone who helped test Python 2.2 during the alpha, beta and release candidate phases, and to everyone who contributed patches, feedback, and suggestions! Please continue to report any bugs you find to the bug tracker: http://sourceforge.net/bugs/?group_id=5470 Highlights of what's new for this release are outlined below. For a more complete list, please see: http://sf.net/project/shownotes.php?release_id=?????? What's new since 2.1 -------------------- In brief (for PEPs, see http://www.python.org/peps/): - type/class unification (PEP 252 and 253) - iterators (PEP 234) and generators (PEP 255) - nested scopes standard (PEP 227) - int overflows return longs (PEP 237) - new operator // to future-proof int division (PEP 238) Andrew Kuchling has written a gentle introduction to the most important changes, titled "What's New in Python 2.2": http://www.amk.ca/python/2.2/ A thorough introduction to the type/class unification is at: http://www.python.org/2.2/descrintro.html Status of the type/class unification ------------------------------------ With a set of changes as large as this, it's unavoidable that after the release we'll find that there are certain things that would work better if we did them a little differently. Also, it is clear that some of the new features (e.g. class and static methods, properties, super, and slots, to mention a few) could use additional syntactic sugar to make their use easier and more intuitive. This will almost certainly require us to make changes in future releases. However, with the release of these features as part of the Python 2.2 release, we are committed to the same standard of release-to-release compatibility that we have used in the past, as explained in PEP 5. When we feel the need to change things, we'll use the PEP system to propose changes, and we'll provide warnings and backwards compatiblity for at least a year -- or more in cases where a year is deemed too short by the user community. In the meantime, we'll work on completing the documentation for these features (both in Python and in the C API) as they are found in Python 2.2. This documentation will clarify the status of individual features. What's new since 2.2c1 ---------------------- (Adapted from the Misc/NEWS file.) Type/class unification and new-style classes - pickle.py, cPickle: allow pickling instances of new-style classes with a custom metaclass. Core and builtins - weakref proxy object: when comparing, unwrap both arguments if both are proxies. Extension modules - binascii.b2a_base64(): fix a potential buffer overrun when encoding very short strings. - cPickle: the obscure "fast" mode caused stack overflows on the Mac. We fixed this by setting the recursion limit much smaller. If the limit is too low (it only affects performance), you can change it by defining PY_CPICKLE_FAST_LIMIT when compiling cPickle.c (or in pyconfig.h). Library - dumbdbm.py: fixed a dumb old bug (the file didn't get synched at close or delete time). - rfc822.py: fixed a bug where the address '<>' was converted to None instead of an empty string (also fixes the email.Utils module). - xmlrpclib.py: version 1.0.0; uses precision for doubles. - test suite: the pickle and cPickle tests were not executing any code when run from the standard regresssion test. Windows - distutils package: fixed broken Windows installers (bdist_wininst). - tempfile.py: prevent mysterious warnings when TemporaryFileWrapper instances are deleted at process exit time. - socket.py: prevent mysterious warnings when socket instances are deleted at process exit time. - posixmodule.c: fix a Windows crash with stat() of a filename ending in backslash. Mac - The Carbon toolbox modules have been upgraded to Universal Headers 3.4, and experimental CoreGraphics and CarbonEvents modules have been added. All only for framework-enabled MacOSX. --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik@pythonware.com Fri Dec 21 20:24:11 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Fri, 21 Dec 2001 21:24:11 +0100 Subject: [Python-Dev] RELEASED - Python 2.2 final References: <200112211954.OAA16972@cj20424-a.reston1.va.home.com> Message-ID: <001401c18a5d$746ebad0$ced241d5@hagrid> > On December 21, just in time to be placed under the Christmas tree, > we're issuing the final release of Python 2.2. hurray for us! see y'all after the holidays. From fdrake@acm.org Fri Dec 21 21:12:12 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 21 Dec 2001 16:12:12 -0500 (EST) Subject: [Python-Dev] Python 2.2.1 Message-ID: <15395.42412.811106.303578@cj42289-a.reston1.va.home.com> I've created the branch release22-maint as the Python 2.2 maintenance branch. We don't yet have a Patch Czar for 2.2.1, though. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From mwh@python.net Fri Dec 21 21:27:43 2001 From: mwh@python.net (Michael Hudson) Date: 21 Dec 2001 21:27:43 +0000 Subject: [Python-Dev] RELEASED - Python 2.2 final In-Reply-To: Guido van Rossum's message of "Fri, 21 Dec 2001 14:54:36 -0500" References: <200112211954.OAA16972@cj20424-a.reston1.va.home.com> Message-ID: <2m8zbwjmb4.fsf@starship.python.net> Guido van Rossum writes: > On December 21, just in time to be placed under the Christmas tree, > we're issuing the final release of Python 2.2. Hurrah! > http://sf.net/project/shownotes.php?release_id=?????? Oops. Cheers, M. -- ARTHUR: Why should a rock hum? FORD: Maybe it feels good about being a rock. -- The Hitch-Hikers Guide to the Galaxy, Episode 8 From guido@python.org Fri Dec 21 21:33:15 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 21 Dec 2001 16:33:15 -0500 Subject: [Python-Dev] RELEASED - Python 2.2 final In-Reply-To: Your message of "21 Dec 2001 21:27:43 GMT." <2m8zbwjmb4.fsf@starship.python.net> References: <200112211954.OAA16972@cj20424-a.reston1.va.home.com> <2m8zbwjmb4.fsf@starship.python.net> Message-ID: <200112212133.QAA18024@cj20424-a.reston1.va.home.com> > > http://sf.net/project/shownotes.php?release_id=?????? Oops, sorry. The correct URL is: http://sourceforge.net/project/shownotes.php?release_id=66610 --Guido van Rossum (home page: http://www.python.org/~guido/) From anthony@interlink.com.au Sun Dec 23 06:16:28 2001 From: anthony@interlink.com.au (Anthony Baxter) Date: Sun, 23 Dec 2001 17:16:28 +1100 Subject: [Python-Dev] 2.1.2rc is ready to go. Message-ID: <200112230616.fBN6GSS07067@mbuna.arbhome.com.au> the 2.1.2 is ready for release candidate - I've yet to go through the hoops to cut the rc1, but please please feel free to pull the CVS version and try it out. Anthony. From guido@python.org Sun Dec 23 14:01:48 2001 From: guido@python.org (Guido van Rossum) Date: Sun, 23 Dec 2001 09:01:48 -0500 Subject: [Python-Dev] 2.1.2rc is ready to go. In-Reply-To: Your message of "Sun, 23 Dec 2001 17:16:28 +1100." <200112230616.fBN6GSS07067@mbuna.arbhome.com.au> References: <200112230616.fBN6GSS07067@mbuna.arbhome.com.au> Message-ID: <200112231401.JAA08934@cj20424-a.reston1.va.home.com> > the 2.1.2 is ready for release candidate - I've yet to go through the hoops > to cut the rc1, but please please feel free to pull the CVS version and try > it out. Excellent! When do you plan to release it? What kind of help do you want from us? We'd be happy to follow PEP 101 but it might have to wait until 2002. --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh@python.net Sun Dec 23 18:08:44 2001 From: mwh@python.net (Michael Hudson) Date: 23 Dec 2001 18:08:44 +0000 Subject: [Python-Dev] Python 2.2.1 In-Reply-To: "Fred L. Drake, Jr."'s message of "Fri, 21 Dec 2001 16:12:12 -0500 (EST)" References: <15395.42412.811106.303578@cj42289-a.reston1.va.home.com> Message-ID: <2mzo49yfkj.fsf@starship.python.net> "Fred L. Drake, Jr." writes: > I've created the branch release22-maint as the Python 2.2 > maintenance branch. > We don't yet have a Patch Czar for 2.2.1, though. I've just spent half an hour fiddling with gnus to the point that it should be easy for me to keep a log of checkins that should be put on the release22-maint branch. This isn't me volunteering to be Patch Czar, by the way... I don't know if I'll have time for that, but I hope that doing this will help whoever it ends up being. Cheers, M. -- About the use of language: it is impossible to sharpen a pencil with a blunt axe. It is equally vain to try to do it with ten blunt axes instead. -- E.W.Dijkstra, 18th June 1975. Perl did not exist at the time. From jack@oratrix.nl Sun Dec 23 22:22:04 2001 From: jack@oratrix.nl (Jack Jansen) Date: Sun, 23 Dec 2001 23:22:04 +0100 Subject: [Python-Dev] Python 2.2.1 In-Reply-To: Message by Michael Hudson , 23 Dec 2001 18:08:44 +0000 , <2mzo49yfkj.fsf@starship.python.net> Message-ID: <20011223222209.C9CE5E8453@oratrix.oratrix.nl> Two questions wrt 2.2.1: - Is there a timeline for it? I couldn't find one in PEP251 (which stops at 2.2 final). The reason I ask is that there's a serious flaw in MacPython (has been there since day one) that renders it useless on multiprocessor Mac OS X machines. I'm waiting for an external party to fix this (as the problem is in an external library), but I could put a bit of pressure on if 2.2.1 is due reasonably soon (as I hope). - The final bits of the Mac tree havne't been checked in yet, should we move the release22-maint branch when that's done? Can you move a branch? How? -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From Anthony Baxter Sun Dec 23 22:25:02 2001 From: Anthony Baxter (Anthony Baxter) Date: Mon, 24 Dec 2001 09:25:02 +1100 Subject: [Python-Dev] Python 2.2.1 In-Reply-To: Message from Michael Hudson of "23 Dec 2001 18:08:44 -0000." <2mzo49yfkj.fsf@starship.python.net> Message-ID: <200112232225.fBNMP2910673@mbuna.arbhome.com.au> >>> Michael Hudson wrote > I've just spent half an hour fiddling with gnus to the point that it > should be easy for me to keep a log of checkins that should be put on > the release22-maint branch. > > This isn't me volunteering to be Patch Czar, by the way... I don't > know if I'll have time for that, but I hope that doing this will help > whoever it ends up being. I'm happy enough to continue doing this - now that it's starting from a caught-up position, it's not that hard. The problem with 2.1.2 was trawling back through 6 months of checkins... Anthony -- Anthony Baxter It's never too late to have a happy childhood. From fdrake@acm.org Sun Dec 23 22:29:56 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Sun, 23 Dec 2001 17:29:56 -0500 (EST) Subject: [Python-Dev] Python 2.2.1 In-Reply-To: <20011223222209.C9CE5E8453@oratrix.oratrix.nl> References: <2mzo49yfkj.fsf@starship.python.net> <20011223222209.C9CE5E8453@oratrix.oratrix.nl> Message-ID: <15398.23268.960166.935728@cj42289-a.reston1.va.home.com> Jack Jansen writes: > - The final bits of the Mac tree havne't been checked in yet, should > we move the release22-maint branch when that's done? Can you move a > branch? How? Yes, it should be moved as necessary to reflect the MacPython release; this can be done using these commands from the dist/src/ directory: cvs tag release22-mac cvs tag -r release22-mac -b -F release22-maint -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From guido@python.org Sun Dec 23 22:31:52 2001 From: guido@python.org (Guido van Rossum) Date: Sun, 23 Dec 2001 17:31:52 -0500 Subject: [Python-Dev] Python 2.2.1 In-Reply-To: Your message of "Sun, 23 Dec 2001 23:22:04 +0100." <20011223222209.C9CE5E8453@oratrix.oratrix.nl> References: <20011223222209.C9CE5E8453@oratrix.oratrix.nl> Message-ID: <200112232231.RAA15943@cj20424-a.reston1.va.home.com> > Two questions wrt 2.2.1: > > - Is there a timeline for it? I couldn't find one in PEP251 (which > stops at 2.2 final). The reason I ask is that there's a serious flaw > in MacPython (has been there since day one) that renders it useless on > multiprocessor Mac OS X machines. I'm waiting for an external party to > fix this (as the problem is in an external library), but I could put a > bit of pressure on if 2.2.1 is due reasonably soon (as I hope). No timeline -- that sort of depends on how many really bad bugs are reported, and whether someone volunteers as a release mananger. > - The final bits of the Mac tree havne't been checked in yet, should > we move the release22-maint branch when that's done? Can you move a > branch? How? Fred anticipated that; he said he would just move the tags when necessary. Don't worry about it. --Guido van Rossum (home page: http://www.python.org/~guido/) From Anthony Baxter Sun Dec 23 22:34:32 2001 From: Anthony Baxter (Anthony Baxter) Date: Mon, 24 Dec 2001 09:34:32 +1100 Subject: [Python-Dev] Python 2.2.1 In-Reply-To: Message from Jack Jansen of "Sun, 23 Dec 2001 23:22:04 BST." <20011223222209.C9CE5E8453@oratrix.oratrix.nl> Message-ID: <200112232234.fBNMYWc10781@mbuna.arbhome.com.au> >>> Jack Jansen wrote > - The final bits of the Mac tree havne't been checked in yet, should > we move the release22-maint branch when that's done? Can you move a > branch? How? D'oh! Hm, I'm not sure how easily that can be done. In theory you should be able to "cvs tag -d release22-maint" in the Mac-specific subdirectory, then do the checkins, then "cvs tag -b release22-maint" in the Mac directory again. So long as there's not been any checkins on the branch in the directories that you're doing the tag shifting, you should be fine. Probably. Although checking on a little toy CVS root is probably a great idea :) In fact, I just did this on a local toy project, and it worked fine. I'd strongly recommend using cvs tag -d followed by retagging, rather than using tag -F to try and move it - I've had bad experiences in the past with -F -- Anthony Baxter It's never too late to have a happy childhood. From martin@v.loewis.de Mon Dec 24 08:32:13 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Mon, 24 Dec 2001 09:32:13 +0100 Subject: [Python-Dev] Python 2.2.1 In-Reply-To: <15398.23268.960166.935728@cj42289-a.reston1.va.home.com> (fdrake@acm.org) References: <2mzo49yfkj.fsf@starship.python.net> <20011223222209.C9CE5E8453@oratrix.oratrix.nl> <15398.23268.960166.935728@cj42289-a.reston1.va.home.com> Message-ID: <200112240832.fBO8WDO01324@mira.informatik.hu-berlin.de> > > - The final bits of the Mac tree havne't been checked in yet, should > > we move the release22-maint branch when that's done? Can you move a > > branch? How? > > Yes, it should be moved as necessary to reflect the MacPython > release; this can be done using these commands from the dist/src/ > directory: > > cvs tag release22-mac > cvs tag -r release22-mac -b -F release22-maint Maybe I'm misunderstanding. It seems pointless to move a branch: Just check in to it. In terms of CVS commands cvs update -rrelease22-maint # get a sandbox of the branch # make modifications cvs commit # make modifications to the branch cvs update -A # return to mainline What am I missing? Regards, Martin From fdrake@acm.org Tue Dec 25 16:49:23 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 25 Dec 2001 11:49:23 -0500 (EST) Subject: [Python-Dev] Python 2.2.1 In-Reply-To: <200112240832.fBO8WDO01324@mira.informatik.hu-berlin.de> References: <2mzo49yfkj.fsf@starship.python.net> <20011223222209.C9CE5E8453@oratrix.oratrix.nl> <15398.23268.960166.935728@cj42289-a.reston1.va.home.com> <200112240832.fBO8WDO01324@mira.informatik.hu-berlin.de> Message-ID: <15400.44563.436248.81801@cj42289-a.reston1.va.home.com> Martin v. Loewis writes: > Maybe I'm misunderstanding. It seems pointless to move a branch: Just > check in to it. In terms of CVS commands That would work too; I'm not going to be picky about the process. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From fdrake@acm.org Wed Dec 26 17:15:40 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Wed, 26 Dec 2001 12:15:40 -0500 (EST) Subject: [Python-Dev] Python 2.2.1 In-Reply-To: <200112232234.fBNMYWc10781@mbuna.arbhome.com.au> References: <20011223222209.C9CE5E8453@oratrix.oratrix.nl> <200112232234.fBNMYWc10781@mbuna.arbhome.com.au> Message-ID: <15402.1468.577251.936398@cj42289-a.reston1.va.home.com> Has anything been done to get the release22-maint branch to see the changes made for MacPython yet? We should do this before we lose track of it. Jack, any preference for how this is done, or did you already do it? -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From jack@oratrix.nl Wed Dec 26 22:37:26 2001 From: jack@oratrix.nl (Jack Jansen) Date: Wed, 26 Dec 2001 23:37:26 +0100 Subject: [Python-Dev] Python 2.2.1 In-Reply-To: Message by "Fred L. Drake, Jr." , Wed, 26 Dec 2001 12:15:40 -0500 (EST) , <15402.1468.577251.936398@cj42289-a.reston1.va.home.com> Message-ID: <20011226223731.262A1E8448@oratrix.oratrix.nl> Recently, "Fred L. Drake, Jr." said: > > Has anything been done to get the release22-maint branch to see the > changes made for MacPython yet? We should do this before we lose > track of it. > Jack, any preference for how this is done, or did you already do it? They haven\t been checked in yet. There was a snag in the Mac distriubution that has just been solved, the checkins will follow shortly. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From fdrake@acm.org Wed Dec 26 22:39:17 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Wed, 26 Dec 2001 17:39:17 -0500 (EST) Subject: [Python-Dev] Python 2.2.1 In-Reply-To: <20011226223731.262A1E8448@oratrix.oratrix.nl> References: <15402.1468.577251.936398@cj42289-a.reston1.va.home.com> <20011226223731.262A1E8448@oratrix.oratrix.nl> Message-ID: <15402.20885.998208.923051@cj42289-a.reston1.va.home.com> Jack Jansen writes: > They haven\t been checked in yet. There was a snag in the Mac > distriubution that has just been solved, the checkins will follow shortly. OK, I'll watch for them. You can make sure the changes hit both the branch and the trunk yourself, or check into the trunk and we can play with CVS to move the changes onto the branch for the Mac tree. If you want me to do anything to help out with this, I should be able to lend a hand later this evening or in the morning. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From fdrake@acm.org Fri Dec 28 04:56:31 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Thu, 27 Dec 2001 23:56:31 -0500 (EST) Subject: [Python-Dev] Documentation for Python 2.2.1 Message-ID: <15403.64383.728258.799543@cj42289-a.reston1.va.home.com> I've brought the docs in Python's release22-maint branch up to date with the corrections that have been made on the trunk. I'll be checking in relevant additions to this branch as well as the trunk when I make the trunk changes. When anyone else makes a change on the trunk which should also be made on the 2.2 maintenance branch, feel free to check it in in both places, or add a note to the checkin message noting that it should be made on the branch as well, and I'll try to make sure that gets done. The 2.2 maintenance branch has replaced the 2.1 maintenance branch as the "maintenance version" at http://python.sourceforge.net/; if anyone thinks we should continue to offer the 2.1 maintenance docs there, please let me know. I don't think we have a Patch Czar for 2.2.1 yet; if anyone would like to take that duty for the documentation tree, I'd certainly appreciate the help! Thanks! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From fdrake@acm.org Fri Dec 28 04:59:57 2001 From: fdrake@acm.org (Fred L. Drake) Date: Thu, 27 Dec 2001 23:59:57 -0500 (EST) Subject: [Python-Dev] [development doc updates] Message-ID: <20011228045957.6E9FD286BB@beowolf.digicool.com> The development version of the documentation has been updated: http://python.sourceforge.net/maint-docs/ The "maintenance version" of the Python documentation on the Python development site at SourceForge has been updated to show the docs being developed for Python 2.2.1. From fdrake@acm.org Fri Dec 28 05:03:42 2001 From: fdrake@acm.org (Fred L. Drake) Date: Fri, 28 Dec 2001 00:03:42 -0500 (EST) Subject: [Python-Dev] [development doc updates] Message-ID: <20011228050342.CB129286BB@beowolf.digicool.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ A variety of updates made since Python 2.2 was released. From martin@v.loewis.de Fri Dec 28 10:13:17 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Fri, 28 Dec 2001 11:13:17 +0100 Subject: [Python-Dev] Re: [XML-SIG] printing Unicode xml to StringIO In-Reply-To: <3C2C3406.CB6D49DB@lemburg.com> (mal@lemburg.com) References: <200112271506.fBRF6uF26242@spinaker.ins.cwi.nl> <200112271736.fBRHaom02848@mira.informatik.hu-berlin.de> <3C2C3406.CB6D49DB@lemburg.com> Message-ID: <200112281013.fBSADHu01819@mira.informatik.hu-berlin.de> [Over to python-dev. Jaco noticed that writing Unicode objects to a StringIO object stopped working in 2.2, see http://mail.python.org/pipermail/xml-sig/2001-December/006891.html ] Marc-Andre writes > Actually, I think that this is a bug in the documentation, not the > code. StringIO and cStringIO were never meant to work on anything but > strings and memory buffers. IMO, "strings" should include both byte strings and Unicode strings. Mixing them may not be allowed, but that is a different story. In fact, there is an open bug (#216388) that cStringIO rejects Unicode objects. If that gets fixed, we get the funny scenario that StringIO rejects Unicode object, whereas cStringIO accepts them. > The note that Fred added to the docs about StringIO's capability of > storing Unicode in it's buffer list is simply an artifact of the > implementation. There are many developers who take this note literally. Claiming that this was not intentional is a mistake. > Please use the .encode() method on Unicode objects before writing > them to a StringIO object. If you want to end up with a byte string, this is a good idea. But I think it is pointless to require encoding them when you want to end up with a Unicode string; you'd have to invoke unicode() on the result, for no apparent reason but a bug in the StringIO implementation. Regards, Martin From mal@lemburg.com Fri Dec 28 10:50:25 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 28 Dec 2001 11:50:25 +0100 Subject: [Python-Dev] Re: [XML-SIG] printing Unicode xml to StringIO References: <200112271506.fBRF6uF26242@spinaker.ins.cwi.nl> <200112271736.fBRHaom02848@mira.informatik.hu-berlin.de> <3C2C3406.CB6D49DB@lemburg.com> <200112281013.fBSADHu01819@mira.informatik.hu-berlin.de> Message-ID: <3C2C4E71.2BA58BF3@lemburg.com> "Martin v. Loewis" wrote: > > [Over to python-dev. Jaco noticed that writing Unicode objects to > a StringIO object stopped working in 2.2, see > http://mail.python.org/pipermail/xml-sig/2001-December/006891.html > ] > > Marc-Andre writes > > Actually, I think that this is a bug in the documentation, not the > > code. StringIO and cStringIO were never meant to work on anything but > > strings and memory buffers. > > IMO, "strings" should include both byte strings and Unicode strings. > Mixing them may not be allowed, but that is a different story. > > In fact, there is an open bug (#216388) that cStringIO rejects Unicode > objects. If that gets fixed, we get the funny scenario that StringIO > rejects Unicode object, whereas cStringIO accepts them. StringIO and cStringIO use different methods for storing the snippets: StringIO makes use of a buffer list which gets compressed every now and then, while cStringIO uses a raw memory buffer for this purpose. Both of these implementation are targetted at providing a file IO like interface to in-memory "files". Since Python file object don't magically support Unicode, I wonder where the idea came from that StringIO/cStringIO should. That patch I applied to StringIO/cStringIO for 2.2 was aimed at making these two more compatible to the standard Python file object. The latter uses the "s#" parser marker for .write() and thus can also accept memory buffers. This was previously not possible with either of the two StringIO implementation (StringIO.py failed when trying to join different buffer compatible objects, cStringIO only accepted real string objects). > > The note that Fred added to the docs about StringIO's capability of > > storing Unicode in it's buffer list is simply an artifact of the > > implementation. > > There are many developers who take this note literally. Claiming that > this was not intentional is a mistake. > > > Please use the .encode() method on Unicode objects before writing > > them to a StringIO object. > > If you want to end up with a byte string, this is a good idea. That's the idea behind StringIO objects... they are in-memory file object emulators. > But I > think it is pointless to require encoding them when you want to end up > with a Unicode string; you'd have to invoke unicode() on the result, > for no apparent reason but a bug in the StringIO implementation. This is a different application. It should be easy enough to subclass StringIO as UnicodeIO class and then have this class implement fast Unicode snippet joining. I'm not sure whether the same can be done with cStringIO's type. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From martin@v.loewis.de Fri Dec 28 11:04:30 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Fri, 28 Dec 2001 12:04:30 +0100 Subject: [Python-Dev] Re: [XML-SIG] printing Unicode xml to StringIO In-Reply-To: <3C2C4E71.2BA58BF3@lemburg.com> (mal@lemburg.com) References: <200112271506.fBRF6uF26242@spinaker.ins.cwi.nl> <200112271736.fBRHaom02848@mira.informatik.hu-berlin.de> <3C2C3406.CB6D49DB@lemburg.com> <200112281013.fBSADHu01819@mira.informatik.hu-berlin.de> <3C2C4E71.2BA58BF3@lemburg.com> Message-ID: <200112281104.fBSB4UI02359@mira.informatik.hu-berlin.de> From martin@v.loewis.de Fri Dec 28 11:25:46 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Fri, 28 Dec 2001 12:25:46 +0100 Subject: [Python-Dev] Re: [XML-SIG] printing Unicode xml to StringIO In-Reply-To: <3C2C4E71.2BA58BF3@lemburg.com> (mal@lemburg.com) References: <200112271506.fBRF6uF26242@spinaker.ins.cwi.nl> <200112271736.fBRHaom02848@mira.informatik.hu-berlin.de> <3C2C3406.CB6D49DB@lemburg.com> <200112281013.fBSADHu01819@mira.informatik.hu-berlin.de> <3C2C4E71.2BA58BF3@lemburg.com> Message-ID: <200112281125.fBSBPkC02403@mira.informatik.hu-berlin.de> > Both of these implementation are targetted at providing > a file IO like interface to in-memory "files". Since Python > file object don't magically support Unicode, I wonder where the > idea came from that StringIO/cStringIO should. The exact source of this idea is unknown. However, there are many early references to it: - codecs.open returns an "encoded file" which "will only accept ... Unicode objects". That is perhaps the earliest precedent of a file object supporting Unicode. - At some point in time, you said that it is a bug that cStringIO does not support Unicode strings, see http://mail.python.org/pipermail/i18n-sig/2000-November/000550.html - the documentation of StringIO suggests that they should accept Unicode. So I would not blame the users for adopting far-off ideas, when the Python core itself suggests that these ideas are Pythonic. > That patch I applied to StringIO/cStringIO for 2.2 was > aimed at making these two more compatible to the standard > Python file object. The latter uses the "s#" parser > marker for .write() and thus can also accept memory > buffers. This was previously not possible with either > of the two StringIO implementation (StringIO.py failed > when trying to join different buffer compatible objects, > cStringIO only accepted real string objects). There is nothing wrong with that. The patch should just have special-cased Unicode objects (and that bug can still be corrected). Regards, Martin From guido@python.org Fri Dec 28 14:20:18 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 28 Dec 2001 09:20:18 -0500 Subject: [Python-Dev] Re: [XML-SIG] printing Unicode xml to StringIO In-Reply-To: Your message of "Fri, 28 Dec 2001 12:25:46 +0100." <200112281125.fBSBPkC02403@mira.informatik.hu-berlin.de> References: <200112271506.fBRF6uF26242@spinaker.ins.cwi.nl> <200112271736.fBRHaom02848@mira.informatik.hu-berlin.de> <3C2C3406.CB6D49DB@lemburg.com> <200112281013.fBSADHu01819@mira.informatik.hu-berlin.de> <3C2C4E71.2BA58BF3@lemburg.com> <200112281125.fBSBPkC02403@mira.informatik.hu-berlin.de> Message-ID: <200112281420.JAA22961@cj20424-a.reston1.va.home.com> Whoa! - Since we added a note to the docs that StringIO supports Unicode, we clearly should continue to support that, and it's a bug if it doesn't. - OTOH, Unicode for cStringIO should be considered at best a feature request. I don't mind if cStringIO doesn't support Unicode -- it never has, AFAIK, so it won't break much code. I don't believe it's much faster than StringIO, unless you use the C API (like cPickle does). - Of course, when Unicode is supported, mixing ASCII and Unicode should be supported too. (But not necessarily mixing 8-bit strings containing characters in the range \200-\377, since there's no default encoding for this range.) - Since this changed from 2.1 to 2.2, we should restore this capability in 2.2.1; I would say that 2.2.1 can't go out until this is fixed. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Fri Dec 28 15:30:34 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 28 Dec 2001 16:30:34 +0100 Subject: [Python-Dev] Re: [XML-SIG] printing Unicode xml to StringIO References: <200112271506.fBRF6uF26242@spinaker.ins.cwi.nl> <200112271736.fBRHaom02848@mira.informatik.hu-berlin.de> <3C2C3406.CB6D49DB@lemburg.com> <200112281013.fBSADHu01819@mira.informatik.hu-berlin.de> <3C2C4E71.2BA58BF3@lemburg.com> <200112281125.fBSBPkC02403@mira.informatik.hu-berlin.de> <200112281420.JAA22961@cj20424-a.reston1.va.home.com> Message-ID: <3C2C901A.D61001D9@lemburg.com> Guido van Rossum wrote: > > Whoa! > > - Since we added a note to the docs that StringIO supports Unicode, we > clearly should continue to support that, and it's a bug if it > doesn't. I still believe that the docs are wrong, but nevermind. I'll fix StringIO.py to continue to support Unicode in addition to strings and buffer objects. It's basically only about special casing Unicode in the .write() method. BTW, I was never aware of the doc changes in this area and the test suite didn't bring up the issues either. > - OTOH, Unicode for cStringIO should be considered at best a feature > request. I don't mind if cStringIO doesn't support Unicode -- it > never has, AFAIK, so it won't break much code. I don't believe it's > much faster than StringIO, unless you use the C API (like cPickle > does). Unicode support in cStringIO would require a new implementation since the machinery uses raw byte buffers. > - Of course, when Unicode is supported, mixing ASCII and Unicode > should be supported too. (But not necessarily mixing 8-bit strings > containing characters in the range \200-\377, since there's no > default encoding for this range.) In StringIO.py this is not much of a problem since it uses a list of snippets. Note that this is also why StringIO.py "supported" Unicode in the first place (and that's why I think it was more an artifact of the implementation than true intent). > - Since this changed from 2.1 to 2.2, we should restore this > capability in 2.2.1; I would say that 2.2.1 can't go out until this > is fixed. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Fri Dec 28 15:27:02 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 28 Dec 2001 10:27:02 -0500 Subject: [Python-Dev] Re: [XML-SIG] printing Unicode xml to StringIO In-Reply-To: Your message of "Fri, 28 Dec 2001 16:30:34 +0100." <3C2C901A.D61001D9@lemburg.com> References: <200112271506.fBRF6uF26242@spinaker.ins.cwi.nl> <200112271736.fBRHaom02848@mira.informatik.hu-berlin.de> <3C2C3406.CB6D49DB@lemburg.com> <200112281013.fBSADHu01819@mira.informatik.hu-berlin.de> <3C2C4E71.2BA58BF3@lemburg.com> <200112281125.fBSBPkC02403@mira.informatik.hu-berlin.de> <200112281420.JAA22961@cj20424-a.reston1.va.home.com> <3C2C901A.D61001D9@lemburg.com> Message-ID: <200112281527.KAA23737@cj20424-a.reston1.va.home.com> > > - Since we added a note to the docs that StringIO supports Unicode, we > > clearly should continue to support that, and it's a bug if it > > doesn't. > > I still believe that the docs are wrong, but nevermind. I'll fix > StringIO.py to continue to support Unicode in addition to strings > and buffer objects. It's basically only about special casing > Unicode in the .write() method. Thanks. > BTW, I was never aware of the doc changes in this area and the > test suite didn't bring up the issues either. Can you please add something to the test suite that makes sure this feature works? > > - OTOH, Unicode for cStringIO should be considered at best a feature > > request. I don't mind if cStringIO doesn't support Unicode -- it > > never has, AFAIK, so it won't break much code. I don't believe it's > > much faster than StringIO, unless you use the C API (like cPickle > > does). > > Unicode support in cStringIO would require a new implementation > since the machinery uses raw byte buffers. That's why I don't care much about it. :-) > > - Of course, when Unicode is supported, mixing ASCII and Unicode > > should be supported too. (But not necessarily mixing 8-bit strings > > containing characters in the range \200-\377, since there's no > > default encoding for this range.) > > In StringIO.py this is not much of a problem since it uses > a list of snippets. Note that this is also why StringIO.py "supported" > Unicode in the first place (and that's why I think it was more an > artifact of the implementation than true intent). But it was useful! :-) > > - Since this changed from 2.1 to 2.2, we should restore this > > capability in 2.2.1; I would say that 2.2.1 can't go out until this > > is fixed. Try to mark the checkin messages as "2.2.1 bugfix", for the 2.2.1 patch czar. --Guido van Rossum (home page: http://www.python.org/~guido/) From martin@v.loewis.de Fri Dec 28 17:02:17 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Fri, 28 Dec 2001 18:02:17 +0100 Subject: [Python-Dev] Re: [XML-SIG] printing Unicode xml to StringIO In-Reply-To: <3C2C901A.D61001D9@lemburg.com> (mal@lemburg.com) References: <200112271506.fBRF6uF26242@spinaker.ins.cwi.nl> <200112271736.fBRHaom02848@mira.informatik.hu-berlin.de> <3C2C3406.CB6D49DB@lemburg.com> <200112281013.fBSADHu01819@mira.informatik.hu-berlin.de> <3C2C4E71.2BA58BF3@lemburg.com> <200112281125.fBSBPkC02403@mira.informatik.hu-berlin.de> <200112281420.JAA22961@cj20424-a.reston1.va.home.com> <3C2C901A.D61001D9@lemburg.com> Message-ID: <200112281702.fBSH2Ha04205@mira.informatik.hu-berlin.de> > Unicode support in cStringIO would require a new implementation > since the machinery uses raw byte buffers. Not necessarily. You could add a flag saying that whether those bytes represent Unicode strings or not. If they do, you pick an encoding of your choice (perhaps unicode-internal), and convert the existing bytes to that encoding on first sighting of a Unicode string (assuming that everything so far is in the system encoding). When returning the bytes to the user, you build a Unicode object if the flag is set. Of course, you'd still have to touch every method, to analyse the flag... Regards, Martin From Jack.Jansen@cwi.nl Mon Dec 31 13:39:09 2001 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Mon, 31 Dec 2001 14:39:09 +0100 Subject: [Python-Dev] Offline for a while Message-ID: <200112311339.fBVDd9O16531@spruit.ins.cwi.nl> Folks, I will probably be unreacable by mail for the next few days, the backup we put in for our move seems to be non-functional. So don't worry about bounceds to the oratrix.nl address, all should be well again on wednseday or thusrday (and backspace also doesn't work on this machine:-) Jack