From collinw at gmail.com Thu Mar 1 00:32:17 2007 From: collinw at gmail.com (Collin Winter) Date: Wed, 28 Feb 2007 17:32:17 -0600 Subject: [Python-Dev] with_traceback In-Reply-To: <45E37D41.8000005@canterbury.ac.nz> References: <5.1.1.6.0.20070226182128.02651388@sparrow.telecommunity.com> <45E37D41.8000005@canterbury.ac.nz> Message-ID: <43aa6ff70702281532k2bc8a5fs65056d13980cb084@mail.gmail.com> On 2/26/07, Greg Ewing wrote: > Phillip J. Eby wrote: > > At 03:38 PM 2/26/2007 -0700, Andrew Dalke wrote: > > > > > NO_END_OF_RECORD = ParserError("Cannot find end of record") > > > > Then don't do that, as it's bad style for Python 3.x. ;-) > > I don't like that answer. I can think of legitimate > reasons for wanting to pre-create exceptions, e.g. if > I'm intending to raise and catch a particular exception > frequently and I don't want the overhead of creating > a new instance each time. Is this really the problem it's being made out to be? I'm guessing the use-case you're suggesting is where certain exceptions are raised and caught inside a library or application, places where the exceptions will never reach the user. If that's the case, does it really matter what the traceback looks like? > For me, this is casting serious doubt on the whole > idea of attaching the traceback to the exception. If attaching the traceback to the exception is bothering you, you should take a look at the other attributes PEP 344 introduces: __cause__ and __context__. I'd say what needs another look is the idea of pre-creating a single exception instance and repeatedly raising it. Collin Winter From greg.ewing at canterbury.ac.nz Thu Mar 1 00:45:17 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Mar 2007 12:45:17 +1300 Subject: [Python-Dev] with_traceback In-Reply-To: <20070228054923.17852.893320900.divmod.quotient.712@ohm> References: <20070228054923.17852.893320900.divmod.quotient.712@ohm> Message-ID: <45E6140D.4090300@canterbury.ac.nz> glyph at divmod.com wrote: > Or modify __new__ on your particular heavily-optimized > exception to have a free-list, Doing that in Python is likely to have as much overhead as creating an instance. The simple and obvious optimisation is to pre-create the instance, but we're proposing to make the obvious way wrong for subtle reasons. That doesn't seem pythonic to me. -- Greg From greg.ewing at canterbury.ac.nz Thu Mar 1 00:57:01 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Mar 2007 12:57:01 +1300 Subject: [Python-Dev] with_traceback In-Reply-To: References: <45E37D41.8000005@canterbury.ac.nz> <20070228054923.17852.893320900.divmod.quotient.712@ohm> Message-ID: <45E616CD.9050502@canterbury.ac.nz> Adam Olsen wrote: > It sounds like we should always copy the exception given to raise, I don't like that either, for all the reasons that make it infeasible to copy an arbitrary object in a general way. -- Greg From rhamph at gmail.com Thu Mar 1 02:29:11 2007 From: rhamph at gmail.com (Adam Olsen) Date: Wed, 28 Feb 2007 18:29:11 -0700 Subject: [Python-Dev] with_traceback In-Reply-To: <45E616CD.9050502@canterbury.ac.nz> References: <45E37D41.8000005@canterbury.ac.nz> <20070228054923.17852.893320900.divmod.quotient.712@ohm> <45E616CD.9050502@canterbury.ac.nz> Message-ID: On 2/28/07, Greg Ewing wrote: > Adam Olsen wrote: > > > It sounds like we should always copy the exception given to raise, > > I don't like that either, for all the reasons that > make it infeasible to copy an arbitrary object in a > general way. Exceptions aren't arbitrary objects though. The requirement that they inherit from BaseException is specifically to create a common interface. Copying would be an extension of that interface. I believe calling copy.copy() would be sufficient. -- Adam Olsen, aka Rhamphoryncus From exarkun at divmod.com Thu Mar 1 02:58:24 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Wed, 28 Feb 2007 20:58:24 -0500 Subject: [Python-Dev] with_traceback In-Reply-To: Message-ID: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> On Wed, 28 Feb 2007 18:29:11 -0700, Adam Olsen wrote: >On 2/28/07, Greg Ewing wrote: >> Adam Olsen wrote: >> >> > It sounds like we should always copy the exception given to raise, >> >> I don't like that either, for all the reasons that >> make it infeasible to copy an arbitrary object in a >> general way. > >Exceptions aren't arbitrary objects though. The requirement that they >inherit from BaseException is specifically to create a common >interface. Copying would be an extension of that interface. > >I believe calling copy.copy() would be sufficient. > Does copying every exception given to `raise' solve the problem being discussed here? Consider the current Python behavior: no copying is performed, most code instantiates a new exception instance for each raise statement, some code creates a single exception and re-raises it repeatedly. And the new behavior? Every raise statement copies an exception instance, some code will create a new exception instance for each raise statement, some code will create a single exception and re-raise it repeatedly. That doesn't sound like an improvement to me. Normal code will be more wasteful. Code which the author has gone out of his way to tune will be as wasteful as /average/ code currently is, and more wasteful than tuned code now is. Plus you now have the added mental burden of keeping track of which objects are copies of what (and if you throw in the refcount=1 optimization, then this burden is increased - was something accidentally relying on copying or non-copying? Did a debugger grab a reference to the exception object, thus changing the programs behavior? Did a third-party hang on to an exception for longer than the raising code expected? etc). Jean-Paul From guido at python.org Thu Mar 1 03:10:21 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 28 Feb 2007 18:10:21 -0800 Subject: [Python-Dev] with_traceback In-Reply-To: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> Message-ID: I am beginning to think that there are serious problems with attaching the traceback to the exception; I really don't like the answer that pre-creating an exception is unpythonic in Py3k. On 2/28/07, Jean-Paul Calderone wrote: > On Wed, 28 Feb 2007 18:29:11 -0700, Adam Olsen wrote: > >On 2/28/07, Greg Ewing wrote: > >> Adam Olsen wrote: > >> > >> > It sounds like we should always copy the exception given to raise, > >> > >> I don't like that either, for all the reasons that > >> make it infeasible to copy an arbitrary object in a > >> general way. > > > >Exceptions aren't arbitrary objects though. The requirement that they > >inherit from BaseException is specifically to create a common > >interface. Copying would be an extension of that interface. > > > >I believe calling copy.copy() would be sufficient. > > > > Does copying every exception given to `raise' solve the problem being > discussed here? > > Consider the current Python behavior: no copying is performed, most code > instantiates a new exception instance for each raise statement, some code > creates a single exception and re-raises it repeatedly. > > And the new behavior? Every raise statement copies an exception instance, > some code will create a new exception instance for each raise statement, > some code will create a single exception and re-raise it repeatedly. > > That doesn't sound like an improvement to me. Normal code will be more > wasteful. Code which the author has gone out of his way to tune will be > as wasteful as /average/ code currently is, and more wasteful than tuned > code now is. > > Plus you now have the added mental burden of keeping track of which objects > are copies of what (and if you throw in the refcount=1 optimization, then > this burden is increased - was something accidentally relying on copying or > non-copying? Did a debugger grab a reference to the exception object, thus > changing the programs behavior? Did a third-party hang on to an exception > for longer than the raising code expected? etc). > > Jean-Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Thu Mar 1 03:36:33 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Mar 2007 15:36:33 +1300 Subject: [Python-Dev] with_traceback In-Reply-To: References: <45E37D41.8000005@canterbury.ac.nz> <20070228054923.17852.893320900.divmod.quotient.712@ohm> <45E616CD.9050502@canterbury.ac.nz> Message-ID: <45E63C31.5050009@canterbury.ac.nz> Adam Olsen wrote: > Exceptions aren't arbitrary objects though. The requirement that they > inherit from BaseException is specifically to create a common > interface. But that doesn't tell you enough. If the exception references some other object, should you copy it? You can't tell just from the fact that it inherits from BaseException. Besides, making a copy of the exception seems just as expensive as creating a new instance, so it does nothing to address the efficiency issue. Maybe it's not as important as I feel it is, but I like the way that exception raising is lightweight enough to use for flow control. When used that way, creating a new instance each time seems wasteful. I accept the overhead because I know that if it were ever a problem I could eliminate it by pre-creating the instance. I'd be disappointed to lose that ability. > I believe calling copy.copy() would be sufficient. I never use that, because I have no confidence that it would DWIM. I'd be unhappy if the system started relying on it anywhere fundamental. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From rhamph at gmail.com Thu Mar 1 03:36:39 2007 From: rhamph at gmail.com (Adam Olsen) Date: Wed, 28 Feb 2007 19:36:39 -0700 Subject: [Python-Dev] with_traceback In-Reply-To: References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> Message-ID: On 2/28/07, Guido van Rossum wrote: > I am beginning to think that there are serious problems with attaching > the traceback to the exception; I really don't like the answer that > pre-creating an exception is unpythonic in Py3k. How plausible would it be to optimize all exception instantiation? Perhaps use slots and a freelist for everything inheriting from BaseException and not inheriting from other builtin types? > On 2/28/07, Jean-Paul Calderone wrote: > > On Wed, 28 Feb 2007 18:29:11 -0700, Adam Olsen wrote: > > > > > >I believe calling copy.copy() would be sufficient. > > > > > That doesn't sound like an improvement to me. Normal code will be more > > wasteful. Code which the author has gone out of his way to tune will be > > as wasteful as /average/ code currently is, and more wasteful than tuned > > code now is. > > -- Adam Olsen, aka Rhamphoryncus From greg.ewing at canterbury.ac.nz Thu Mar 1 03:42:00 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Mar 2007 15:42:00 +1300 Subject: [Python-Dev] with_traceback In-Reply-To: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> Message-ID: <45E63D78.1020700@canterbury.ac.nz> Jean-Paul Calderone wrote: > And the new behavior? Every raise statement copies an exception instance, > some code will create a new exception instance for each raise statement, > some code will create a single exception and re-raise it repeatedly. Make that "most code will create a new exception instance and then make a copy of it", unless this can be optimised away somehow, and it's not necessarily obvious that the refcount == 1 trick will work (it depends on the exact details of how the references flow through the exception raising machinery). -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From greg.ewing at canterbury.ac.nz Thu Mar 1 03:52:33 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Mar 2007 15:52:33 +1300 Subject: [Python-Dev] with_traceback In-Reply-To: References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> Message-ID: <45E63FF1.6030503@canterbury.ac.nz> Adam Olsen wrote: > How plausible would it be to optimize all exception instantiation? > Perhaps use slots and a freelist for everything inheriting from > BaseException and not inheriting from other builtin types? I'm not sure a free list would help much for instances of user define classes, since creating one involves setting up a dict, etc. And if you use __slots__ you end up with objects of different sizes, which isn't free-list-friendly. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From rhamph at gmail.com Thu Mar 1 03:55:29 2007 From: rhamph at gmail.com (Adam Olsen) Date: Wed, 28 Feb 2007 19:55:29 -0700 Subject: [Python-Dev] with_traceback In-Reply-To: <45E63FF1.6030503@canterbury.ac.nz> References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <45E63FF1.6030503@canterbury.ac.nz> Message-ID: On 2/28/07, Greg Ewing wrote: > Adam Olsen wrote: > > > How plausible would it be to optimize all exception instantiation? > > Perhaps use slots and a freelist for everything inheriting from > > BaseException and not inheriting from other builtin types? > > I'm not sure a free list would help much for instances > of user define classes, since creating one involves setting > up a dict, etc. And if you use __slots__ you end up with > objects of different sizes, which isn't free-list-friendly. Not easy, but doable. Perhaps a plan B if nobody comes up with a plan A. -- Adam Olsen, aka Rhamphoryncus From kbk at shore.net Thu Mar 1 05:02:12 2007 From: kbk at shore.net (Kurt B. Kaiser) Date: Wed, 28 Feb 2007 23:02:12 -0500 (EST) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200703010402.l2142C59003869@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 416 open ( +8) / 3593 closed ( +8) / 4009 total (+16) Bugs : 974 open ( +6) / 6520 closed (+15) / 7494 total (+21) RFE : 268 open ( +1) / 251 closed ( +0) / 519 total ( +1) New / Reopened Patches ______________________ Allow specifying headers for MIME parts (2007-02-23) http://python.org/sf/1666625 opened by J?rg Sonnenberger Time zone-capable variant of time.localtime (2007-02-24) http://python.org/sf/1667546 opened by Paul Boddie urllib2 raises an UnboundLocalError if "auth-int" is the qop (2007-02-24) http://python.org/sf/1667860 opened by Atul Varma urllib2.urlopen() raises OSError instead of URLError (2007-02-24) http://python.org/sf/1668100 opened by Jerry Seutter Fix for 767111, 'AttributeError thrown by urllib.open_http' (2007-02-25) http://python.org/sf/1668132 opened by Atul Varma don't use '-' and '_' in mkstemp (2007-02-25) http://python.org/sf/1668482 opened by Arvin Schnell bytes.fromhex() (2007-02-26) CLOSED http://python.org/sf/1669379 opened by Georg Brandl subprocess: Support close_fds on Win32 (2007-02-26) http://python.org/sf/1669481 opened by Jon Foster Change (fix!) os.path.isabs() semantics on Win32 (2007-02-26) http://python.org/sf/1669539 opened by Jon Foster methods for bytes (2007-02-27) CLOSED http://python.org/sf/1669633 opened by Pete Shinners Remove Py_PROTO from socket in py3k (2007-02-27) CLOSED http://python.org/sf/1670209 opened by Pete Shinners email.Generator: no header wrapping for multipart/signed (2007-02-28) http://python.org/sf/1670765 opened by Martin von Gagern Refactor test_threadedtempfile.py to use unittest. (2007-02-28) http://python.org/sf/1670993 opened by Jerry Seutter Class Decorators (2007-02-28) http://python.org/sf/1671208 opened by Jack Diederich Refactor test_class to use unittest lib (2007-02-28) http://python.org/sf/1671298 opened by Mike Verdone New File I/O type for Python 3000, plus .h and unit tests (2007-02-28) http://python.org/sf/1671314 opened by Daniel Stutzbach Patches Closed ______________ setuptools: avoid sets module for python>2.3 (2007-02-19) http://python.org/sf/1663226 closed by pje documentation for element interface (2007-02-11) http://python.org/sf/1657613 closed by fdrake fast subclasses of builtin types (2006-12-28) http://python.org/sf/1624059 closed by nnorwitz bytes.fromhex() (2007-02-26) http://python.org/sf/1669379 closed by gbrandl Optional Argument Syntax (2006-12-02) http://python.org/sf/1607548 closed by gvanrossum methods for bytes (2007-02-26) http://python.org/sf/1669633 closed by nnorwitz Remove Py_PROTO from socket in py3k (2007-02-27) http://python.org/sf/1670209 closed by nnorwitz The Unicode "lazy strings" patches (2007-01-06) http://python.org/sf/1629305 closed by gvanrossum New / Reopened Bugs ___________________ Calling tparm from extension lib fails in Python 2.5 (2007-02-13) http://python.org/sf/1659171 reopened by richyk shutil.copytree doesn't preserve directory permissions (2007-02-22) http://python.org/sf/1666318 opened by Jeff McNeil Incorrect file path reported by inspect.getabsfile() (2007-02-23) http://python.org/sf/1666807 opened by Fernando P?rez terminalcommand doesn't work under Darwin (2007-02-23) http://python.org/sf/1666952 opened by Jurjen N.E. Bos Install fails with no error (2007-02-24) http://python.org/sf/1667877 reopened by widgeteye Install fails with no error (2007-02-24) http://python.org/sf/1667877 opened by larry PyMem_Realloc docs don't specifiy out-of-mem behavior (2007-02-24) http://python.org/sf/1668032 opened by Daniel Stutzbach PyMem_Resize docs don't specify that it modifies an argument (2007-02-24) http://python.org/sf/1668036 opened by Daniel Stutzbach python-2.4.4 on freebsd-6: _curses extension doesn't build (2007-02-25) CLOSED http://python.org/sf/1668133 opened by clemens fischer Strange unicode behaviour (2007-02-25) CLOSED http://python.org/sf/1668295 reopened by sgala Strange unicode behaviour (2007-02-25) CLOSED http://python.org/sf/1668295 opened by Santiago Gala I can't change attribute __op__ in new-style classes (2007-02-25) CLOSED http://python.org/sf/1668540 opened by netimen inspect.getargspec() fails with keyword-only arguments (2007-02-25) http://python.org/sf/1668565 opened by Brett Cannon distutils chops the first character of filenames (2007-02-25) http://python.org/sf/1668596 opened by Sam Pointon PyErr_WriteUnraisable lacks exception type check (2007-02-26) CLOSED http://python.org/sf/1669182 opened by Gabriel Becedillas Clarify PyMem_Realloc and PyMem_Resize docs (2007-02-26) http://python.org/sf/1669304 opened by Steven Bethard document that shutil.copyfileobj does not seek() (2007-02-26) http://python.org/sf/1669331 opened by Steven Bethard make install fails if no previous Python installation (2007-02-26) http://python.org/sf/1669349 opened by Matthias S. Benkmann 2.4.4 Logging LogRecord attributes broken (2007-02-26) http://python.org/sf/1669498 opened by Glenn Murray 2.4.4 Logging LogRecord attributes broken (2007-02-26) CLOSED http://python.org/sf/1669578 opened by Glenn Murray Some Compiler Warnings on VC6 (2007-02-27) http://python.org/sf/1669637 opened by Hirokazu Yamamoto 2.4.4 Logging LogRecord attributes broken (2007-02-26) CLOSED http://python.org/sf/1669646 opened by Glenn Murray Python needs a way to detect implementation (2007-02-27) http://python.org/sf/1669743 opened by Kay Hayen slice obj with no start index is 0 instead of None sometimes (2007-02-28) http://python.org/sf/1671137 opened by Mike Verdone Bugs Closed ___________ finditer stuck in infinite loop (2007-02-16) http://python.org/sf/1661745 closed by gbrandl python-2.4.4 on freebsd-6: _curses extension doesn't build (2007-02-25) http://python.org/sf/1668133 closed by loewis Strange unicode behaviour (2007-02-25) http://python.org/sf/1668295 closed by gbrandl Strange unicode behaviour (2007-02-25) http://python.org/sf/1668295 closed by gbrandl crash in exec statement if uncode filename cannot be decoded (2007-02-21) http://python.org/sf/1664966 closed by jhylton sys.settrace cause curried parms to show up as attributes (2006-10-02) http://python.org/sf/1569356 closed by jhylton I can't change attribute __op__ in new-style classes (2007-02-25) http://python.org/sf/1668540 closed by gbrandl nested variables in 'class:' statements (2003-09-22) http://python.org/sf/810714 closed by jhylton os.wait child process fail when under stress (2007-02-13) http://python.org/sf/1658959 closed by thegroff mention side-lists from python-dev description (2007-01-03) http://python.org/sf/1627039 closed by jhylton minor inconsistency in socket.close (2006-12-22) http://python.org/sf/1620945 closed by jhylton PyErr_WriteUnraisable lacks exception type check (2007-02-26) http://python.org/sf/1669182 closed by nnorwitz 2.4.4 Logging LogRecord attributes broken (2007-02-26) http://python.org/sf/1669578 closed by nnorwitz 2.4.4 Logging LogRecord attributes broken (2007-02-26) http://python.org/sf/1669646 closed by nnorwitz New / Reopened RFE __________________ isinstance.__doc__ is somewhat irritating (2007-02-27) http://python.org/sf/1670167 opened by Ren? Fleschenberg From foom at fuhm.net Thu Mar 1 05:15:24 2007 From: foom at fuhm.net (James Y Knight) Date: Wed, 28 Feb 2007 23:15:24 -0500 Subject: [Python-Dev] with_traceback In-Reply-To: References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> Message-ID: <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> On Feb 28, 2007, at 9:10 PM, Guido van Rossum wrote: > I am beginning to think that there are serious problems with attaching > the traceback to the exception; I really don't like the answer that > pre-creating an exception is unpythonic in Py3k. I'll say up front that I haven't been paying as much attention to the topic of exception behavior as perhaps I should before attempting to contribute to a thread about it...but... It seems to me that a stack trace should always be attached to an exception object at creation time of the exception, and never at any other time. Then, if someone pre-creates an exception object, they get consistent and easily explainable behavior (the traceback to the creation point). The traceback won't necessarily be *useful*, but presumably someone pre-creating an exception object did so to save run-time, and creating the traceback is generally very expensive, so doing that only once, too, seems like a win to me. FWIW, that's basically how exceptions work in Java. From http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Throwable.html: > Instances of two subclasses, Error and Exception, are > conventionally used to indicate that exceptional situations have > occurred. Typically, these instances are freshly created in the > context of the exceptional situation so as to include relevant > information (such as stack trace data). > A throwable contains a snapshot of the execution stack of its > thread at the time it was created. It can also contain a message > string that gives more information about the error. Finally, it can > contain a cause: another throwable that caused this throwable to > get thrown. The cause facility is new in release 1.4. It is also > known as the chained exception facility, as the cause can, itself, > have a cause, and so on, leading to a "chain" of exceptions, each > caused by another. There's probably a million reasons why this doesn't work for python, but they don't immediately jump out at me. :) Migration from 2.X to 3.X would consist of recommending not to create an exception outside of a raise line, unless you're okay with the traceback location changing from the raise point to the creation point. James From dalke at dalkescientific.com Thu Mar 1 06:56:31 2007 From: dalke at dalkescientific.com (Andrew Dalke) Date: Wed, 28 Feb 2007 22:56:31 -0700 Subject: [Python-Dev] with_traceback In-Reply-To: <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> Message-ID: On 2/28/07, James Y Knight wrote: > It seems to me that a stack trace should always be attached to an > exception object at creation time of the exception, and never at any > other time. Then, if someone pre-creates an exception object, they > get consistent and easily explainable behavior (the traceback to the > creation point). The traceback won't necessarily be *useful*, but > presumably someone pre-creating an exception object did so to save > run-time, and creating the traceback is generally very expensive, so > doing that only once, too, seems like a win to me. The only example I found in about 2 dozen packages where the exception was precreated was in pyparsing. I don't know the reason why it was done that way, but I'm certain it wasn't for performance. The exception is created as part of the format definition. In that case if the traceback is important then it's important to know which code was attempting the parse. The format definition was probably certainly done at module import time. In any case, reraising the same exception instance is a rare construct in current Python code. PJE had never seen it before. It's hard to get a good intuition from zero data points. :) From fuzzyman at voidspace.org.uk Thu Mar 1 07:08:26 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 01 Mar 2007 06:08:26 +0000 Subject: [Python-Dev] with_traceback In-Reply-To: References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> Message-ID: <45E66DDA.8090406@voidspace.org.uk> Andrew Dalke wrote: > On 2/28/07, James Y Knight wrote: > >> It seems to me that a stack trace should always be attached to an >> exception object at creation time of the exception, and never at any >> other time. Sounds good in principle - but don't forget that normally the exception will be instantiated and *then* passed to the raise statement. I've never seen a module level exception instance before. With the proposed changes, modules that do this would *continue* to work, surely ? So they lose nothing (compared to the current situation) by having the traceback information overwritten, they just can't take direct advantage of the new attribute. Michael Foord From martin at v.loewis.de Thu Mar 1 07:21:54 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 01 Mar 2007 07:21:54 +0100 Subject: [Python-Dev] Windows compiler for Python 2.6+ In-Reply-To: <7790b6530702281424j5bd269u3a1e88dd844dbf1@mail.gmail.com> References: <7790b6530702281344n6122637fg1de2daf52e3ec67e@mail.gmail.com> <20070228135629.AE80.JCARLSON@uci.edu> <7790b6530702281424j5bd269u3a1e88dd844dbf1@mail.gmail.com> Message-ID: <45E67102.7020308@v.loewis.de> Chris AtLee schrieb: >> I would guess it is more an issue of 32bit + 64bit dynamic linking >> having issues, but I could certainly be wrong. > > I don't think so, this was the 64bit version of Python 2.5. When I > recompiled with the 2003 compiler it worked fine. Again, what 2003 compiler did you recompile with? There is none. Regards, Martin From glyph at divmod.com Thu Mar 1 07:53:19 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Thu, 1 Mar 2007 01:53:19 -0500 Subject: [Python-Dev] with_traceback In-Reply-To: Message-ID: <20070301065319.17852.1514562790.divmod.quotient.1256@ohm> On Wed, 28 Feb 2007 18:10:21 -0800, Guido van Rossum wrote: >I am beginning to think that there are serious problems with attaching >the traceback to the exception; I really don't like the answer that >pre-creating an exception is unpythonic in Py3k. In Twisted, to deal with asynchronous exceptions, we needed an object to specifically represent a "raised exception", i.e. an Exception instance with its attached traceback and methods to manipulate it. You can find its API here: http://twistedmatrix.com/documents/current/api/twisted.python.failure.Failure.html Perhaps the use-cases for attaching the traceback object to the exception would be better satisfied by simply having sys.exc_info() return an object with methods like Failure? Reading the "motivation" section of PEP 344, it describes "passing these three things in parallel" as "tedious and error-prone". Having one object one could call methods on instead of a 3-tuple which needed to be selectively passed on would simplify things. For example, chaining could be accomplished by doing something like this: sys.current_exc_thingy().chain() I can't think of a good name for the new object type, since "traceback", "error", "exception" and "stack" all already mean things in Python. From greg.ewing at canterbury.ac.nz Thu Mar 1 09:27:26 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Mar 2007 21:27:26 +1300 Subject: [Python-Dev] with_traceback In-Reply-To: <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> Message-ID: <45E68E6E.8050000@canterbury.ac.nz> James Y Knight wrote: > It seems to me that a stack trace should always be attached to an > exception object at creation time Um. Yes. Well, that's certainly an innovative solution... > The traceback won't necessarily be *useful*, Almost completely use*less*, I would have thought. The traceback is mostly used to find out where something went wrong, not where it went right (i.e. successful creation of the exception). > creating the traceback is generally very expensive, I don't think so -- isn't it just a linked list of existing stack frames? That should be very cheap to create. > From http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Throwable.html: > >>A throwable contains a snapshot of the execution stack of its >>thread at the time it was created. This would be a major and surprising change to Python users. It would also be considerably *more* expensive to implement than the current scheme, because it would require copying the entire stack, instead of just linking stack frames together as they are unwound during the search for an exception handler. -- Greg From greg.ewing at canterbury.ac.nz Thu Mar 1 09:32:19 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Mar 2007 21:32:19 +1300 Subject: [Python-Dev] with_traceback In-Reply-To: <45E66DDA.8090406@voidspace.org.uk> References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> <45E66DDA.8090406@voidspace.org.uk> Message-ID: <45E68F93.2080607@canterbury.ac.nz> Michael Foord wrote: > With the > proposed changes, modules that do this would *continue* to work, surely > ? Probably, but it might mean they were no longer thread safe. An exception caught and raised in one thread would be vulnerable to having its traceback clobbered by another thread raising the same instance. There's also the possibility of a traceback unexpectedly kept alive causing GC problems -- cycles, files not closed when you expect, etc. -- Greg From greg.ewing at canterbury.ac.nz Thu Mar 1 09:42:09 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Mar 2007 21:42:09 +1300 Subject: [Python-Dev] with_traceback In-Reply-To: <20070301065319.17852.1514562790.divmod.quotient.1256@ohm> References: <20070301065319.17852.1514562790.divmod.quotient.1256@ohm> Message-ID: <45E691E1.8030604@canterbury.ac.nz> glyph at divmod.com wrote: > Perhaps the use-cases for attaching the traceback object > to the exception would be better satisfied by simply having > sys.exc_info() return an object with methods like Failure? > I can't think of a good name for the new object type, Maybe we could call it a 'catch' (used as a noun, as when a fisherman might say "That's a good catch!") -- Greg From ncoghlan at gmail.com Thu Mar 1 11:44:52 2007 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 01 Mar 2007 20:44:52 +1000 Subject: [Python-Dev] PEP 306 changes (How to Change Python's Grammar) In-Reply-To: <20070228211757.GE5537@performancedrivers.com> References: <20070228211757.GE5537@performancedrivers.com> Message-ID: <45E6AEA4.9090203@gmail.com> Jack Diederich wrote: > __ Python/compile.c: You will need to create or modify the > compiler_* functions for your productions. Python/symtable.c will likely also need attention at this point (it handles the symbol collection pass that occurs before the actual compilation pass) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From guido at python.org Thu Mar 1 16:57:31 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 1 Mar 2007 07:57:31 -0800 Subject: [Python-Dev] PEP 306 changes (How to Change Python's Grammar) In-Reply-To: <45E6AEA4.9090203@gmail.com> References: <20070228211757.GE5537@performancedrivers.com> <45E6AEA4.9090203@gmail.com> Message-ID: Someone please check this in! On 3/1/07, Nick Coghlan wrote: > Jack Diederich wrote: > > __ Python/compile.c: You will need to create or modify the > > compiler_* functions for your productions. > > Python/symtable.c will likely also need attention at this point (it > handles the symbol collection pass that occurs before the actual > compilation pass) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > --------------------------------------------------------------- > http://www.boredomandlaziness.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From foom at fuhm.net Thu Mar 1 19:51:54 2007 From: foom at fuhm.net (James Y Knight) Date: Thu, 1 Mar 2007 13:51:54 -0500 Subject: [Python-Dev] with_traceback In-Reply-To: <45E68E6E.8050000@canterbury.ac.nz> References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> <45E68E6E.8050000@canterbury.ac.nz> Message-ID: <42F5EBBA-F88E-4ED0-B821-21249F69C441@fuhm.net> On Mar 1, 2007, at 3:27 AM, Greg Ewing wrote: > James Y Knight wrote: >> The traceback won't necessarily be *useful*, > > Almost completely use*less*, I would have thought. > The traceback is mostly used to find out where > something went wrong, not where it went right (i.e. > successful creation of the exception). The advantages are that it's an easily understandable and explainable behavior, and the traceback points you (the programmer) to the exact location where you went wrong: creating the exception at module level. Creating an exception with a non-exceptional stacktrace isn't always useless: sometimes you have exceptions where you know you never care about the stacktrace (internal flow control/etc). > This would be a major and surprising change to Python > users. It's only a major change if you don't raise the exception in the same place you create it. (which other people are claiming is extremely rare). > It would also be considerably *more* expensive to implement > than the current scheme, because it would require copying the > entire stack, instead of just linking stack frames together > as they are unwound during the search for an exception > handler. Yes of course, you're right, I withdraw the proposal. I had forgotten that python doesn't currently save the entire stack, only that between the 'raise' and the 'except'. (I had forgotten, because Twisted's "Failure" objects do save and print the entire stacktrace, both above and below the catch-point). James From guido at python.org Fri Mar 2 02:16:17 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 1 Mar 2007 17:16:17 -0800 Subject: [Python-Dev] Class destructor In-Reply-To: References: Message-ID: On 2/28/07, Nick Maclaren wrote: > I am gradually making progress with my binary floating-point software, > but have had to rewrite several times as I have forgotten most of the > details of how to do it! After 30 years, I can't say I am surprised. > > But I need to clean up workspace when a class (not object) is > deallocated. I can't easily use attributes, as people suggested, > because there is no anonymous storage built-in type. I could subvert > one of the existing storage types (buffer, string etc.), but that is > unclean. And I could write one, but that is excessive. Can you explain the reason for cleaning up in this scenario? Are you rapidly creating and destroying temporary class objects? Why can't you rely on the regular garbage collection process? Or does you class create an external resource like a temp file? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Mar 2 02:23:19 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 1 Mar 2007 17:23:19 -0800 Subject: [Python-Dev] with_traceback In-Reply-To: <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> Message-ID: On 2/28/07, James Y Knight wrote: > On Feb 28, 2007, at 9:10 PM, Guido van Rossum wrote: > > I am beginning to think that there are serious problems with attaching > > the traceback to the exception; I really don't like the answer that > > pre-creating an exception is unpythonic in Py3k. > > I'll say up front that I haven't been paying as much attention to the > topic of exception behavior as perhaps I should before attempting to > contribute to a thread about it...but... > > It seems to me that a stack trace should always be attached to an > exception object at creation time of the exception, and never at any > other time. Then, if someone pre-creates an exception object, they > get consistent and easily explainable behavior (the traceback to the > creation point). The traceback won't necessarily be *useful*, but > presumably someone pre-creating an exception object did so to save > run-time, and creating the traceback is generally very expensive, so > doing that only once, too, seems like a win to me. I agree. Since by far the most common use case is to create the exception in the raise statement, the behavior there won't be any different than it is today; the traceback on precreated objects will be useless, but folks who precreate them for performance reasons presumably won't care; and those that create global exception instances by mistakenly copying the wrong idiom, well, they'll learn quickly (and a lot more quickly than when we try to override the exception). > FWIW, that's basically how exceptions work in Java. > From http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Throwable.html: > > Instances of two subclasses, Error and Exception, are > > conventionally used to indicate that exceptional situations have > > occurred. Typically, these instances are freshly created in the > > context of the exceptional situation so as to include relevant > > information (such as stack trace data). > > > A throwable contains a snapshot of the execution stack of its > > thread at the time it was created. It can also contain a message > > string that gives more information about the error. Finally, it can > > contain a cause: another throwable that caused this throwable to > > get thrown. The cause facility is new in release 1.4. It is also > > known as the chained exception facility, as the cause can, itself, > > have a cause, and so on, leading to a "chain" of exceptions, each > > caused by another. > > There's probably a million reasons why this doesn't work for python, > but they don't immediately jump out at me. :) Not at me either. Java exceptions weren't around when Python's exceptions were first invented! > Migration from 2.X to 3.X would consist of recommending not to create > an exception outside of a raise line, unless you're okay with the > traceback location changing from the raise point to the creation point. Sounds fine with me. (But I haven't digested Glyph's response.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Mar 2 02:32:10 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 1 Mar 2007 17:32:10 -0800 Subject: [Python-Dev] with_traceback In-Reply-To: <20070301065319.17852.1514562790.divmod.quotient.1256@ohm> References: <20070301065319.17852.1514562790.divmod.quotient.1256@ohm> Message-ID: On 2/28/07, glyph at divmod.com wrote: > > On Wed, 28 Feb 2007 18:10:21 -0800, Guido van Rossum wrote: > >I am beginning to think that there are serious problems with attaching > >the traceback to the exception; I really don't like the answer that > >pre-creating an exception is unpythonic in Py3k. > > In Twisted, to deal with asynchronous exceptions, we needed an object to specifically represent a "raised exception", i.e. an Exception instance with its attached traceback and methods to manipulate it. > > You can find its API here: > > http://twistedmatrix.com/documents/current/api/twisted.python.failure.Failure.html > > Perhaps the use-cases for attaching the traceback object to the exception would be better satisfied by simply having sys.exc_info() return an object with methods like Failure? Reading the "motivation" section of PEP 344, it describes "passing these three things in parallel" as "tedious and error-prone". Having one object one could call methods on instead of a 3-tuple which needed to be selectively passed on would simplify things. > > For example, chaining could be accomplished by doing something like this: > > sys.current_exc_thingy().chain() > > I can't think of a good name for the new object type, since "traceback", "error", "exception" and "stack" all already mean things in Python. I'm guessing you didn't see James Knight's proposal. If we can agree on more Java-esque exception semantics, the exception object could serve this purpose just fine. I'm thinking that in that case an explicit with_traceback(tb) should perhaps clone the exception; the clone could be fairly simple by constructing a new uninitialized instance (the way unpickling does) and filling its dict with a copy of the original's dict, overwriting the traceback. (Also, if Brett's exception reform is accepted, we should call this attribute just traceback, not __traceback__.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Mar 2 02:45:56 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 1 Mar 2007 17:45:56 -0800 Subject: [Python-Dev] with_traceback In-Reply-To: <45E68E6E.8050000@canterbury.ac.nz> References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> <45E68E6E.8050000@canterbury.ac.nz> Message-ID: [Summary: James Knight's idea can't work unless we copy the entire stack, which is bad. Disregard my previous posts in this thread of a few minutes ago. See the end of this post why.] On 3/1/07, Greg Ewing wrote: > James Y Knight wrote: > > > It seems to me that a stack trace should always be attached to an > > exception object at creation time > > Um. Yes. Well, that's certainly an innovative solution... In the Python context perhaps, but given the Java precedent I would hardly call it innovative. > > The traceback won't necessarily be *useful*, > > Almost completely use*less*, I would have thought. > The traceback is mostly used to find out where > something went wrong, not where it went right (i.e. > successful creation of the exception). Which is one opcode before it is raised, in 99.99% of all cases. > > creating the traceback is generally very expensive, > > I don't think so -- isn't it just a linked list > of existing stack frames? That should be very cheap > to create. This is a difference between Python and Java that we should preserve. Java's traceback is a string; Python's is a linked list of traceback objects > > From http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Throwable.html: > > > >>A throwable contains a snapshot of the execution stack of its > >>thread at the time it was created. > > This would be a major and surprising change to Python > users. > > It would also be considerably *more* expensive to implement > than the current scheme, because it would require copying the > entire stack, instead of just linking stack frames together > as they are unwound during the search for an exception > handler. Oh bah, you're right. This sounds like a deal killer c.q. show shopper. In my earlier responses forgot the details of how Python exceptions work. You start with a traceback object pointing to the current frame object (traceback objects are distinct from frame objects, they are linked in the *opposite* direction, so no cycles are created); then each time the exception bubbles up a stack frame, a new traceback object pointing to the next frame object is inserted in front of the traceback. This requires updating the traceback pointer each time we bubble up a frame. Then when you catch the exception, the chain of tracebacks points to all frames between the catching and the raising frame (I forget whether catching frame is included). Since this can conceivably be going on in parallel in multiple threads, we really don't ever want to be sharing whatever object contains the head of the chain of tracebacks since it mutates at every frame bubble-up. I guess our next best option would be Glyph's suggested object to represent a caught exception, which could indeed be named "catch" per Greg's suggestion. The next-best option would be to clone the exception object whenever it is raised, but that seems wasteful in the normal case. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Fri Mar 2 05:05:12 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 02 Mar 2007 17:05:12 +1300 Subject: [Python-Dev] with_traceback In-Reply-To: References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> <45E68E6E.8050000@canterbury.ac.nz> Message-ID: <45E7A278.1080005@canterbury.ac.nz> Guido van Rossum wrote: > You start with a traceback object pointing to the current frame > object (traceback objects are distinct from frame objects, Just out of curiosity, is it really necessary to have a distinct traceback object? Couldn't the traceback just be made of dead frame objects linked the opposite way through their f_next pointers? Seems to me it would be advantageous if raising an exception (once it's created) could be done without having to allocate any memory. Otherwise you could get the situation where you're trying to raise a MemoryError, but there's no memory to allocate a traceback object, so you raise a MemoryError... etc... That might be a reason for pre-allocating the MemoryError exception, too. -- Greg From guido at python.org Fri Mar 2 05:52:51 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 1 Mar 2007 20:52:51 -0800 Subject: [Python-Dev] with_traceback In-Reply-To: <45E7A278.1080005@canterbury.ac.nz> References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> <45E68E6E.8050000@canterbury.ac.nz> <45E7A278.1080005@canterbury.ac.nz> Message-ID: On 3/1/07, Greg Ewing wrote: > Guido van Rossum wrote: > > You start with a traceback object pointing to the current frame > > object (traceback objects are distinct from frame objects, > > Just out of curiosity, is it really necessary to have > a distinct traceback object? Couldn't the traceback > just be made of dead frame objects linked the opposite > way through their f_next pointers? That would be rather fragile; there's tons of code that follows f_next pointers, without regard for whether the frame is alive or dead. Using a separate pointer would be a possibility, but it would probably still mean setting the f_next pointers to NULL to avoid hard cycles. Maybe this idea could be used for a new VM design though. > Seems to me it would be advantageous if raising an > exception (once it's created) could be done without > having to allocate any memory. Otherwise you could > get the situation where you're trying to raise a > MemoryError, but there's no memory to allocate a > traceback object, so you raise a MemoryError... > etc... > > That might be a reason for pre-allocating the > MemoryError exception, too. I think it is preallocated. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From nmm1 at cus.cam.ac.uk Fri Mar 2 10:23:58 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Fri, 02 Mar 2007 09:23:58 +0000 Subject: [Python-Dev] Class destructor Message-ID: "Guido van Rossum" wrote: > > Can you explain the reason for cleaning up in this scenario? Are you > rapidly creating and destroying temporary class objects? Why can't you > rely on the regular garbage collection process? Or does you class > create an external resource like a temp file? Effectively the latter. The C level defines a meta-class, which is instantiated with a specific precision, range etc. to derive the class that can actually be used. There can be an arbitrary number of such derived classes, with different properties. Very like Decimal, but with the context as part of the derived class. The instantiation creates quite a lot of constants and scratch space, some of which are Python objects but others of which are just Python memory (PyMem_Malloc); this is where an anonymous storage built-in type would be useful. The contents of these are of no interest to any Python code, and even the objects are ones which mustn't be accessed by the exported interfaces. Also, on efficiency grounds, all of those need to be accessible by C pointers from the exported class. Searching by name every time they are needed is far too much overhead. Note that, as with Decimal, the issue is that they are arbitrary sized and therefore can't simply be put in the class structure. Now, currently, I have implemented the suggestion of using the callback on the C object that points to the structure that contains the pointers to all of those. I need to investigate it in more detail, because I have had mixed success - that could well be the result of another bug in my code, so let's not worry about it. In THIS case, I am now pretty sure that I don't need any more, but I can imagine classes where it wouldn't be adequate. In particular, THIS code doesn't need to do anything other than free memory, so I don't care whether the C object attribute callback is called before or after the class object is disposed of. But that is obviously not the case in general. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From nmm1 at cus.cam.ac.uk Fri Mar 2 10:53:21 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Fri, 02 Mar 2007 09:53:21 +0000 Subject: [Python-Dev] Class destructor Message-ID: Sorry about a second message, but I mentioned this aspect earlier, and it is semi-independent. If I want to produce statistics, such as the times spent in various operations, I need a callback when the class is disposed of. Now, that WOULD be inconvenient to use the C object attribute callback, unless I could be sure that would be called while the class structure is still around. That could be resolved by taking a copy, of course, but that is messy. This also relates to one of my problems with the callback. I am not being called back if the class is still live at program termination; ones that have had their use counts drop to zero do cause a callback, but not ones whose use count is above zero. I am not sure whether this is my error or a feature of the garbage collector. If the latter, it doesn't matter from the point of view of freeing space, but is assuredly a real pain for producing statistics. I haven't looked into it, as it is not an immediate task. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From jimjjewett at gmail.com Fri Mar 2 21:35:10 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 2 Mar 2007 15:35:10 -0500 Subject: [Python-Dev] except Exception as err, tb [was: with_traceback] Message-ID: Guido van Rossum wrote: > Since this can conceivably be going on in parallel in multiple > threads, we really don't ever want to be sharing whatever object > contains the head of the chain of tracebacks since it mutates at every > frame bubble-up. So (full) exceptions can't be unitary objects. In theory, raising an already-instantiated instance could indicate "no traceback", which could make pre-cooked exceptions even lighter. Otherwise, there is no way to make exceptions reference their traceback directly. Using a property which depends on both the exception and the frame asking (catching frame? thread?) is ... probably worse than keeping them separate. >>> except MyException as err, tb isn't so awful, and makes it clear that the traceback is no longer needed after the more typical >>> except MyException as err ... # no raise statements needing a context or cause -jJ From brett at python.org Fri Mar 2 21:46:22 2007 From: brett at python.org (Brett Cannon) Date: Fri, 2 Mar 2007 12:46:22 -0800 Subject: [Python-Dev] except Exception as err, tb [was: with_traceback] In-Reply-To: References: Message-ID: On 3/2/07, Jim Jewett wrote: > Guido van Rossum wrote: > > > Since this can conceivably be going on in parallel in multiple > > threads, we really don't ever want to be sharing whatever object > > contains the head of the chain of tracebacks since it mutates at every > > frame bubble-up. > > So (full) exceptions can't be unitary objects. > > In theory, raising an already-instantiated instance could indicate "no > traceback", which could make pre-cooked exceptions even lighter. > > Otherwise, there is no way to make exceptions reference their > traceback directly. Using a property which depends on both the > exception and the frame asking (catching frame? thread?) is ... > probably worse than keeping them separate. > > >>> except MyException as err, tb > > isn't so awful, and makes it clear that the traceback is no longer > needed after the more typical > > >>> except MyException as err > ... # no raise statements needing a context or cause > Just because it popped in my head, I think:: except Exception as exc with tb: ... reads better. -Brett From nmm1 at cus.cam.ac.uk Fri Mar 2 22:02:22 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Fri, 02 Mar 2007 21:02:22 +0000 Subject: [Python-Dev] except Exception as err, tb [was: with_traceback] Message-ID: "Jim Jewett" wrote: > Guido van Rossum wrote: > > > Since this can conceivably be going on in parallel in multiple > > threads, we really don't ever want to be sharing whatever object > > contains the head of the chain of tracebacks since it mutates at every > > frame bubble-up. > > So (full) exceptions can't be unitary objects. > > In theory, raising an already-instantiated instance could indicate "no > traceback", which could make pre-cooked exceptions even lighter. Grrk. I think that this is right, but the wrong way to think of it! If we regard a kind of exception as a class, and an actual occurrence as an instance, things become a lot cleaner. The class is very simple, because all it says is WHAT happened - let's say divide by zero, or an attempt to finagle an object of class chameleon. The instance contains all of the information about the details, such as the exact operation, the values and the context (including the traceback). It CAN'T be an object, because it is not 'assignable' (i.e. a value) - it is inherently bound to its context. You can turn it into an object by copying its context into an assignable form, but the actual instance is not assignable. This becomes VERY clear when you try to implement advanced exception handling - rare nowadays - including the ability to trap exceptions, fix up the failure and continue (especially in a threaded environment). This makes no sense whatsoever in another context, and it becomes clear that the action of turning an instance into an object disables the ability to fix up the exception and continue. You can still raise a Python-style exception (i.e. abort up to the closest handler), but you can't resume transparently. I have implemented such a system, IBM CEL was one, and VMS had/has one. I don't know of any in the Unix or Microsoft environments, but there may be a few in specialised areas. Harking back to your point, your "already-instantiated instance" is actually an object derived directly from the exception class, and everything becomes clear. Because it is an object, any context it includes was a snapshot and is no longer valid. In your case, you would want it to have "context: unknown". Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From greg.ewing at canterbury.ac.nz Fri Mar 2 22:42:01 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 03 Mar 2007 10:42:01 +1300 Subject: [Python-Dev] except Exception as err, tb [was: with_traceback] In-Reply-To: References: Message-ID: <45E89A29.5040801@canterbury.ac.nz> Brett Cannon wrote: > except Exception as exc with tb: > ... I was thinking of that too, plus a matching raise Exception with tb The only use for a 'catch' object would then be for returning from sys.exc_info(), and I'm not sure it's worth creating one just for that rather than using a 2-tuple. -- Greg From greg.ewing at canterbury.ac.nz Fri Mar 2 23:00:53 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 03 Mar 2007 11:00:53 +1300 Subject: [Python-Dev] Another traceback idea [was: except Exception as err, tb] In-Reply-To: References: Message-ID: <45E89E95.5080804@canterbury.ac.nz> Nick Maclaren wrote: > The instance contains all of the information about the details, such > as the exact operation, the values and the context (including the > traceback). It CAN'T be an object, because it is not 'assignable' > (i.e. a value) - it is inherently bound to its context. You can > turn it into an object by copying its context into an assignable form, > but the actual instance is not assignable. This has given me another idea: Instead of instantiating the exception when it's raised, what about instantiating it when it's *caught* instead? Suppose for a moment that we take what might seem to be a retrograde step, and go back to the old way of raising exceptions, where you supply the type and arguments separately: raise MyException, ("Bogus value:", x) Now instead of immediately "normalising" this by creating the exception, we keep all three parts (type, args, traceback) separate while we search for a handler. If we find one of the form except MyException as e: then at that point we instantiate the exception, attach the traceback and assign it to e. However, if we find one of the form except MyException: then we don't need to instantiate it at all! This would address the issue of efficiently using exceptions for flow control, since for that kind of use you're probably not going to want the exception object, in which case your except clauses will be of the latter form. Now, I'm not proposing that the raise statement should actually have the above syntax -- that really would be a step backwards. Instead it would be required to have one of the following forms: raise ExceptionClass or raise ExceptionClass(args) plus optional 'with traceback' clauses in both cases. However, the apparent instantiation call wouldn't be made -- it's just part of the syntax. -- Greg From jimjjewett at gmail.com Fri Mar 2 23:44:38 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 2 Mar 2007 17:44:38 -0500 Subject: [Python-Dev] except Exception as err with tb [was: with_traceback] Message-ID: Proposal changed from "err, tb" to "err with tb" at Brett's good suggestion. The rest of this message is a bit of a sidetrack; skimmers can stop now. Nick Maclaren wrote: >"Jim Jewett" wrote: >> Guido van Rossum wrote: >> > [multiple threads => don't share] whatever object >> > contains the head of the chain of tracebacks >> So (full) exceptions can't be unitary objects. As Shane pointed out[1], the *traceback* could be a unitary object holding all three pieces of information. The problem is that we don't want every except clause adding boilerplate to get the exception back out. [1] http://mail.python.org/pipermail/python-dev/2007-February/071417.html >> In theory, raising an already-instantiated instance could indicate "no >> traceback", which could make pre-cooked exceptions even lighter. > The instance contains all of the information about the details, such > as the exact operation, the values and the context (including the > traceback). ... > ... the action of turning an instance into an object disables > the ability to fix up the exception and continue. ... > I don't know of any in the Unix or Microsoft environments, but > there may be a few in specialised areas. Lisp does it. The cost is that instead of dying, those frames become a continuation, and you need to keep around lots of extra (probable) garbage. Dealing with this cleanly was (once upon a time) one of the advantages of Stackless Python. Unfortunately, extension modules can call back into python from the middle of a C function, so the clean restarting was largely limited to pure python frames; trying to get as close as possible for other code made the implementation fairly complicated. > Harking back to your point, your "already-instantiated instance" > is actually an object derived directly from the exception class, and > everything becomes clear. Because it is an object, any context it > includes was a snapshot and is no longer valid. In your case, you > would want it to have "context: unknown". Rather, I want "context: irrelevant" so it won't bother to keep the context alive. That is clearly the right thing for a normal StopIteration. It isn't the right thing for all pre-instantiated exceptions. Whether it is the right thing often enough to ask those others to use the typical idiom -- maybe. -jJ From dustin at v.igoro.us Fri Mar 2 23:50:46 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Fri, 2 Mar 2007 16:50:46 -0600 Subject: [Python-Dev] Another traceback idea [was: except Exception as err, tb] In-Reply-To: <45E89E95.5080804@canterbury.ac.nz> References: <45E89E95.5080804@canterbury.ac.nz> Message-ID: <20070302225046.GG8385@v.igoro.us> On Sat, Mar 03, 2007 at 11:00:53AM +1300, Greg Ewing wrote: > Now, I'm not proposing that the raise statement should > actually have the above syntax -- that really would be > a step backwards. Instead it would be required to have > one of the following forms: > > raise ExceptionClass > > or > > raise ExceptionClass(args) Eep, that's awkward. If you are using exceptions for flow control, why would you use the second form? Why not just allow both exception classes and exception instances to be raised, and only instantiate-at-catch in the case of a raise of a class and a catch with an "as" clause? Then the auto-instantiation becomes a "convenience" feature of catch, safely relegating it to an easily-understood and easily-ignored corner of the user's conceptualization of exception handling. The above also looks a lot like the current syntax, but (unless I'm mistaken) ExceptionClass will be instantiated immediately right now. It seems best not to change the semantics of existing syntax if not necessary. I've been snoozing though this conversation until now, so if I've spoken out of turn, please forgive me. Dustin From dalke at dalkescientific.com Fri Mar 2 23:58:52 2007 From: dalke at dalkescientific.com (Andrew Dalke) Date: Fri, 2 Mar 2007 15:58:52 -0700 Subject: [Python-Dev] with_traceback In-Reply-To: References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> Message-ID: On 3/1/07, Guido van Rossum wrote: > Since by far the most common use case is to create the > exception in the raise statement, the behavior there won't be any > different than it is today; the traceback on precreated objects will > be useless, but folks who precreate them for performance reasons > presumably won't care; and those that create global exception > instances by mistakenly copying the wrong idiom, well, they'll learn > quickly (and a lot more quickly than when we try to override the > exception). Here's a few more examples of code which don't follow the idiom raise ExceptionClass(args) Zope's ZConfig/cmdline.py def addOption(self, spec, pos=None): if pos is None: pos = "", -1, -1 if "=" not in spec: e = ZConfig.ConfigurationSyntaxError( "invalid configuration specifier", *pos) e.specifier = spec raise e The current xml.sax.handler.Error handler includes def error(self, exception): "Handle a recoverable error." raise exception def fatalError(self, exception): "Handle a non-recoverable error." raise exception and is used like this in xml.sax.expatreader.ExpatParser.feed try: # The isFinal parameter is internal to the expat reader. # If it is set to true, expat will check validity of the entire # document. When feeding chunks, they are not normally final - # except when invoked from close. self._parser.Parse(data, isFinal) except expat.error, e: exc = SAXParseException(expat.ErrorString(e.code), e, self) # FIXME: when to invoke error()? self._err_handler.fatalError(exc) Note that the handler may decide to ignore the exception, based on which error occured. The traceback should show where in the handler the exception was raised, and not the point at which the exception was created. ZODB/Connection.py: ... if isinstance(store_return, str): assert oid is not None self._handle_one_serial(oid, store_return, change) else: for oid, serial in store_return: self._handle_one_serial(oid, serial, change) def _handle_one_serial(self, oid, serial, change): if not isinstance(serial, str): raise serial ... Andrew dalke at dalkescientific.com From collinw at gmail.com Sat Mar 3 00:45:20 2007 From: collinw at gmail.com (Collin Winter) Date: Fri, 2 Mar 2007 17:45:20 -0600 Subject: [Python-Dev] PEP 344 (was: with_traceback) Message-ID: <43aa6ff70703021545k110d9e60ld384c5768668f469@mail.gmail.com> On 2/26/07, Andrew Dalke wrote: > My concern when I saw Guido's keynote was the worry that > people do/might write code like this > > NO_END_OF_RECORD = ParserError("Cannot find end of record") > > def parse_record(input_file): > ... > raise NO_END_OF_RECORD > ... > > > That is, create instances at the top of the module, to be used > later. This code assume that the NO_END_OF_RECORD > exception instance is never modified. > > If the traceback is added to its __traceback__ attribute then > I see two problems if I were to write code like the above: > > - the traceback stays around "forever" > - the code is no longer thread-safe. While there are now quite a few threads discussing the problems related to the proposed __traceback__ attribute (thread-safety being the primary one, from what I can tease out), I haven't seen anyone address the rest of PEP 344: the __cause__ and __context__ attributes. Do those who oppose __traceback__ also oppose __cause__ and __context__? Should PEP 344 be rejected on these grounds, or should we just learn not to pre-instance exceptions if we're interested in these attributes? Collin Winter From greg.ewing at canterbury.ac.nz Sat Mar 3 00:57:15 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 03 Mar 2007 12:57:15 +1300 Subject: [Python-Dev] Another traceback idea [was: except Exception as err, tb] In-Reply-To: <20070302225046.GG8385@v.igoro.us> References: <45E89E95.5080804@canterbury.ac.nz> <20070302225046.GG8385@v.igoro.us> Message-ID: <45E8B9DB.6080209@canterbury.ac.nz> dustin at v.igoro.us wrote: > Why not just allow both exception classes and exception instances to be > raised, and only instantiate-at-catch in the case of a raise of a class > and a catch with an "as" clause? Because that doesn't solve the problem of pre-instantiated exceptions. What I'm proposing is a way of ensuring that a new instance is always created for each raise, so that a traceback can safely be attached to it -- but only if it's going to be used. > The above also looks a lot like the current syntax, but (unless I'm > mistaken) ExceptionClass will be instantiated immediately right now. I think the number of situations where it would make any difference in existing code would be very small. You'd have to go to considerable lengths to detect whether the instantiation was being done at the time of raising or the time of catching. The main difference would be that use of preinstantiated exceptions would become illegal. But this is a 3.0 suggestion, so we can cope with that if we think it's a good enough idea. > It > seems best not to change the semantics of existing syntax if not > necessary. A different syntax could be used, but it would be a big upheaval to ask everyone to un-learn the current way of raising exceptions and go back to something that we thought we had left behind as embarrassingly old-fashioned. And there wouldn't really be any option of supporting both ways, because if you can still raise a preinstantiated exception, it wouldn't be safe to attach a traceback to it, meaning we'd have to support two different ways of handling tracebacks as well. It could all get terribly messy. -- Greg From greg.ewing at canterbury.ac.nz Sat Mar 3 01:04:04 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 03 Mar 2007 13:04:04 +1300 Subject: [Python-Dev] PEP 344 (was: with_traceback) In-Reply-To: <43aa6ff70703021545k110d9e60ld384c5768668f469@mail.gmail.com> References: <43aa6ff70703021545k110d9e60ld384c5768668f469@mail.gmail.com> Message-ID: <45E8BB74.7010309@canterbury.ac.nz> Collin Winter wrote: > Do those who oppose __traceback__ also oppose __cause__ and > __context__? They would seem to have the same problems. Whatever solution is adopted for the traceback should probably be applied to them as well, perhaps by generalising the traceback into an "exception context" object that can hold other info. -- Greg From dalke at dalkescientific.com Sat Mar 3 01:15:06 2007 From: dalke at dalkescientific.com (Andrew Dalke) Date: Fri, 2 Mar 2007 17:15:06 -0700 Subject: [Python-Dev] Another traceback idea [was: except Exception as err, tb] In-Reply-To: <45E89E95.5080804@canterbury.ac.nz> References: <45E89E95.5080804@canterbury.ac.nz> Message-ID: On 3/2/07, Greg Ewing wrote: > This has given me another idea: ... > Now, I'm not proposing that the raise statement should > actually have the above syntax -- that really would be > a step backwards. Instead it would be required to have > one of the following forms: > > raise ExceptionClass > > or > > raise ExceptionClass(args) > > plus optional 'with traceback' clauses in both cases. > However, the apparent instantiation call wouldn't be > made -- it's just part of the syntax. Elsewhere here I listed several examples of existing code which raises an instance which was caught or created earlier. That would not be supported if the raise had to be written in your given forms. Andrew dalke at dalkescientific.com From fuzzyman at voidspace.org.uk Sat Mar 3 01:20:24 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 03 Mar 2007 00:20:24 +0000 Subject: [Python-Dev] with_traceback In-Reply-To: <45E68F93.2080607@canterbury.ac.nz> References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> <45E66DDA.8090406@voidspace.org.uk> <45E68F93.2080607@canterbury.ac.nz> Message-ID: <45E8BF48.9030906@voidspace.org.uk> Greg Ewing wrote: > Michael Foord wrote: > >> With the >> proposed changes, modules that do this would *continue* to work, surely >> ? >> > > Probably, but it might mean they were no longer thread > safe. An exception caught and raised in one thread would > be vulnerable to having its traceback clobbered by > another thread raising the same instance. > Right - but that would still be *no worse* than the current situation where that information isn't available on the instance. The current patterns would continue to work unchanged, but the new information wouldn't be available because a single instance is being reused. > There's also the possibility of a traceback unexpectedly > kept alive causing GC problems -- cycles, files not > closed when you expect, etc. > That *could* be a problem, although explicitly closing files is always a good practise :-) Michael Foord > -- > Greg > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > From guido at python.org Sat Mar 3 01:22:18 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 2 Mar 2007 16:22:18 -0800 Subject: [Python-Dev] PEP 344 (was: with_traceback) In-Reply-To: <43aa6ff70703021545k110d9e60ld384c5768668f469@mail.gmail.com> References: <43aa6ff70703021545k110d9e60ld384c5768668f469@mail.gmail.com> Message-ID: On 3/2/07, Collin Winter wrote: > On 2/26/07, Andrew Dalke wrote: > > My concern when I saw Guido's keynote was the worry that > > people do/might write code like this > > > > NO_END_OF_RECORD = ParserError("Cannot find end of record") > > > > def parse_record(input_file): > > ... > > raise NO_END_OF_RECORD > > ... > > > > > > That is, create instances at the top of the module, to be used > > later. This code assume that the NO_END_OF_RECORD > > exception instance is never modified. > > > > If the traceback is added to its __traceback__ attribute then > > I see two problems if I were to write code like the above: > > > > - the traceback stays around "forever" > > - the code is no longer thread-safe. > > While there are now quite a few threads discussing the problems > related to the proposed __traceback__ attribute (thread-safety being > the primary one, from what I can tease out), I haven't seen anyone > address the rest of PEP 344: the __cause__ and __context__ attributes. > > Do those who oppose __traceback__ also oppose __cause__ and > __context__? Should PEP 344 be rejected on these grounds, or should we > just learn not to pre-instance exceptions if we're interested in these > attributes? Those attributes have exactly the same problems as __traceback__. They exist mostly to save the butt of those folks who didn't even know they had a problem, so I don't think that saying "if you want this functionality, don't do X" is very helpful. I'm afraid we're back at square zero; perhaps we should keep the existing (type, value, traceback) API rather than going for radical (and unproven) innovation like "except E as v with tb: ". In the future we will still have the path open to a "Catch" object as proposed by Glyph and Greg; all we'd have to do is give the Catch object sequence semantics during the transition. I wonder if we should also roll back the new semantics of "except E as v: " of deleting v when exiting ; that was proposed in anticipation of the traceback attribute on exceptions. If we're not doing that, using vanilla variable semantics will give better migration from 2.6 and fewer surprises. On 3/2/07, Greg Ewing wrote: > They would seem to have the same problems. Whatever > solution is adopted for the traceback should probably > be applied to them as well, perhaps by generalising > the traceback into an "exception context" object > that can hold other info. That would work for the "Catch" object. It wouldn't work so well for the original (t, v, tb) API, nor for the "except E as v with tb" syntax ("except E as v with tb in context for cause" anyone? :-) The one thing that makes me not want to give up yet is that having the traceback, context, cause etc. as attributes of the exception object would seem to make the API for handling exceptions that much cleaner, compared to having to call sys.exc_info() or even "except E as v with tb". So, despite the existence of libraries that pre-create exceptions, how bad would it really be if we declared that use unsafe? It wouldn't be hard to add some kind of boobytrap that goes off when pre-created exceptions are raised multiple times. If this had always been the semantics I'm sure nobody would have complained and I doubt that it would have been a common pitfall either (since if it doesn't work, there's no bad code abusing it, and so there are no bad examples that newbies could unwittingly emulate). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Mar 3 01:30:51 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 2 Mar 2007 16:30:51 -0800 Subject: [Python-Dev] Another traceback idea [was: except Exception as err, tb] In-Reply-To: References: <45E89E95.5080804@canterbury.ac.nz> Message-ID: On 3/2/07, Andrew Dalke wrote: > On 3/2/07, Greg Ewing wrote: > > This has given me another idea: > ... > > Now, I'm not proposing that the raise statement should > > actually have the above syntax -- that really would be > > a step backwards. Instead it would be required to have > > one of the following forms: > > > > raise ExceptionClass > > > > or > > > > raise ExceptionClass(args) > > > > plus optional 'with traceback' clauses in both cases. > > However, the apparent instantiation call wouldn't be > > made -- it's just part of the syntax. > > Elsewhere here I listed several examples of existing > code which raises an instance which was caught or > created earlier. That would not be supported if the > raise had to be written in your given forms. Raising an instance that was just caught or created a bit earlier should definitely be supported. Especially this pattern is useful, in all its variations: def handle_error(self, exc): """Subclass can override to ignore it.""" raise exc def foo(self): if self.bar(): self.handle_error(RuntimeError("can't foo because of bar")) # if it returns, ignore bar # do foo's thing The question is, how to distinguish this from an exception created *much* earlier and raised repeatedly? Adding an attribute pointing to the "owning thread" might help catch concurrent use, but that wouldn't help if the pattern is used in a single-threaded environment, even though it could still be an error there (e.g. in used in a recursive context). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fuzzyman at voidspace.org.uk Sat Mar 3 01:32:56 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 03 Mar 2007 00:32:56 +0000 Subject: [Python-Dev] PEP 344 In-Reply-To: References: <43aa6ff70703021545k110d9e60ld384c5768668f469@mail.gmail.com> Message-ID: <45E8C238.6030400@voidspace.org.uk> Guido van Rossum wrote: > [snip..] > The one thing that makes me not want to give up yet is that having the > traceback, context, cause etc. as attributes of the exception object > would seem to make the API for handling exceptions that much cleaner, > compared to having to call sys.exc_info() or even "except E as v with > tb". > > So, despite the existence of libraries that pre-create exceptions, how > bad would it really be if we declared that use unsafe? It wouldn't be > hard to add some kind of boobytrap that goes off when pre-created > exceptions are raised multiple times. If this had always been the > semantics I'm sure nobody would have complained and I doubt that it > would have been a common pitfall either (since if it doesn't work, > there's no bad code abusing it, and so there are no bad examples that > newbies could unwittingly emulate). > > Personally, I think the new attributes and the new syntax are *great* improvements to exception handling. I would be sad to see parts of these proposals not happen. Michael Foord From greg.ewing at canterbury.ac.nz Sat Mar 3 01:32:56 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 03 Mar 2007 13:32:56 +1300 Subject: [Python-Dev] with_traceback In-Reply-To: <45E8BF48.9030906@voidspace.org.uk> References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> <45E66DDA.8090406@voidspace.org.uk> <45E68F93.2080607@canterbury.ac.nz> <45E8BF48.9030906@voidspace.org.uk> Message-ID: <45E8C238.3020002@canterbury.ac.nz> Michael Foord wrote: > Greg Ewing wrote: > > > An exception caught and raised in one thread would > > be vulnerable to having its traceback clobbered by > > another thread raising the same instance. > > Right - but that would still be *no worse* than the current situation > where that information isn't available on the instance. Um -- yes, it would, because currently you don't *expect* the traceback to be available from the exception. If that became the standard way to handle tracebacks, then you would expect it to work reliably. -- Greg From ked+python at tao-group.com Sat Mar 3 01:51:55 2007 From: ked+python at tao-group.com (K D) Date: Sat, 03 Mar 2007 00:51:55 +0000 Subject: [Python-Dev] NOARGS_NULL In-Reply-To: <45E1CEFA.2070601@v.loewis.de> References: <45D81DDA.5060905@v.loewis.de> <9e804ac0702231627h4c2f9434v15e038b2de67b6b5@mail.gmail.com> <45E0C5E3.4030002@tao-group.com> <45E1CEFA.2070601@v.loewis.de> Message-ID: <45E8C6AB.5040204@tao-group.com> Martin v. L?wis wrote: > The style-guide in C is that macro names are spelled in all-caps, so > if you see an all-caps identifier, you expect it to be a macro name. Hi Martin. OK, so somewhere we have: /* NOARGS_NULL: A parameter declared in this way is present only because * the function is registered as a METH_NOARGS function. Due to the * way such functions are called [reference], the parameter will always * be NULL (as the name suggests) and should always be ignored. The * reason for this is [reference|"historical"]. */ #define NOARGS_NULL _noargs_null_ ... this is tongue-in-cheek, but I'm only half-joking (perhaps this is an acceptable compromise). My whole reason for naming it that way was to help people who might be grepping through the source to understand what's going on (and the two references/grep strings are in upper-case elsewhere). FWIW, I'm not overly concerned about the spelling - as the SF comment indicated I thought that the all-caps name might be an issue, but really I'd rather you called it "PyObject *_" than not have it in at all because of the amount of work I'd have to do each time there was a new release without it ;) Thanks, Kev. From dalke at dalkescientific.com Sat Mar 3 01:41:39 2007 From: dalke at dalkescientific.com (Andrew Dalke) Date: Fri, 2 Mar 2007 17:41:39 -0700 Subject: [Python-Dev] PEP 344 (was: with_traceback) In-Reply-To: References: <43aa6ff70703021545k110d9e60ld384c5768668f469@mail.gmail.com> Message-ID: On 3/2/07, Guido van Rossum wrote: > So, despite the existence of libraries that pre-create exceptions, how > bad would it really be if we declared that use unsafe? It wouldn't be > hard to add some kind of boobytrap that goes off when pre-created > exceptions are raised multiple times. If this had always been the > semantics I'm sure nobody would have complained and I doubt that it > would have been a common pitfall either (since if it doesn't work, > there's no bad code abusing it, and so there are no bad examples that > newbies could unwittingly emulate). Here's code from os._execvpe which reraises an exception instance which was created earlier saved_exc = None saved_tb = None for dir in PATH: fullname = path.join(dir, file) try: func(fullname, *argrest) except error, e: tb = sys.exc_info()[2] if (e.errno != ENOENT and e.errno != ENOTDIR and saved_exc is None): saved_exc = e saved_tb = tb if saved_exc: raise error, saved_exc, saved_tb raise error, e, tb Would the boobytrap go off in this case? I think it would, because a "saved_exc" is raised twice. Andrew dalke at dalkescientific.com From fuzzyman at voidspace.org.uk Sat Mar 3 01:58:59 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 03 Mar 2007 00:58:59 +0000 Subject: [Python-Dev] with_traceback In-Reply-To: <45E8C238.3020002@canterbury.ac.nz> References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> <45E66DDA.8090406@voidspace.org.uk> <45E68F93.2080607@canterbury.ac.nz> <45E8BF48.9030906@voidspace.org.uk> <45E8C238.3020002@canterbury.ac.nz> Message-ID: <45E8C853.507@voidspace.org.uk> Greg Ewing wrote: > Michael Foord wrote: > >> Greg Ewing wrote: >> >> >>> An exception caught and raised in one thread would >>> be vulnerable to having its traceback clobbered by >>> another thread raising the same instance. >>> >> Right - but that would still be *no worse* than the current situation >> where that information isn't available on the instance. >> > > Um -- yes, it would, because currently you don't *expect* > the traceback to be available from the exception. If > that became the standard way to handle tracebacks, then > you would expect it to work reliably. > Um... except that the new attributes *obviously* means that the traceback information is obviously not going to work where you reuse a single instance and to expect otherwise seems naive. If the new pattern *doesn't* break existing code, but means that using a single instance for optimisation (the only justification put forward - re-raising being a slightly different case) makes that information unreliable; then I don't see that as a reason to reject the change. Michael Foord > -- > Greg > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > From greg.ewing at canterbury.ac.nz Sat Mar 3 02:05:41 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 03 Mar 2007 14:05:41 +1300 Subject: [Python-Dev] PEP 344 (was: with_traceback) In-Reply-To: References: <43aa6ff70703021545k110d9e60ld384c5768668f469@mail.gmail.com> Message-ID: <45E8C9E5.3060009@canterbury.ac.nz> Guido van Rossum wrote: > I'm afraid we're back at square zero; perhaps we should keep the > existing (type, value, traceback) API Whatever happens, we should be able to get that down to at most two things: (exception, context) where exception is either a class or an instance, and context includes the traceback and anything else that varies from one raising to another. > rather than going for radical > (and unproven) innovation like "except E as v with tb: ". The "with something" clause isn't mutually exclusive with that -- rather it goes hand-in-hand as a nicer way of getting hold of the context than calling sys.exc_info(). I'd consider it an optional extra. > "except E as v with tb in context for cause" anyone? :-) I was thinking more of except E as v with context: and context would have attributes such as traceback, cause, etc. (Although calling it a "context" could be seriously confusing, considering we also have a "with" statement having something called a context that's completely different.) > The one thing that makes me not want to give up yet is that having the > traceback, context, cause etc. as attributes of the exception object > would seem to make the API for handling exceptions that much cleaner, If we can get it down to a 2-tuple, with no fudging around to allow for sometimes having the args separate from the type but not always, that should simplify it considerably. One thing I would like to get out of it if possible is the chance to raise and catch an exception class without instantiating it at all under favourable circumstances. That should be possible if we allow the first item of the tuple to be either a class or instance, and not bother instantiating if it's a class and gets caught with except E: or except E with context: -- Greg From guido at python.org Sat Mar 3 02:09:00 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 2 Mar 2007 17:09:00 -0800 Subject: [Python-Dev] PEP 344 In-Reply-To: <45E8C238.6030400@voidspace.org.uk> References: <43aa6ff70703021545k110d9e60ld384c5768668f469@mail.gmail.com> <45E8C238.6030400@voidspace.org.uk> Message-ID: Personally, +1 for new attributes, -1 for more syntax, +0 for making it bad style to pre-create exceptions. One object should have all you need. On 3/2/07, Michael Foord wrote: > Guido van Rossum wrote: > > [snip..] > > The one thing that makes me not want to give up yet is that having the > > traceback, context, cause etc. as attributes of the exception object > > would seem to make the API for handling exceptions that much cleaner, > > compared to having to call sys.exc_info() or even "except E as v with > > tb". > > > > So, despite the existence of libraries that pre-create exceptions, how > > bad would it really be if we declared that use unsafe? It wouldn't be > > hard to add some kind of boobytrap that goes off when pre-created > > exceptions are raised multiple times. If this had always been the > > semantics I'm sure nobody would have complained and I doubt that it > > would have been a common pitfall either (since if it doesn't work, > > there's no bad code abusing it, and so there are no bad examples that > > newbies could unwittingly emulate). > > > > > Personally, I think the new attributes and the new syntax are *great* > improvements to exception handling. I would be sad to see parts of these > proposals not happen. > > Michael Foord > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Sat Mar 3 02:16:44 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 03 Mar 2007 14:16:44 +1300 Subject: [Python-Dev] with_traceback In-Reply-To: <45E8C853.507@voidspace.org.uk> References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> <45E66DDA.8090406@voidspace.org.uk> <45E68F93.2080607@canterbury.ac.nz> <45E8BF48.9030906@voidspace.org.uk> <45E8C238.3020002@canterbury.ac.nz> <45E8C853.507@voidspace.org.uk> Message-ID: <45E8CC7C.3060900@canterbury.ac.nz> Michael Foord wrote: > Um... except that the new attributes *obviously* means that the > traceback information is obviously not going to work where you reuse a > single instance and to expect otherwise seems naive. Yes, but this means that the __traceback__ attribute couldn't be *the* way of handling tracebacks in Py3k. The old way would still have to be there, and the new way could only ever be a convenience feature that might not be available and might not work properly if it is. That doesn't seem like a tidy situation to me. Py3k is supposed to me making things cleaner, not more messy. -- Greg From guido at python.org Sat Mar 3 02:21:04 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 2 Mar 2007 17:21:04 -0800 Subject: [Python-Dev] PEP 344 (was: with_traceback) In-Reply-To: References: <43aa6ff70703021545k110d9e60ld384c5768668f469@mail.gmail.com> Message-ID: On 3/2/07, Andrew Dalke wrote: > On 3/2/07, Guido van Rossum wrote: > > So, despite the existence of libraries that pre-create exceptions, how > > bad would it really be if we declared that use unsafe? It wouldn't be > > hard to add some kind of boobytrap that goes off when pre-created > > exceptions are raised multiple times. If this had always been the > > semantics I'm sure nobody would have complained and I doubt that it > > would have been a common pitfall either (since if it doesn't work, > > there's no bad code abusing it, and so there are no bad examples that > > newbies could unwittingly emulate). > > Here's code from os._execvpe which reraises an exception > instance which was created earlier > > saved_exc = None > saved_tb = None > for dir in PATH: > fullname = path.join(dir, file) > try: > func(fullname, *argrest) > except error, e: > tb = sys.exc_info()[2] > if (e.errno != ENOENT and e.errno != ENOTDIR > and saved_exc is None): > saved_exc = e > saved_tb = tb > if saved_exc: > raise error, saved_exc, saved_tb > raise error, e, tb > > Would the boobytrap go off in this case? I think it would, > because a "saved_exc" is raised twice. It shouldn't. This is a requirement for the boobytrap (which is vaporware as yet :-). Perhaps we could allow reraising whenever the existing traceback chain contains a reference to a frame that is an ancestor of (or equal to) the newly raising frame? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Mar 3 02:23:48 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 2 Mar 2007 17:23:48 -0800 Subject: [Python-Dev] PEP 344 (was: with_traceback) In-Reply-To: <45E8C9E5.3060009@canterbury.ac.nz> References: <43aa6ff70703021545k110d9e60ld384c5768668f469@mail.gmail.com> <45E8C9E5.3060009@canterbury.ac.nz> Message-ID: On 3/2/07, Greg Ewing wrote: > Guido van Rossum wrote: > > > I'm afraid we're back at square zero; perhaps we should keep the > > existing (type, value, traceback) API > > Whatever happens, we should be able to get that down > to at most two things: (exception, context) where > exception is either a class or an instance, and > context includes the traceback and anything else > that varies from one raising to another. > > > rather than going for radical > > (and unproven) innovation like "except E as v with tb: ". > > The "with something" clause isn't mutually exclusive > with that -- rather it goes hand-in-hand as a nicer > way of getting hold of the context than calling > sys.exc_info(). I'd consider it an optional extra. > > > "except E as v with tb in context for cause" anyone? :-) > > I was thinking more of > > except E as v with context: > > and context would have attributes such as traceback, > cause, etc. > > (Although calling it a "context" could be seriously > confusing, considering we also have a "with" statement > having something called a context that's completely > different.) > > > The one thing that makes me not want to give up yet is that having the > > traceback, context, cause etc. as attributes of the exception object > > would seem to make the API for handling exceptions that much cleaner, > > If we can get it down to a 2-tuple, with no fudging > around to allow for sometimes having the args separate > from the type but not always, that should simplify it > considerably. > > One thing I would like to get out of it if possible > is the chance to raise and catch an exception class > without instantiating it at all under favourable > circumstances. That should be possible if we allow > the first item of the tuple to be either a class or > instance, and not bother instantiating if it's a > class and gets caught with > > except E: > > or > > except E with context: But what's the advantage of not instantiating the exception if we instantiate the context instead? I really think that it would be nicer if the exception object *was* the context. We'd still not have to instantiate if it was just "except E:" but that can be handled behind the scenes. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From rhamph at gmail.com Sat Mar 3 04:19:09 2007 From: rhamph at gmail.com (Adam Olsen) Date: Fri, 2 Mar 2007 20:19:09 -0700 Subject: [Python-Dev] Performance of pre-creating exceptions? Message-ID: $ python2.5 -m timeit -r 10 -n 1000000 -s 'class Foo(Exception): pass' 'try: raise Foo()' 'except: pass' 1000000 loops, best of 10: 2.49 usec per loop $ python2.5 -m timeit -r 10 -n 1000000 -s 'class Foo(Exception):' -s ' def __init__(self): pass' 'try: raise Foo()' 'except: pass' 1000000 loops, best of 10: 3.15 usec per loop $ python2.5 -m timeit -r 10 -n 1000000 -s 'e = Exception()' 'try: raise e' 'except: pass' 1000000 loops, best of 10: 2.03 usec per loop We can get more than half of the benefit simply by using a default __init__ rather than a python one. If you need custom attributes but they're predefined you could subclass the exception and have them as class attributes. Given that, is there really a need to pre-create exceptions? -- Adam Olsen, aka Rhamphoryncus From greg.ewing at canterbury.ac.nz Sat Mar 3 04:47:16 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 03 Mar 2007 16:47:16 +1300 Subject: [Python-Dev] PEP 344 (was: with_traceback) In-Reply-To: References: <43aa6ff70703021545k110d9e60ld384c5768668f469@mail.gmail.com> <45E8C9E5.3060009@canterbury.ac.nz> Message-ID: <45E8EFC4.2090903@canterbury.ac.nz> Guido van Rossum wrote: > But what's the advantage of not instantiating the exception if we > instantiate the context instead? Probably not much. But most control-flow-exception catching will just be 'except E:' in which case you don't need to instantiate anything. (Assuming we get rid of traceback objects and just link the frames together directly.) -- Greg From greg.ewing at canterbury.ac.nz Sat Mar 3 05:12:04 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 03 Mar 2007 17:12:04 +1300 Subject: [Python-Dev] PEP 344 (was: with_traceback) In-Reply-To: References: <43aa6ff70703021545k110d9e60ld384c5768668f469@mail.gmail.com> Message-ID: <45E8F594.1090605@canterbury.ac.nz> Guido van Rossum wrote: > Perhaps we could allow reraising whenever the > existing traceback chain contains a reference to a frame that is an > ancestor of (or equal to) the newly raising frame? This is starting to sound terribly hairy. Would it help if a different syntax were used for raising and reraising? Raising would complain if there were already a traceback, and reraising would expect an existing traceback and extend it. -- Greg From guido at python.org Sat Mar 3 05:52:45 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 2 Mar 2007 20:52:45 -0800 Subject: [Python-Dev] Performance of pre-creating exceptions? In-Reply-To: References: Message-ID: Often people build mental models of performance that have little bearing on reality. Thanks for measuring! On 3/2/07, Adam Olsen wrote: > $ python2.5 -m timeit -r 10 -n 1000000 -s 'class Foo(Exception): pass' > 'try: raise Foo()' 'except: pass' > 1000000 loops, best of 10: 2.49 usec per loop > $ python2.5 -m timeit -r 10 -n 1000000 -s 'class Foo(Exception):' -s ' > def __init__(self): pass' 'try: raise Foo()' 'except: pass' > 1000000 loops, best of 10: 3.15 usec per loop > $ python2.5 -m timeit -r 10 -n 1000000 -s 'e = Exception()' 'try: > raise e' 'except: pass' > 1000000 loops, best of 10: 2.03 usec per loop > > We can get more than half of the benefit simply by using a default > __init__ rather than a python one. If you need custom attributes but > they're predefined you could subclass the exception and have them as > class attributes. Given that, is there really a need to pre-create > exceptions? > > -- > Adam Olsen, aka Rhamphoryncus > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From dalke at dalkescientific.com Sat Mar 3 06:42:38 2007 From: dalke at dalkescientific.com (Andrew Dalke) Date: Fri, 2 Mar 2007 22:42:38 -0700 Subject: [Python-Dev] Performance of pre-creating exceptions? In-Reply-To: References: Message-ID: On 3/2/07, Adam Olsen wrote: > We can get more than half of the benefit simply by using a default > __init__ rather than a python one. If you need custom attributes but > they're predefined you could subclass the exception and have them as > class attributes. Given that, is there really a need to pre-create > exceptions? The only real world example of (re)using pre-computed exceptions I found was in pyparsing. Here are two examples: def parseImpl( self, instring, loc, doActions=True ): if (instring[loc] == self.firstMatchChar and (self.matchLen==1 or instring.startswith(self.match,loc)) ): return loc+self.matchLen, self.match #~ raise ParseException( instring, loc, self.errmsg ) exc = self.myException exc.loc = loc exc.pstr = instring raise exc (The Token's constructor is class Token(ParserElement): def __init__( self ): super(Token,self).__init__( savelist=False ) self.myException = ParseException("",0,"",self) and the exception class uses __slots__ thusly: class ParseBaseException(Exception): """base exception class for all parsing runtime exceptions""" __slots__ = ( "loc","msg","pstr","parserElement" ) # Performance tuning: we construct a *lot* of these, so keep this # constructor as small and fast as possible def __init__( self, pstr, loc, msg, elem=None ): self.loc = loc self.msg = msg self.pstr = pstr self.parserElement = elem so you can see that each raised exception modifies 2 of the 4 instance variables in a ParseException.) -and- # this method gets repeatedly called during backtracking with the same arguments - # we can cache these arguments and save ourselves the trouble of re-parsing # the contained expression def _parseCache( self, instring, loc, doActions=True, callPreParse=True ): lookup = (self,instring,loc,callPreParse) if lookup in ParserElement._exprArgCache: value = ParserElement._exprArgCache[ lookup ] if isinstance(value,Exception): if isinstance(value,ParseBaseException): value.loc = loc raise value return value else: try: ParserElement._exprArgCache[ lookup ] = \ value = self._parseNoCache( instring, loc, doActions, callPreParse ) return value except ParseBaseException, pe: ParserElement._exprArgCache[ lookup ] = pe raise The first definitely has the look of a change for better performance. I have not asked the author nor researched to learn how much gain there was with this code. Because the saved exception is tweaked each time (hence not thread-safe), your timing tests aren't directly relevant as your solution of creating the exception using the default constructor then tweaking the instance attributes directly would end up doing 4 setattrs instead of 2. % python -m timeit -r 10 -n 1000000 -s 'e = Exception()' 'try: raise e' 'except: pass' 1000000 loops, best of 10: 1.55 usec per loop % python -m timeit -r 10 -n 1000000 -s 'e = Exception()' 'try: e.x=1;e.y=2;raise e' 'except: pass' 1000000 loops, best of 10: 1.98 usec per loop so 4 attributes should be about 2.5usec, or 25% slower than 2 attributes. There's also a timing difference between looking for the exception class name in module scope vs. using self.myException I've tried to find other examples but couldn't in the 20 or so packages I have on my laptop. I used searches like # find module variables assigned to exception instances egrep "^[a-z].*=.*Error\(" *.py egrep "^[a-z].*=.*Exception\(" *.py # find likely instances being raised grep "^ *raise [a-z]" *.py # find likely cases of 3-arg raises grep "^ *raise .*,.*," *.py to find candidates. Nearly all false positives. Andrew dalke at dalkescientific.com From martin at v.loewis.de Sat Mar 3 09:08:53 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 03 Mar 2007 09:08:53 +0100 Subject: [Python-Dev] PEP 306 changes (How to Change Python's Grammar) In-Reply-To: <20070228211757.GE5537@performancedrivers.com> References: <20070228211757.GE5537@performancedrivers.com> Message-ID: <45E92D15.1010602@v.loewis.de> Jack Diederich schrieb: > __ Parser/Python.asdl may need changes to match the Grammar. > Use Parser/asdl_c.py to regenerate Include/Python-ast.h On Unix, make will automatically run it when Python.asdl changes. > __ Python/Python-ast.c may need changes to create the AST > objects involved with the Grammar change. This will also change when asdl_c.py runs. > __ Parser/pgen needs to be rerun to regenerate Include/graminit.h > and Include/graminit.c This will also be done by the makefile automatically. Regards, Martin From python at rcn.com Mon Mar 5 02:03:58 2007 From: python at rcn.com (Raymond Hettinger) Date: Sun, 4 Mar 2007 17:03:58 -0800 Subject: [Python-Dev] [Python-3000] PEP 3113 (Removal of Tuple Parameter Unpacking) References: <001a01c75ea3$2ee099a0$6800a8c0@RaymondLaptop1> <45EB5DFE.60702@canterbury.ac.nz> Message-ID: <00ee01c75ec2$33cc93a0$6800a8c0@RaymondLaptop1> [Greg Ewing] > Writing code that depends on long sequences having things > in particular positions is an anti-pattern, Seems like you're against positional parameters in general. What makes argument unpacking in a function definition different that other positional parameter use cases. Until csv and sql start returning records with field name attributes instead of tuples, we'll need something like we have now. Also attribute access has performance issues compared to tuple unpacking (sad fact of life). Raymond From facundo at taniquetil.com.ar Mon Mar 5 16:43:20 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Mon, 5 Mar 2007 15:43:20 +0000 (UTC) Subject: [Python-Dev] Python-3000 upgrade path References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> Message-ID: Thomas Wouters wrote: > developers and people who develop their own software. I would like to hear > from people who have concrete doubts about this upgrade path. I don't mean Disclaimer: I'm not involved in Py3k, and not even tried it once. And don't know the details of the tool to "transform" Py2 to Py3. Having said that, the *only* fear I have, is how safe it will be. And here comes the explanation. I'm now in an enviroment where we rely on Python, and I advocates on it everyday. We use 2.3, and 2.4, some servers we're about to deploy have 2.5, and we'll use the lastest Py2.x everytime we deploy a new machine. But, as we have always running behind our ToDo, we don't have time to say "Ok, this server runs 2.3, how can I start using 2.5?". Too many servers, too many applications, too many undocumented-and-nobody-knows-about-it applications. And they all are applications running 7x24. And Py3k is around the corner, and I even heard some guys saying "If I'd spent time taking care that this app runs ok after changing a bit of it, I'll wait to Python 3000". So, the enviroment is explained, now I go with "how safe it will be". I'd love to know that there'll be a tool that tells me, after running it over my application, Your app is "ready for migration"'. I do *not* care if it takes a lot of work to make the proper changes, but I need to feel confident that after running the last changed version during a week in, say, Py2.7, and no warnings appear, and the "verification tool" say green light, I can start running it in Py3k. And look at it a week more. And say "life is great", :) Anyway, I know how hard is it, and I regret not having the time I'd love to have to help you. All I can do is thank you. Thank you very much! > One thing in particular I wonder about is the warning about mixing tabs and > spaces. Should it be in category 2) (on by default) or 3) (still off by > default, but part of -Wpy3k)? For me, I'd love to 2.6 start warning "you're mixing tab and spaces, shame yourself!". Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From facundo at taniquetil.com.ar Mon Mar 5 16:43:43 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Mon, 5 Mar 2007 15:43:43 +0000 (UTC) Subject: [Python-Dev] Adding socket timeout to urllib2 Message-ID: I studied which modifications I need to make into urllib2 to support a socket timeout. - I'll modify urlopen for it to accept a socket_timeout parameter, default to None - Request will also accept a socket_timeout parameter, default to None. It will keep it in a socket_timeout attribute, so it could be added/modified from outside. - OpenerDirector's 'open' method will accept socket_timeout, and build the Request object with it So, when it gets into the Handlers, how it'll be used? I checked the Handlers, and here is a resume of their 'open' or 'open'-like methods: - AbstractHTTPHandler: receives a class, and uses its .request and .getresponse methods - HTTPHandler: just calls AbstractHTTPHandler.do_open with httplib.HTTPConnection - HTTPSHandler: just calls AbstractHTTPHandler.do_open with httplib.HTTPSConnection - UnknownHandler: raises an Error - FileHandler: opens a file in a local host - FTPHandler: uses sockets in ftplib.FTP().connect, .login, .cwd, and .ntransfercmd, all through urllib's ftwrapper object - CacheFTPHandler: instantiates ftpwrapper object; does not uses it directly, though. - GopherHandler: is deprecated... So, these are the necessary further modifications in urllib2 beyond those detailed at the beggining of the mail: - Modify AbstractHTTPHandler to pass the socket_timeout parameter to the received class' methods. - Modify FTPHandler to pass the socket_timeout to ftpwrapper class. - Modify CacheFTPHandler the same way that FTPHandler Beyond that, I'll also need to: - Modify urllib.ftpwrapper to accept a socket_timeout parameter and pass it to ftplib.FTP object. - Add support to socket_timeout to ftplib's FTP object - Add support to socket_timeout to httplib's HTTPConnection and HTTPSConnection objects So, I think that in the name of clarity, I shall first modify ftplib and httplib, add socket_timeout where is necessary, and then add support to that in the more general urllib2. What do you think? Comments are very appreciated. Thank you very much! -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From facundo at taniquetil.com.ar Mon Mar 5 16:44:01 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Mon, 5 Mar 2007 15:44:01 +0000 (UTC) Subject: [Python-Dev] About code comments policy (and Handler order in urllib2) Message-ID: One question and one answer (this is a balanced post, :p). The question is what to do when we found a question in a code. Reading urllib2 I found: # XXX why does self.handlers need to be sorted? I found the answer, so I deleted that line, and added a comment in that place just to clarify. Shall I do something else? Like posting here the answer? BTW, the answer to that question is that the handlers must work in an specific order. The order is specified in each Handler (handler_order class attribute), and BaseHandler has __lt__ defined using that attribute. So, OpenerDirector, keeps self.handler ordered with bisect.insort. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From skip at pobox.com Mon Mar 5 17:39:42 2007 From: skip at pobox.com (skip at pobox.com) Date: Mon, 5 Mar 2007 10:39:42 -0600 Subject: [Python-Dev] Adding socket timeout to urllib2 In-Reply-To: References: Message-ID: <17900.18382.491570.534945@montanaro.dyndns.org> Facundo> So, I think that in the name of clarity, I shall first modify Facundo> ftplib and httplib, add socket_timeout where is necessary, and Facundo> then add support to that in the more general urllib2. I own a patch on SF to do that much: http://python.org/sf/723312 Feel free to take it from me and use as a starting point. Skip From dave at boost-consulting.com Mon Mar 5 18:45:38 2007 From: dave at boost-consulting.com (David Abrahams) Date: Mon, 05 Mar 2007 12:45:38 -0500 Subject: [Python-Dev] PCBuild8 Message-ID: <87lkibd1od.fsf@valverde.peloton> Hi, I tried building with MS Visual Studio 2005 from PCBuild8/pcbuild.sln, and for me it fails miserably. The first major complaint comes when linking pythoncore, where the _init_types symbol can't be found. On the other hand, allowing MSVS to convert PCBuild/pcbuild.sln works okay. Am I missing something? Thanks, -- Dave Abrahams Boost Consulting www.boost-consulting.com From martin at v.loewis.de Mon Mar 5 19:16:12 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 05 Mar 2007 19:16:12 +0100 Subject: [Python-Dev] About code comments policy (and Handler order in urllib2) In-Reply-To: References: Message-ID: <45EC5E6C.60204@v.loewis.de> Facundo Batista schrieb: > The question is what to do when we found a question in a code. Reading > urllib2 I found: > > # XXX why does self.handlers need to be sorted? > > I found the answer, so I deleted that line, and added a comment in that > place just to clarify. > > Shall I do something else? You should check whether documentation needs to be clarified. In this case, I see a single mention of handler_order in the documentation, and that only mentions that the attribute exists and can be set. Not having looked at all at the functionality: Do you consider this documentation sufficient? Would you think users may need to know about this feature? If so, do you think they have what they need to use it just by reading the documentation? If you really feel zestful (is that proper usage of the word?), you could try to study who added the remark, and when, and who added the documentation, and when. I often do that to verify that I really understood the purpose of the question. Regards, Martin From martin at v.loewis.de Mon Mar 5 19:18:21 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 05 Mar 2007 19:18:21 +0100 Subject: [Python-Dev] PCBuild8 In-Reply-To: <87lkibd1od.fsf@valverde.peloton> References: <87lkibd1od.fsf@valverde.peloton> Message-ID: <45EC5EED.1050507@v.loewis.de> David Abrahams schrieb: > I tried building with MS Visual Studio 2005 from PCBuild8/pcbuild.sln, > and for me it fails miserably. The first major complaint comes when > linking pythoncore, where the _init_types symbol can't be found. On > the other hand, allowing MSVS to convert PCBuild/pcbuild.sln works > okay. Am I missing something? Yes, it doesn't work in Python 2.5 as released. This specific problem has been fixed in the trunk and the 2.5 branch several months ago, so I recommend to use either of these (or use VS 2003 to build the released version). HTH, Martin From amk at amk.ca Mon Mar 5 19:46:14 2007 From: amk at amk.ca (A.M. Kuchling) Date: Mon, 5 Mar 2007 13:46:14 -0500 Subject: [Python-Dev] Encouraging developers Message-ID: <20070305184614.GA19051@localhost.localdomain> >From : 4. The patch mafia. I like everyone on python-dev that I meet, but somehow it is annoyingly difficult to get a patch into Python. Like threading, and the stdlib, this is a mixed blessing: you certainly don't want every Joe Schmoe checking in whatever crud he wants. However, the barrier is high enough that I no longer have much interest in spending the time to shepherd a patch through. Yes, this is probably all my fault -- but I still hate it! FWIW, I have a related perception that we aren't getting new core developers. These two problems are probably related: people don't get patches processed and don't become core developers, and we don't have enough core developers to process patches in a timely way. And so we're stuck. Any ideas for fixing this problem? Tangentially related: At PyCon, there was general agreement that exposing a read-only Bazaar/Mercurial/git/whatever version of the repository wouldn't be too much effort, and might make things easier for external people developing patches. Thomas Wouters apparently has private scripts that perform the conversion. What needs to be done to move ahead with this idea? --amk From skip at pobox.com Mon Mar 5 19:58:13 2007 From: skip at pobox.com (skip at pobox.com) Date: Mon, 5 Mar 2007 12:58:13 -0600 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070305184614.GA19051@localhost.localdomain> References: <20070305184614.GA19051@localhost.localdomain> Message-ID: <17900.26693.804101.291817@montanaro.dyndns.org> amk> Any ideas for fixing this problem? Not a one. amk> Tangentially related: amk> At PyCon, there was general agreement that exposing a read-only amk> Bazaar/Mercurial/git/whatever version of the repository wouldn't be amk> too much effort, and might make things easier for external people amk> developing patches. I'm not much of a version control wonk. How would these help? Can't the folks who need such stuff do some sort of anonymous svn checkout? Skip From barry at python.org Mon Mar 5 20:11:09 2007 From: barry at python.org (Barry Warsaw) Date: Mon, 5 Mar 2007 14:11:09 -0500 Subject: [Python-Dev] Encouraging developers In-Reply-To: <17900.26693.804101.291817@montanaro.dyndns.org> References: <20070305184614.GA19051@localhost.localdomain> <17900.26693.804101.291817@montanaro.dyndns.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mar 5, 2007, at 1:58 PM, skip at pobox.com wrote: > amk> Tangentially related: > amk> At PyCon, there was general agreement that exposing a read- > only > amk> Bazaar/Mercurial/git/whatever version of the repository > wouldn't be > amk> too much effort, and might make things easier for external > people > amk> developing patches. > > I'm not much of a version control wonk. How would these help? > Can't the > folks who need such stuff do some sort of anonymous svn checkout? Part of the problem is that you want to be able to publish your changes in a form that's much more conducive to collaboration than diff files. An anonymous svn checkout can't effectively be published or shared and you don't want to hand out writable access to svn to everyone. With a dvcs like Bazaar/Mercurial, et all, control of writable branches is totally decentralized. Yes, there's always the blessed central branch(es) coming from python.org, but it's very easy for Joe Shmoe to create their own writable branches, make their changes, publish them as full-fledged branches, etc. I see adopting a dvcs as beneficial in a number of ways. First, it allows people to participate even without write access, as almost first class citizens. About the only thing they can't do is commit back to the blessed branches, but that doesn't stop them from sharing their branches exactly the way we do. Also, it gives core developers a better way to review and test other people's branches because you can just pull their branch and start using it. In fact you can even branch their branch and publish your own version as a "review branch". Depending on the specific dvcs chosen, there are lots of ways to share branches that are much more "live" than diff files. Finally, because branching is so cheap (even cheaper than with svn), a dvcs is great for us even if it doesn't get us more developers. It's so much nicer to have a vcs that understands merges natively, and can track them across branches, so it lets us create cheap little branches for little features. For example, I'm sitting on a checkout of the implementation for a new PEP we worked on at PyCon. I could create an svn branch for this work, but that's fairly heavyweight, / and/ it requires me to have access to the server (which I didn't have on the train from PyCon, and I screwed up my checkout because of that). With a dvcs, I could publish my changes in any number of ways and still just have this one little branch for this one little feature. Hope that makes sense! I haven't used a real dvcs for very long, but I'm starting to see the fundamental improvements to the development process that it makes possible. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRexrTXEjvBPtnXfVAQLXjwP/YKfdMy0QYzbIEWHOGgcAAAnIoVpdN7Vm yo2qYgFwE4hrTSCiKuA5QBqF8GfbFKUpPUlMhr1OiRzrR8VVLaJpDTyFzwaukndm DbFtB/wG9GAKAS0ExIQy2wNx1ddtIWbnvsFYAnpKPDpaEadhM9pHfafiuc9C9Rme XlGB2T3SMZw= =EBbc -----END PGP SIGNATURE----- From amk at amk.ca Mon Mar 5 20:15:26 2007 From: amk at amk.ca (A.M. Kuchling) Date: Mon, 5 Mar 2007 14:15:26 -0500 Subject: [Python-Dev] Encouraging developers In-Reply-To: <17900.26693.804101.291817@montanaro.dyndns.org> References: <20070305184614.GA19051@localhost.localdomain> <17900.26693.804101.291817@montanaro.dyndns.org> Message-ID: <20070305191526.GA19923@localhost.localdomain> On Mon, Mar 05, 2007 at 12:58:13PM -0600, skip at pobox.com wrote: > I'm not much of a version control wonk. How would these help? Can't the > folks who need such stuff do some sort of anonymous svn checkout? The external developers can commit changes, revert them, etc. to their local repository, and still keep pulling changes from the python.org master as we commit them. With an anonymous SVN checkout, you can never commit changes back, so you can't commit your work in stages or roll back changes as those of us w/ commit privileges can. (svk adds distributed features on top of SVN; that would be another option.) --amk From phil at riverbankcomputing.co.uk Mon Mar 5 20:30:20 2007 From: phil at riverbankcomputing.co.uk (Phil Thompson) Date: Mon, 5 Mar 2007 19:30:20 +0000 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070305184614.GA19051@localhost.localdomain> References: <20070305184614.GA19051@localhost.localdomain> Message-ID: <200703051930.20868.phil@riverbankcomputing.co.uk> On Monday 05 March 2007 6:46 pm, A.M. Kuchling wrote: > >From : > > 4. The patch mafia. I like everyone on python-dev that I meet, > but somehow it is annoyingly difficult to get a patch into > Python. Like threading, and the stdlib, this is a mixed > blessing: you certainly don't want every Joe Schmoe checking > in whatever crud he wants. However, the barrier is high enough > that I no longer have much interest in spending the time to > shepherd a patch through. Yes, this is probably all my fault > -- but I still hate it! > > FWIW, I have a related perception that we aren't getting new core > developers. These two problems are probably related: people don't get > patches processed and don't become core developers, and we don't have > enough core developers to process patches in a timely way. And so > we're stuck. > > Any ideas for fixing this problem? 1. Don't suggest to people that, in order to get their patch reviewed, they should review other patches. The level of knowledge required to put together a patch is much less than that required to know if a patch is the right one. 2. Publically identify the core developers and their areas of expertise and responsibility (ie. which parts of the source tree they "own"). 3. Provide a forum (a python-patch mailing list) where patches can be proposed, reviewed and revised informally but quickly. 4. Acceptance by core developers that only half the "job" is developing the core - the other half is mentoring potential future core developers. Phil From phd at phd.pp.ru Mon Mar 5 21:09:02 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Mon, 5 Mar 2007 23:09:02 +0300 Subject: [Python-Dev] Encouraging developers In-Reply-To: <200703051930.20868.phil@riverbankcomputing.co.uk> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> Message-ID: <20070305200902.GA21284@phd.pp.ru> On Mon, Mar 05, 2007 at 07:30:20PM +0000, Phil Thompson wrote: > 1. Don't suggest to people that, in order to get their patch reviewed, they > should review other patches. The level of knowledge required to put together > a patch is much less than that required to know if a patch is the right one. I am afraid this could lead to proliferation of low-quality patches. A patch must touch at least code, documentation and tests, be tested itself and must not break other tests. These requirements demand higher expertise. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From glyph at divmod.com Mon Mar 5 21:50:46 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Mon, 05 Mar 2007 20:50:46 -0000 Subject: [Python-Dev] Encouraging developers In-Reply-To: <200703051930.20868.phil@riverbankcomputing.co.uk> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> Message-ID: <20070305205046.22258.117669926.divmod.xquotient.56@joule.divmod.com> A few meta-points: On 07:30 pm, phil at riverbankcomputing.co.uk wrote: >2. Publically identify the core developers and their areas of expertise >and >responsibility (ie. which parts of the source tree they "own"). I'd like to stress that this is an important point; although we all know that Guido's the eventual decision makers, there are people whose opinions need to be influenced around particular areas of the code and whose opinions carry particular weight. *I* have trouble figuring out who these people are, and I think I have more than a casual outsider's understanding of the Python development process. >3. Provide a forum (a python-patch mailing list) where patches can be >proposed, reviewed and revised informally but quickly. This reminds me of a post I really wanted to make after PyCon but rapidly became too sick to. The patches list really ought to be _this_ list. The fact that it isn't is indicative of a failure of the community. A good deal of the discussion here in recent months has either been highly speculative, or only tangentially related to Python's development, which is ostensibly its topic. We shouldn't really be talking about PR or deployment or any issues which aren't bug reports or patches here. I've certainly contributed somewhat to this problem myself, and I've made a resolution to stick to development issues here. This post itself is obviously in a grey area near the edge of that, but I do feel that, given the rather diverse population of readers here, we should collectively make the purpose of this forum explicit so that the python developers can use it to develop Python. One worrying trend I noticed at PyCon is that it seems that quite a lot of communication between core developers these days happens over private email. Core developers use private email to deal with pressing issues because python-dev has become crowded. This makes it difficult to see high-priority issues, as well as fostering an environment where every minor detail might get responded to with a cascade of "me too" posts or bike-shed discussions. The core guys have a lot of stuff to get done, and if there isn't an environment where they can do that in public, they're going to get it done in private. Taken together, all this has the overall effect of making the development process a lot harder to follow, which worsens, for example, issue #2 which I responded to above. It also creates a number of false impressions about what sort of development is really going on, since many posters here are not, in fact, working on core Python at all, just speculating about it. A few others have already pointed out the python-ideas list: http://mail.python.org/mailman/listinfo/python-ideas where the more speculative ideas should be discussed before being brought here as a patch or PEP. Of course, for more general discussion there's always good old "python-list". As far as bike-shed discussions, we can all do our part by just considering what threads we all really have something useful to contribute to. Let's all try to put the python dev back in python-dev! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070305/e1a5daf6/attachment.htm From dave at boost-consulting.com Mon Mar 5 21:12:03 2007 From: dave at boost-consulting.com (David Abrahams) Date: Mon, 05 Mar 2007 15:12:03 -0500 Subject: [Python-Dev] PCBuild8 References: <87lkibd1od.fsf@valverde.peloton> <45EC5EED.1050507@v.loewis.de> Message-ID: <87bqj7a1rg.fsf@valverde.peloton> on Mon Mar 05 2007, "Martin v. L?wis" writes: > David Abrahams schrieb: >> I tried building with MS Visual Studio 2005 from PCBuild8/pcbuild.sln, >> and for me it fails miserably. The first major complaint comes when >> linking pythoncore, where the _init_types symbol can't be found. On >> the other hand, allowing MSVS to convert PCBuild/pcbuild.sln works >> okay. Am I missing something? > > Yes, it doesn't work in Python 2.5 as released. This specific problem > has been fixed in the trunk and the 2.5 branch several months ago, > so I recommend to use either of these Okay, thanks. > (or use VS 2003 to build the released version). VS2005 seems to work just fine for the released version as long as I stay away from the PCBuild8/ stuff. -- Dave Abrahams Boost Consulting www.boost-consulting.com From martin at v.loewis.de Mon Mar 5 22:15:02 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 05 Mar 2007 22:15:02 +0100 Subject: [Python-Dev] PCBuild8 In-Reply-To: <45EC7CBA.5050301@abaqus.com> References: <87lkibd1od.fsf@valverde.peloton> <45EC5EED.1050507@v.loewis.de> <45EC7CBA.5050301@abaqus.com> Message-ID: <45EC8856.1060305@v.loewis.de> Andrew MacKeith schrieb: > Is there a scheduled date for the release of Python-2.5.1 ? There was a scheduled date, but it then interfered with the schedule of the people executing it, so there is none at the moment. I think we will release 2.5.1 in April. > I presume that the PCBuild8 directory should be used when building in 64 > bit on Win64. No. The official binaries both for Itanium and AMD64 (which are both "in 64 bit on Win64") are created with the PCbuild directory. See PCbuild/readme.txt. Regards, Martin From phil at riverbankcomputing.co.uk Mon Mar 5 22:16:47 2007 From: phil at riverbankcomputing.co.uk (Phil Thompson) Date: Mon, 5 Mar 2007 21:16:47 +0000 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070305200902.GA21284@phd.pp.ru> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <20070305200902.GA21284@phd.pp.ru> Message-ID: <200703052116.48035.phil@riverbankcomputing.co.uk> On Monday 05 March 2007 8:09 pm, Oleg Broytmann wrote: > On Mon, Mar 05, 2007 at 07:30:20PM +0000, Phil Thompson wrote: > > 1. Don't suggest to people that, in order to get their patch reviewed, > > they should review other patches. The level of knowledge required to put > > together a patch is much less than that required to know if a patch is > > the right one. > > I am afraid this could lead to proliferation of low-quality patches. A > patch must touch at least code, documentation and tests, be tested itself > and must not break other tests. These requirements demand higher expertise. I'm not sure what your point is. My point is that, if you want to encourage people to become core developers, they have to have a method of graduating through the ranks - learning (and being taught) as they go. To place a very high obstacle in their way right at the start is completely counter-productive. Phil From thomas at python.org Mon Mar 5 22:38:01 2007 From: thomas at python.org (Thomas Wouters) Date: Mon, 5 Mar 2007 22:38:01 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070305184614.GA19051@localhost.localdomain> References: <20070305184614.GA19051@localhost.localdomain> Message-ID: <9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> On 3/5/07, A.M. Kuchling wrote: > > >From >: > > 4. The patch mafia. I like everyone on python-dev that I meet, > but somehow it is annoyingly difficult to get a patch into > Python. Like threading, and the stdlib, this is a mixed > blessing: you certainly don't want every Joe Schmoe checking > in whatever crud he wants. However, the barrier is high enough > that I no longer have much interest in spending the time to > shepherd a patch through. Yes, this is probably all my fault > -- but I still hate it! > > FWIW, I have a related perception that we aren't getting new core > developers. These two problems are probably related: people don't get > patches processed and don't become core developers, and we don't have > enough core developers to process patches in a timely way. And so > we're stuck. > > Any ideas for fixing this problem? A better patch-tracker, better procedures for reviewing patches surounding this new tracker, one or more proper dvcs's for people to work off of. I'm not sure about 'identifying core developers' as we're all volunteers, with dayjobs for the most part, and only a few people seem to care enough about python as a whole. Putting the burden of patch review on the developers that say they can cover it might easily burn them out. (I see Martin handle a lot of patches, for instance, and I would love to help him, but I just can't find the time to review the patches on subjects I know much about, let alone the rest of the patches.) While submitting patches is good, there's a lot more to it than just submitting the 5-line code change to submit a bug/feature, and reviewing takes a lot of time and effort. I don't think it's unreasonable to ask for help from the submitters like we do, or ask them to write tests and docs and such. Tangentially related: > At PyCon, there was general agreement that exposing a read-only > Bazaar/Mercurial/git/whatever version of the repository wouldn't be > too much effort, and might make things easier for external people > developing patches. Thomas Wouters apparently has private scripts > that perform the conversion. What needs to be done to move ahead with > this idea? I need to get time, or people need to volunteer to do the work :) It's not entirely easy to do, depending on the dvcs in question. Canonical will be getting us an up to date conversion to Bazaar in a couple of weeks or so; I will be using that, or a Mercurial conversion I do myself, to refactor all Py3K work into separate branches for easier[*] backporting. I would love to do one for Monotone too (my personal favourite, but I'm weird/paranoid/fascinated by the potential and the elegance) -- but I doubt I'll get the time, and the monotone conversion takes many times as long as the rest. Of course, anything I set up for py3k will be accessible by anyone, and it would include a straight conversion from the trunk, too. And for those developers that worry about having to switch (or others having to switch) from svn to something confusing: we'll keep the p3yk branch intact (possibly after a rebuild) for people to keep submitting patches against. (I could even grab commits from the svn branch and stuff them into a bzr/hg branch, but we'll see about that when we get there.) [*]: by easier, I mean simple, straightforward, explainable, reproducible and scalable, rather than a godawful pigfucking lot of work that will undoubedly get messy bugs crept in. I doubt I did all the py3k merges properly as-is, and that's not dealing with backports yet :) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070305/db0d0cf4/attachment.htm From dinov at exchange.microsoft.com Mon Mar 5 22:26:19 2007 From: dinov at exchange.microsoft.com (Dino Viehland) Date: Mon, 5 Mar 2007 13:26:19 -0800 Subject: [Python-Dev] locals(), closures, and IronPython... Message-ID: <7AD436E4270DD54A94238001769C22276A624C7482@DF-GRTDANE-MSG.exchange.corp.microsoft.com> def a(): x = 4 y = 2 def b(): print y, locals() print locals() b() a() in CPython prints: {'y': 2, 'x': 4, 'b': } 2 {'y': 2} I'm wondering if it's intentional that these don't print dictionaries w/ the same contents or if it's more an accident of the implementation. In other words would it be reasonable for IronPython to promote all of the locals of a into b's dictionary when both a and b call locals? We currently match CPython's behavior here - at least in what we display although possibly not in the lifetimes of objects. We're considering an internal change which might alter the behavior here though and end up displaying all the members. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070305/9d9e617c/attachment.html From guido at python.org Mon Mar 5 23:03:05 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 5 Mar 2007 14:03:05 -0800 Subject: [Python-Dev] Adding socket timeout to urllib2 In-Reply-To: References: Message-ID: On 3/5/07, Facundo Batista wrote: > I studied which modifications I need to make into urllib2 to support a > socket timeout. > > - I'll modify urlopen for it to accept a socket_timeout parameter, > default to None I'd call it timeout. There can't really be much ambiguity can there? > - Request will also accept a socket_timeout parameter, default to None. > It will keep it in a socket_timeout attribute, so it could be > added/modified from outside. Ditto for the name. > - OpenerDirector's 'open' method will accept socket_timeout, and build > the Request object with it > > So, when it gets into the Handlers, how it'll be used? I checked the > Handlers, and here is a resume of their 'open' or 'open'-like methods: > > - AbstractHTTPHandler: receives a class, and uses its .request and > .getresponse methods > - HTTPHandler: just calls AbstractHTTPHandler.do_open with > httplib.HTTPConnection > - HTTPSHandler: just calls AbstractHTTPHandler.do_open with > httplib.HTTPSConnection > - UnknownHandler: raises an Error > - FileHandler: opens a file in a local host > - FTPHandler: uses sockets in ftplib.FTP().connect, .login, .cwd, and > .ntransfercmd, all through urllib's ftwrapper object > - CacheFTPHandler: instantiates ftpwrapper object; does not uses it > directly, though. > - GopherHandler: is deprecated... > > So, these are the necessary further modifications in urllib2 beyond > those detailed at the beggining of the mail: > > - Modify AbstractHTTPHandler to pass the socket_timeout parameter to > the received class' methods. > - Modify FTPHandler to pass the socket_timeout to ftpwrapper class. > - Modify CacheFTPHandler the same way that FTPHandler > > Beyond that, I'll also need to: > > - Modify urllib.ftpwrapper to accept a socket_timeout parameter and > pass it to ftplib.FTP object. > - Add support to socket_timeout to ftplib's FTP object > - Add support to socket_timeout to httplib's HTTPConnection and > HTTPSConnection objects > > So, I think that in the name of clarity, I shall first modify ftplib > and httplib, add socket_timeout where is necessary, and then add > support to that in the more general urllib2. > > What do you think? Comments are very appreciated. I say your next step should be to submit a patch, perhaps incorporating the work that Skip has already done. You could also reduce the amount of work to be done initially by only handling http. At this point I expect that'll cover 99% of all uses. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Mar 5 23:13:39 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 5 Mar 2007 14:13:39 -0800 Subject: [Python-Dev] locals(), closures, and IronPython... In-Reply-To: <7AD436E4270DD54A94238001769C22276A624C7482@DF-GRTDANE-MSG.exchange.corp.microsoft.com> References: <7AD436E4270DD54A94238001769C22276A624C7482@DF-GRTDANE-MSG.exchange.corp.microsoft.com> Message-ID: Jeremy Hylton has been asking questions about this too at the sprint after PyCon. I'm tempted to accept that the exact behavior of locals() is implementation-defined (IOW undefined :-) as long as it includes the locals defined in the current scope; whether it also includes free variables could be debatable. I don't know too many good use cases for locals() apart from "learning about the implementation" I think this might be okay. Though a PEP might be in order to get agreement between users, developers and other implementation efforts (e.g. PyPy, Jython). On 3/5/07, Dino Viehland wrote: > > > > > def a(): > > x = 4 > > y = 2 > > def b(): > > print y, locals() > > print locals() > > b() > > > > a() > > > > in CPython prints: > > > > {'y': 2, 'x': 4, 'b': } > > 2 {'y': 2} > > > > I'm wondering if it's intentional that these don't print dictionaries w/ the > same contents or if it's more an accident of the implementation. In other > words would it be reasonable for IronPython to promote all of the locals of > a into b's dictionary when both a and b call locals? > > > > We currently match CPython's behavior here ? at least in what we display > although possibly not in the lifetimes of objects. We're considering an > internal change which might alter the behavior here though and end up > displaying all the members. > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From nas at arctrix.com Tue Mar 6 00:18:16 2007 From: nas at arctrix.com (Neil Schemenauer) Date: Mon, 5 Mar 2007 23:18:16 +0000 (UTC) Subject: [Python-Dev] Encouraging developers References: <20070305184614.GA19051@localhost.localdomain> Message-ID: A.M. Kuchling wrote: > At PyCon, there was general agreement that exposing a read-only > Bazaar/Mercurial/git/whatever version of the repository wouldn't be > too much effort, and might make things easier for external people > developing patches. Thomas Wouters apparently has private scripts > that perform the conversion. What needs to be done to move ahead with > this idea? Using git-svn to track a SVN repository seems to work well. It would be trivial to setup a cron job on one of the python.org machines that would create a publicly accessible repository. To get changes back into SVN is pretty easy too. Someone with SVN access would pull the changes into their own git repository and then use git-svn to commit them. In my experience, committed changes look just like they would if they were done without git. There are other tools out there for Bzr and Mercurial but I have no experience with them. I've used Bzr but git seems a better fit for python-dev, even though it's harder to learn. If some decides to do this, I suggest reading http://git.or.cz/gitwiki/GitCommonMistakes . Neil From phil at riverbankcomputing.co.uk Tue Mar 6 00:26:46 2007 From: phil at riverbankcomputing.co.uk (Phil Thompson) Date: Mon, 5 Mar 2007 23:26:46 +0000 Subject: [Python-Dev] Encouraging developers In-Reply-To: <9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> References: <20070305184614.GA19051@localhost.localdomain> <9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> Message-ID: <200703052326.46667.phil@riverbankcomputing.co.uk> On Monday 05 March 2007 9:38 pm, Thomas Wouters wrote: > On 3/5/07, A.M. Kuchling wrote: > > >From > > > 4. The patch mafia. I like everyone on python-dev that I meet, > > but somehow it is annoyingly difficult to get a patch into > > Python. Like threading, and the stdlib, this is a mixed > > blessing: you certainly don't want every Joe Schmoe checking > > in whatever crud he wants. However, the barrier is high enough > > that I no longer have much interest in spending the time to > > shepherd a patch through. Yes, this is probably all my fault > > -- but I still hate it! > > > > FWIW, I have a related perception that we aren't getting new core > > developers. These two problems are probably related: people don't get > > patches processed and don't become core developers, and we don't have > > enough core developers to process patches in a timely way. And so > > we're stuck. > > > > Any ideas for fixing this problem? > > A better patch-tracker, better procedures for reviewing patches surounding > this new tracker, one or more proper dvcs's for people to work off of. I'm > not sure about 'identifying core developers' as we're all volunteers, with > dayjobs for the most part, and only a few people seem to care enough about > python as a whole. I don't think that that is true. I think a lot of people care, but many can't do anything about because the barrier to entry is too great. > Putting the burden of patch review on the developers > that say they can cover it might easily burn them out. (I see Martin handle > a lot of patches, for instance, and I would love to help him, but I just > can't find the time to review the patches on subjects I know much about, > let alone the rest of the patches.) > > While submitting patches is good, there's a lot more to it than just > submitting the 5-line code change to submit a bug/feature, and reviewing > takes a lot of time and effort. So there is something wrong there as well. > I don't think it's unreasonable to ask for > help from the submitters like we do, or ask them to write tests and docs > and such. Of course it's not unreasonable. I would expect to be told that a patch must have tests and docs before it will be finally accepted. However, before I add those things to the patch I would like some timely feedback from those with more experience that my patch is going in the right direction. I want somebody to give it a quick look, not a full blown review. The process needs to keep people involved in it - at the moment submitting a patch is fire-and-forget. Phil From nas at arctrix.com Tue Mar 6 00:30:06 2007 From: nas at arctrix.com (Neil Schemenauer) Date: Mon, 5 Mar 2007 23:30:06 +0000 (UTC) Subject: [Python-Dev] Encouraging developers References: <20070305184614.GA19051@localhost.localdomain> Message-ID: Neil Schemenauer wrote: > Using git-svn to track a SVN repository seems to work well. It > would be trivial to setup a cron job on one of the python.org > machines that would create a publicly accessible repository. I guess Andrew was looking for specific instructions. Install git and git-svn. For Debian stable, you can get them from http://backports.org/debian/pool/main/g/git-core/. Initialize the repository: git-svn init http://svn.foo.org/project/trunk Fetch versions from SVN: git-svn fetch I think the fetch can be run periodically from a cron job. The repository can be cloned via HTTP but it's much faster to use the git server which runs on it's own TCP port. Neil From tjreedy at udel.edu Tue Mar 6 00:31:05 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 5 Mar 2007 18:31:05 -0500 Subject: [Python-Dev] Encouraging developers References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> Message-ID: "Phil Thompson" wrote in message news:200703051930.20868.phil at riverbankcomputing.co.uk... | On Monday 05 March 2007 6:46 pm, A.M. Kuchling wrote: | > FWIW, I have a related perception that we aren't getting new core | > developers. These two problems are probably related: people don't get | > patches processed and don't become core developers, and we don't have | > enough core developers to process patches in a timely way. And so | > we're stuck. | > | > Any ideas for fixing this problem? | | 1. Don't suggest to people that, in order to get their patch reviewed, they | should review other patches. The level of knowledge required to put together | a patch is much less than that required to know if a patch is the right one. | | 2. Publically identify the core developers and their areas of expertise and | responsibility (ie. which parts of the source tree they "own"). | | 3. Provide a forum (a python-patch mailing list) where patches can be | proposed, reviewed and revised informally but quickly. | | 4. Acceptance by core developers that only half the "job" is developing the | core - the other half is mentoring potential future core developers. Tracker item review is an obvious bottleneck in Python development. In particular, reviewing patches appears not to be nearly as self-motivating as writing them, and other activities. So I think the PSF should pay one or more people to do so. Possibly set up a patch review fund and solicit donations. And, donators should get priority attention to their submissions. For commercial developers, this would probably be cheaper, given the value of their time, than reviewing five other submissions. Terry Jan Reedy From dinov at exchange.microsoft.com Tue Mar 6 00:45:48 2007 From: dinov at exchange.microsoft.com (Dino Viehland) Date: Mon, 5 Mar 2007 15:45:48 -0800 Subject: [Python-Dev] locals(), closures, and IronPython... In-Reply-To: References: <7AD436E4270DD54A94238001769C22276A624C7482@DF-GRTDANE-MSG.exchange.corp.microsoft.com> Message-ID: <7AD436E4270DD54A94238001769C22276A624C7574@DF-GRTDANE-MSG.exchange.corp.microsoft.com> Thanks Guido. It might take some time (and someone may very well beat me to it if they care a lot) but I'll see if we can get the PEP started. -----Original Message----- From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido van Rossum Sent: Monday, March 05, 2007 2:14 PM To: Dino Viehland Cc: python-dev at python.org Subject: Re: [Python-Dev] locals(), closures, and IronPython... Jeremy Hylton has been asking questions about this too at the sprint after PyCon. I'm tempted to accept that the exact behavior of locals() is implementation-defined (IOW undefined :-) as long as it includes the locals defined in the current scope; whether it also includes free variables could be debatable. I don't know too many good use cases for locals() apart from "learning about the implementation" I think this might be okay. Though a PEP might be in order to get agreement between users, developers and other implementation efforts (e.g. PyPy, Jython). On 3/5/07, Dino Viehland wrote: > > > > > def a(): > > x = 4 > > y = 2 > > def b(): > > print y, locals() > > print locals() > > b() > > > > a() > > > > in CPython prints: > > > > {'y': 2, 'x': 4, 'b': } > > 2 {'y': 2} > > > > I'm wondering if it's intentional that these don't print dictionaries w/ the > same contents or if it's more an accident of the implementation. In other > words would it be reasonable for IronPython to promote all of the locals of > a into b's dictionary when both a and b call locals? > > > > We currently match CPython's behavior here - at least in what we display > although possibly not in the lifetimes of objects. We're considering an > internal change which might alter the behavior here though and end up > displaying all the members. > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Tue Mar 6 00:54:04 2007 From: aahz at pythoncraft.com (Aahz) Date: Mon, 5 Mar 2007 15:54:04 -0800 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> Message-ID: <20070305235404.GB10038@panix.com> On Mon, Mar 05, 2007, Terry Reedy wrote: > > Tracker item review is an obvious bottleneck in Python development. > In particular, reviewing patches appears not to be nearly as > self-motivating as writing them, and other activities. So I think the > PSF should pay one or more people to do so. Possibly set up a patch > review fund and solicit donations. And, donators should get priority > attention to their submissions. For commercial developers, this would > probably be cheaper, given the value of their time, than reviewing > five other submissions. There also seems to be a sense that we're waiting to get off SourceForge and using our own tracker. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "I disrespectfully agree." --SJM From tjreedy at udel.edu Tue Mar 6 01:02:47 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 5 Mar 2007 19:02:47 -0500 Subject: [Python-Dev] locals(), closures, and IronPython... References: <7AD436E4270DD54A94238001769C22276A624C7482@DF-GRTDANE-MSG.exchange.corp.microsoft.com> Message-ID: "Dino Viehland" wrote in message news:7AD436E4270DD54A94238001769C22276A624C7482 at DF-GRTDANE-MSG.exchange.corp.microsoft.com... def a(): x = 4 y = 2 def b(): print y, locals() print locals() b() a() in CPython prints: {'y': 2, 'x': 4, 'b': } 2 {'y': 2} I'm wondering if it's intentional that these don't print dictionaries w/ the same contents or if it's more an accident of the implementation. In other words would it be reasonable for IronPython to promote all of the locals of a into b's dictionary when both a and b call locals? ========================================== This version def a(): x = 4 y = 2 def b(): print y, locals() print locals() return b a()() has essentially the same output, as it should. Do you really want the binding of 'x' and 'b' to survive the a's return? I see no reason why a's call of locals() should affect this either way. Which is to say, why the compilation of b should be affected by the code that follows it. This version also has the same output def a(): x = 4 def b(): print y, locals() y = 2 print locals() return b a()() whereas this omits y from a's output, but not b's: def a(): x = 4 def b(): print y, locals() print locals() y = 2 return b a()() and would also if b were called instead of returned, as in your version. So it would not make too much sense for the two printouts to match. Terry Jan Reedy From rasky at develer.com Tue Mar 6 01:32:55 2007 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 06 Mar 2007 01:32:55 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <200703051930.20868.phil@riverbankcomputing.co.uk> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> Message-ID: On 05/03/2007 20.30, Phil Thompson wrote: > 1. Don't suggest to people that, in order to get their patch reviewed, they > should review other patches. The level of knowledge required to put together > a patch is much less than that required to know if a patch is the right one. +1000. > 2. Publically identify the core developers and their areas of expertise and > responsibility (ie. which parts of the source tree they "own"). I think this should be pushed to its extreme consequences for the standard library. Patching the standard library requires *much less* knowledge than patching the standard core. Basically, almost any Python developer in the wild can quickly learn a module and start patching it in a few days/weeks -- still, the stdlib is a total mess of outdated and broken modules. My suggestion is: - keep a public list of official maintainers for each and every package/module in the standard library (if any, otherwise explicitly specify that it's unmaintained). - if there's no maintainer for a module, the *first* volunteer can become so. - *any* patch to stdlib which follows the proper guidelines (have a test, don't break compatibility, etc.) *must* be applied *unless* the maintainer objects in X days (if a maintainer exists... otherwise it will just go in). > 4. Acceptance by core developers that only half the "job" is developing the > core - the other half is mentoring potential future core developers. Acceptance that any patch is better than no patch. There are many valid Python programmers out there, and there are many many patches to stdlib which really don't even require a good programmer to be written. -- Giovanni Bajo From rasky at develer.com Tue Mar 6 01:38:30 2007 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 06 Mar 2007 01:38:30 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070305184614.GA19051@localhost.localdomain> References: <20070305184614.GA19051@localhost.localdomain> Message-ID: On 05/03/2007 19.46, A.M. Kuchling wrote: > At PyCon, there was general agreement that exposing a read-only > Bazaar/Mercurial/git/whatever version of the repository wouldn't be > too much effort, and might make things easier for external people > developing patches. I really believe this is just a red herring, pushed by some SCM wonk. The problem with patch submission has absolutely *nothing* to do with tools. Do we have any evidence that new developers are getting frustrated because they can't handle their patches well enough with the current tools? -- Giovanni Bajo From brett at python.org Tue Mar 6 01:42:31 2007 From: brett at python.org (Brett Cannon) Date: Mon, 5 Mar 2007 16:42:31 -0800 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> Message-ID: On 3/5/07, Giovanni Bajo wrote: > On 05/03/2007 19.46, A.M. Kuchling wrote: > > > At PyCon, there was general agreement that exposing a read-only > > Bazaar/Mercurial/git/whatever version of the repository wouldn't be > > too much effort, and might make things easier for external people > > developing patches. > > I really believe this is just a red herring, pushed by some SCM wonk. The > problem with patch submission has absolutely *nothing* to do with tools. Do we > have any evidence that new developers are getting frustrated because they > can't handle their patches well enough with the current tools? We asked people attending the Python-Dev panel at PyCon whether the use of a distributed VCS would encourage them to work on the core so that they could do their own commits in their own branch and some people did raise their hands. Plus it's just handy sometimes to be able to do commits when you lack network access. -Brett From collinw at gmail.com Tue Mar 6 01:45:07 2007 From: collinw at gmail.com (Collin Winter) Date: Mon, 5 Mar 2007 18:45:07 -0600 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> Message-ID: <43aa6ff70703051645j20f15dc8s39a1218de927f0c5@mail.gmail.com> On 3/5/07, Brett Cannon wrote: > On 3/5/07, Giovanni Bajo wrote: > > On 05/03/2007 19.46, A.M. Kuchling wrote: > > > > > At PyCon, there was general agreement that exposing a read-only > > > Bazaar/Mercurial/git/whatever version of the repository wouldn't be > > > too much effort, and might make things easier for external people > > > developing patches. > > > > I really believe this is just a red herring, pushed by some SCM wonk. The > > problem with patch submission has absolutely *nothing* to do with tools. Do we > > have any evidence that new developers are getting frustrated because they > > can't handle their patches well enough with the current tools? > [snip] > > Plus it's just handy sometimes to be able to do commits when you lack > network access. Seconded. I don't know how much it impacts new developers, but I know I get frustrated because of this. Collin Winter From amk at amk.ca Tue Mar 6 02:41:38 2007 From: amk at amk.ca (A.M. Kuchling) Date: Mon, 5 Mar 2007 20:41:38 -0500 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> Message-ID: <20070306014138.GA9697@andrew-kuchlings-computer.local> On Mon, Mar 05, 2007 at 11:30:06PM +0000, Neil Schemenauer wrote: > I guess Andrew was looking for specific instructions. ... I'm happy to let the ball sit in Thomas's court until the Bazaar developers come out with 0.15 and run their conversion on the SVN repository. There's no burning hurry about getting this set up, so a few weeks of waiting is fine. --amk From amk at amk.ca Tue Mar 6 02:53:07 2007 From: amk at amk.ca (A.M. Kuchling) Date: Mon, 5 Mar 2007 20:53:07 -0500 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070305205046.22258.117669926.divmod.xquotient.56@joule.divmod.com> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <20070305205046.22258.117669926.divmod.xquotient.56@joule.divmod.com> Message-ID: <20070306015307.GB9697@andrew-kuchlings-computer.local> On Mon, Mar 05, 2007 at 08:50:46PM -0000, glyph at divmod.com wrote: > is indicative of a failure of the community. A good deal of the > discussion here in recent months has either been highly speculative, or > only tangentially related to Python's development, which is ostensibly > its topic. We shouldn't really be talking about PR or deployment or any > issues which aren't bug reports or patches here. I don't recall any PR threads in python-dev, but do agree that language speculations usually lead to long threads that are very boring. It would be nice to focus more on concrete development questions, and usually we do manage to focus when there's an impending release. One problem may be that there *aren't* maintainers for various subsystems; various people have contributed bugfixes and patches for, say, urllib, but I have no idea what single person I would go to for a problem. Is it worth creating a wiki page listing people and the modules they're responsible for? Or does something like this already exist? --amk From jcarlson at uci.edu Tue Mar 6 03:11:43 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 05 Mar 2007 18:11:43 -0800 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <200703051930.20868.phil@riverbankcomputing.co.uk> Message-ID: <20070305175116.749D.JCARLSON@uci.edu> Giovanni Bajo wrote: > On 05/03/2007 20.30, Phil Thompson wrote: > > 2. Publically identify the core developers and their areas of expertise and > > responsibility (ie. which parts of the source tree they "own"). > > I think this should be pushed to its extreme consequences for the standard > library. Patching the standard library requires *much less* knowledge than > patching the standard core. Basically, almost any Python developer in the wild > can quickly learn a module and start patching it in a few days/weeks -- still, > the stdlib is a total mess of outdated and broken modules. Sometimes code is correct. Sometimes code is complete. Just because a module hasn't been significantly altered or updated recently doesn't mean that it is outdated or broken. Asyncore, for example, is more or less feature complete for a minimal asynchronous socket framework. Could it gain more features (like good SSL support) or have better error handling? Of course. But it passes the test suite (via test_asynchat), so I would consider it *not* broken. > My suggestion is: > > - keep a public list of official maintainers for each and every > package/module in the standard library (if any, otherwise explicitly specify > that it's unmaintained). > - if there's no maintainer for a module, the *first* volunteer can become so. > - *any* patch to stdlib which follows the proper guidelines (have a test, > don't break compatibility, etc.) *must* be applied *unless* the maintainer > objects in X days (if a maintainer exists... otherwise it will just go in). Just because a patch doesn't break a module, doesn't mean the patch should be applied. Take, for example, any one of the patches currently offered against asyncore. One of them more or less changes the internal structure of the module for no other reason than to (in my opinion) be gratuitous. Certainly it has some good ideas, but it would be better to adapt portions rather than take it completely - which is what you are suggesting, and which is what would have happened due to the absence of an asyncore maintainer (until I took it up). > > 4. Acceptance by core developers that only half the "job" is developing the > > core - the other half is mentoring potential future core developers. > > Acceptance that any patch is better than no patch. There are many valid Python > programmers out there, and there are many many patches to stdlib which really > don't even require a good programmer to be written. Maybe, but there are also many patches I've seen that cause the resulting code to not run, revert changes made to fix bugs, etc. Vetting patches is a pain in the butt, and accepting all patches that aren't objected to is a braindead approach to patching the standard library. Indeed, every module and package should have a known and documented maintainer, but that's really the only part of your suggestions in the message I'm replying to that I would agree with. The rest gets a -1. - Josiah From collinw at gmail.com Tue Mar 6 03:12:07 2007 From: collinw at gmail.com (Collin Winter) Date: Mon, 5 Mar 2007 20:12:07 -0600 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070306015307.GB9697@andrew-kuchlings-computer.local> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <20070305205046.22258.117669926.divmod.xquotient.56@joule.divmod.com> <20070306015307.GB9697@andrew-kuchlings-computer.local> Message-ID: <43aa6ff70703051812l5412087fgd5d9c5990549a1d2@mail.gmail.com> On 3/5/07, A.M. Kuchling wrote: [snip] > One problem may be that there *aren't* maintainers for various > subsystems; various people have contributed bugfixes and patches for, > say, urllib, but I have no idea what single person I would go to for a > problem. > > Is it worth creating a wiki page listing people and the modules they're > responsible for? Or does something like this already exist? I'd rather have this kind of ownership information attached to the individual module documentation, so interested parties don't have to go hunting around in the wiki for it. It would also be helpful to have some loose guidelines/requirements for how to become a module maintainer (e.g., "we're looking for the following traits..."). Collin Winter From eric+python-dev at trueblade.com Tue Mar 6 02:57:38 2007 From: eric+python-dev at trueblade.com (Eric V. Smith) Date: Mon, 05 Mar 2007 20:57:38 -0500 Subject: [Python-Dev] Access to bits for a PyLongObject Message-ID: <45ECCA92.8090709@trueblade.com> I'm working on PEP 3101, Advanced String Formatting. About the only built-in numeric formatting I have left to do is for converting a PyLongOjbect to binary. I need to know how to access the bits in a PyLong. After reading longobject.c, I can figure it out. But I'm looking for something that might be preferable to me poking around directly in ob_size and ob_digit[]. I'm looking for something along the lines of: for (i = 0; i < _PyLong_NumBits(v); i++) { // get i-th bit of v } I don't care if it's increasing or decreasing bits, I can handle either. I realize the actual code will likely involve the 2 nested loops, but logically this is what I need. I can certainly do this myself, by looping over ob_digit and then over each bit. But there's a comment in longintrepr.h that says the internals are published only for marshal.c. Should I go ahead and include longintrepr.h and loop over ob_digit myself, or is there some other method? I really don't want to use _PyLong_AsByteArray, since I don't want to do the copy. If I'm missing some PyLongObject API, please let me know. By the way, while doing this I noticed a bug in stringobject.c and unicodeobject.c, relating to a missing check for 'G' precision. I'm not sure if it could be a buffer overflow or not, without spending some more time analyzing at it. But it seems like the potential is certainly there. The bug is at http://python.org/sf/1673757 and a patch at http://python.org/sf/1673759. (I'll save my comments on how "approachable" python-dev is after I have this question answered!) Thanks for your time. Eric. From martin at v.loewis.de Tue Mar 6 05:59:49 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 05:59:49 +0100 Subject: [Python-Dev] Access to bits for a PyLongObject In-Reply-To: <45ECCA92.8090709@trueblade.com> References: <45ECCA92.8090709@trueblade.com> Message-ID: <45ECF545.2080806@v.loewis.de> Eric V. Smith schrieb: > I'm working on PEP 3101, Advanced String Formatting. About the only > built-in numeric formatting I have left to do is for converting a > PyLongOjbect to binary. > > I need to know how to access the bits in a PyLong. I think it would be a major flaw in PEP 3101 if you really needed it. The long int representation should be absolutely opaque - even the fact that it is a sign+magnitude representation should be hidden. Looking at the PEP, I see that a class can implement __format__. Wouldn't it be appropriate if the long type implemented that? Implementation-wise, I would expect that long_format already does the bulk of what you need to do. OTOH, also look at _PyString_FormatLong. Regards, Martin From martin at v.loewis.de Tue Mar 6 06:20:37 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 06:20:37 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070305184614.GA19051@localhost.localdomain> References: <20070305184614.GA19051@localhost.localdomain> Message-ID: <45ECFA25.6070506@v.loewis.de> A.M. Kuchling schrieb: > FWIW, I have a related perception that we aren't getting new core > developers. These two problems are probably related: people don't get > patches processed and don't become core developers, and we don't have > enough core developers to process patches in a timely way. And so > we're stuck. I think this perception is partially wrong. The number of unreviewed patches has been sitting around 400 for quite some time now - so it has not been getting worse. This is mostly thanks to the (very) few reviewers that deal with patches in areas out of their primary interest. I'd like to mention Georg Brandl here. I also doubt that accepting patches more quickly would give many more patch reviewers. Most submitters are happy if their patch is accepted, and couldn't care less about other people's patches. This is fine, of course - it just means that becoming more responsive (by whatever means) would not necessarily sustain. > Any ideas for fixing this problem? I had this long-term offer of a 5:1 deal. I wish more current committers would offer a similar deal, then we would be able to promise this policy prominently. This, of course, requires that committers are willing to deal with patches even if they are no experts in the subject (i.e. they will need to become experts in the process of reviewing). It might be possible to reverse this policy also: we could decide that committers maintain their write privilege only if they process patches (say, one patch per month). That would be quite intrusive, so I doubt that committers will agree to it. > Tangentially related: > At PyCon, there was general agreement that exposing a read-only > Bazaar/Mercurial/git/whatever version of the repository wouldn't be > too much effort, and might make things easier for external people > developing patches. Thomas Wouters apparently has private scripts > that perform the conversion. What needs to be done to move ahead with > this idea? If it is this "distributed repository" aspect that people are after, I suggest they use svk (http://svk.elixus.org/view/HomePage). People can use it now if they want to, no need to provide additional infrastructure. For other systems, there is a choice of either hosting them at svn.python.org as well (i.e. dinsdale), or having the community host them. For dinsdale hosting, it requires a volunteer to set it up and maintain it. Given the low number of volunteers available for such tasks, I doubt this can work. For community hosting, again there is little administration necessary: hosters can already mirror svn.python.org, and run whatever conversion scripts are necessary. In this case, the task would be merely to communicate that people can already do that if they want to. Regards Martin P.S. I'm really pissed as being described as member of a mafia. From martin at v.loewis.de Tue Mar 6 06:42:15 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 06:42:15 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <200703051930.20868.phil@riverbankcomputing.co.uk> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> Message-ID: <45ECFF37.6090901@v.loewis.de> Phil Thompson schrieb: > 1. Don't suggest to people that, in order to get their patch reviewed, they > should review other patches. The level of knowledge required to put together > a patch is much less than that required to know if a patch is the right one. People don't *have* to review patches. They just can do that if they want expedite review of their code. > 2. Publically identify the core developers and their areas of expertise and > responsibility (ie. which parts of the source tree they "own"). I doubt this will help. Much of the code isn't owned by anybody specifically. Those parts that are owned typically find their patches reviewed and committed quickly (e.g. the tar file module, maintained by Lars Gust?bel). > 4. Acceptance by core developers that only half the "job" is developing the > core - the other half is mentoring potential future core developers. So what do you do with core developers that don't do their job? Fire them? Regards, Martin From martin at v.loewis.de Tue Mar 6 06:47:49 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 06 Mar 2007 06:47:49 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070305205046.22258.117669926.divmod.xquotient.56@joule.divmod.com> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <20070305205046.22258.117669926.divmod.xquotient.56@joule.divmod.com> Message-ID: <45ED0085.3040803@v.loewis.de> glyph at divmod.com schrieb: > The patches list really ought to be _this_ list. The fact that it isn't > is indicative of a failure of the community. I disagree that python-dev isn't the patches list. People often discuss patches here (although much discussion is also in the tracker). > One worrying trend I noticed at PyCon is that it seems that quite a lot > of communication between core developers these days happens over private > email. Do you know this for a fact? I'm a core developer, and I don't use private email much to discuss patches or other changes. About the only private mail that I exchange is about the mechanics of the release process (e.g. agreeing on specific days for a release). Regards, Martin From martin at v.loewis.de Tue Mar 6 06:49:48 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 06:49:48 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <200703052116.48035.phil@riverbankcomputing.co.uk> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <20070305200902.GA21284@phd.pp.ru> <200703052116.48035.phil@riverbankcomputing.co.uk> Message-ID: <45ED00FC.6050406@v.loewis.de> Phil Thompson schrieb: > I'm not sure what your point is. My point is that, if you want to encourage > people to become core developers, they have to have a method of graduating > through the ranks - learning (and being taught) as they go. To place a very > high obstacle in their way right at the start is completely > counter-productive. And please be assured that no such obstacle is in the submitters way. Most patches are accepted without the submitter actually reviewing any other patches. Regards, Martin From martin at v.loewis.de Tue Mar 6 07:00:17 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 07:00:17 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <200703052326.46667.phil@riverbankcomputing.co.uk> References: <20070305184614.GA19051@localhost.localdomain> <9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> Message-ID: <45ED0371.5070108@v.loewis.de> Phil Thompson schrieb: >>> Any ideas for fixing this problem? >> A better patch-tracker, better procedures for reviewing patches surounding >> this new tracker, one or more proper dvcs's for people to work off of. I'm >> not sure about 'identifying core developers' as we're all volunteers, with >> dayjobs for the most part, and only a few people seem to care enough about >> python as a whole. > > I don't think that that is true. I think a lot of people care, but many can't > do anything about because the barrier to entry is too great. He was talking about the committers specifically who don't care about Python as-a-whole, and I think this is true. But I also believe that many contributors don't "care" about Python as-a-whole, in the sense that they are uninterested in learning about implementation details of libraries they will never use. What they do care about is the problems they have, and they do contribute patches for them. >> While submitting patches is good, there's a lot more to it than just >> submitting the 5-line code change to submit a bug/feature, and reviewing >> takes a lot of time and effort. > > So there is something wrong there as well. > >> I don't think it's unreasonable to ask for >> help from the submitters like we do, or ask them to write tests and docs >> and such. > > Of course it's not unreasonable. I would expect to be told that a patch must > have tests and docs before it will be finally accepted. However, before I add > those things to the patch I would like some timely feedback from those with > more experience that my patch is going in the right direction. This cannot work. It is very difficult to review a patch to fix a presumed bug if there is no test case. You might not be able to reproduce the patch without a test case at all - how could you then know whether the patch actually fixes the bug? So I really think patches should be formally complete before being submitted. This is an area were anybody can review: you don't need to be an expert to see that no test cases are contributed to a certain patch. If you really want to learn and help, review a few patches, to see what kinds of problems you detect, and then post your findings to python-dev. People then will comment on whether they agree with your review, and what additional changes they like to see. Regards, Martin From martin at v.loewis.de Tue Mar 6 07:02:03 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 07:02:03 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> Message-ID: <45ED03DB.6060506@v.loewis.de> Neil Schemenauer schrieb: > I guess Andrew was looking for specific instructions. Install git > and git-svn. For Debian stable, you can get them from > http://backports.org/debian/pool/main/g/git-core/. > > Initialize the repository: > > git-svn init http://svn.foo.org/project/trunk > > Fetch versions from SVN: > > git-svn fetch > > I think the fetch can be run periodically from a cron job. The > repository can be cloned via HTTP but it's much faster to use the > git server which runs on it's own TCP port. And that's it? Won't you need to publish the repository somehow? Apache configuration? init.d scripts? Regards, Martin From martin at v.loewis.de Tue Mar 6 07:08:41 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 07:08:41 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <43aa6ff70703051812l5412087fgd5d9c5990549a1d2@mail.gmail.com> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <20070305205046.22258.117669926.divmod.xquotient.56@joule.divmod.com> <20070306015307.GB9697@andrew-kuchlings-computer.local> <43aa6ff70703051812l5412087fgd5d9c5990549a1d2@mail.gmail.com> Message-ID: <45ED0569.4080903@v.loewis.de> Collin Winter schrieb: > It would also be helpful to have some loose guidelines/requirements > for how to become a module maintainer (e.g., "we're looking for the > following traits..."). That is easy to answer: Review the patches of the module, and post a message to python-dev about your findings (proposals of acceptance or rejection). Regards, Martin From python at rcn.com Tue Mar 6 07:15:33 2007 From: python at rcn.com (Raymond Hettinger) Date: Mon, 5 Mar 2007 22:15:33 -0800 Subject: [Python-Dev] Encouraging developers References: <20070305184614.GA19051@localhost.localdomain><9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> Message-ID: <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> [Phil Thompson] > I think a lot of people care, but many can't > do anything about because the barrier to entry is too great. Do you mean commit priviledges? ISTM, those tend to be handed out readily to people who assert that they have good use for them. Ask the Georg-bot how readily he was accepted and coached. IMO, his acceptance was a model that all open source projects should aspire to. If you meant something else like knowing how to make a meaningful patch, then you've got a point. It takes a while to learn your way around the source tree and to learn the inter-relationships between the moving parts. That is just the nature of the beast. [MvL] >> While submitting patches is good, there's a lot more to it than just >> submitting the 5-line code change to submit a bug/feature, and reviewing >> takes a lot of time and effort. [Phil] > So there is something wrong there as well. I have not idea what you're getting at. Martin's comment seems accurate to me. Unless it is a simple typo/doc fix, it takes some time to assess whether the bug is real (some things are bugs only in the eye of the submitter) and whether the given fix is the right thing to do. Of course, automatic acceptance of patches would be a crummy idea. There have been no shortage of patches complete with docs and tests that were simply not the right thing to do. [Phil] > The process needs > to keep people involved in it - at the moment submitting a patch is > fire-and-forget. Such is the nature of a system of volunteers. If we had full-time people, it could be a different story. IMO, given an 18 month release cycle, it is perfectly acceptable for a patch to sit for a while until someone with the relavant expertise can address it. Even with a tests and docs, patch acceptance is far from automatic. That being said, I think history has shown that important bugs get addressed and put into bug fix releases without much loss of time. When Py2.5.1 goes out, I expect that all known, important bugs will have been addressed and that's not bad. Raymond From martin at v.loewis.de Tue Mar 6 07:19:26 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 07:19:26 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45ED0353.2090201@scottdial.com> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <45ED0353.2090201@scottdial.com> Message-ID: <45ED07EE.4060402@v.loewis.de> Scott Dial schrieb: > While I understand that this tit-for-tat mechanism is meant to ensure > participation, I believe in reality it doesn't, as the 400-some > outstanding patches you referenced elswhere indicate. I can personally > attest to having a patch that is over a year old with no "core > developer" having any interest at all with the subject matter. And to be > frank, nor did I really, but I saw a problem and was capable of solving > it. My lack of caring about the patch means I am not going to beat > people over the head to pay attention. This system is broken for someone > like me (coder) that just wants to help out (non-coders). If you don't care that much about the patch, it's not broken. As I said before, the number of unreviewed patches has been roughly stable for some time now. If the patch is not really important, it may take two years now to get it in, but eventually, it will (if you then still are interested to work on it to complete it). > If nothing else, as an outsider there is no way to know why your patch > gets ignored while others get swiftly dealt with. Any sort of > information like this would at least provide more transparency in what > may appear to be elitest processes. This is what we would need volunteer reviewers for. We can send machine confirmations, but I doubt it would help. If you need a human response, somebody must send you one, demonstrating that they actually did look at the patch. If none of the committers have the time to do so, somebody else must send the manual confirmation. Regards, Martin From martin at v.loewis.de Tue Mar 6 07:22:12 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 07:22:12 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> References: <20070305184614.GA19051@localhost.localdomain><9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> Message-ID: <45ED0894.2060209@v.loewis.de> Raymond Hettinger schrieb: > [Phil Thompson] >> I think a lot of people care, but many can't >> do anything about because the barrier to entry is too great. > > Do you mean commit priviledges? ISTM, those tend to be > handed out readily to people who assert that they have good use for them. > Ask the Georg-bot how readily he was accepted and coached. IMO, > his acceptance was a model that all open source projects should aspire to. Indeed. IIRC, he first posted a message (under pseudonym at the time): "I reviewed these patches because I didn't have anything better to do". Shortly afterwards, he was a committer. Regards, Martin From stephen at xemacs.org Tue Mar 6 07:46:30 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 06 Mar 2007 15:46:30 +0900 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> Message-ID: <87irdeamyh.fsf@uwakimon.sk.tsukuba.ac.jp> Giovanni Bajo writes: > On 05/03/2007 20.30, Phil Thompson wrote: >> 1. Don't suggest to people that, in order to get their patch >> reviewed, they should review other patches. The level of knowledge >> required to put together a patch is much less than that required >> to know if a patch is the right one. > +1000. Been there, done that. If the submitter doesn't have a pretty good idea that a patch is "the right one", he's making substantial demands on core developer time. "+1000" suggests you think that core developers are using their time with ruinously low efficiency. I think the "review some patches" test is exactly the right one. The requirement is not that you channel the BFDL, it's that you read the patch and bring your own knowledge of the problem to bear. This has three benefits: (1) the reviewed patches get some comments; (2) the reviewed patches come to the attention of a committer; (3) the reviewer comes to the attention of a core developer, and can be considered for commit privileges, etc. An informal version of this process is how XEmacs identifies its Reviewers (the title we give to those privileged to authorize commits to all parts of XEmacs). People who care enough to make technical comments on *others'* patches are rare, and we grab the competent ones pretty quickly. >> 2. Publically identify the core developers and their areas of >> expertise and responsibility (ie. which parts of the source tree >> they "own"). The XEmacs experience has been that the core developers are "ministers without portfolio". You can't wait around for the "owner", who may be on Mars, and you rarely need to. > I think this should be pushed to its extreme consequences for the > standard library. Patching the standard library requires *much > less* knowledge than patching the standard core. Basically, almost > any Python developer in the wild can quickly learn a module and > start patching it in a few days/weeks -- still, the stdlib is a > total mess of outdated and broken modules. The mess is not "total", as Josiah Carlson points out. To the extent that it is a mess, it is the consequence of a process similar to the one you propose to institutionalize. > My suggestion is: > > - keep a public list of official maintainers for each and every > package/module in the standard library (if any, otherwise > explicitly specify that it's unmaintained). This is what XEmacs does; it works, but it's not as effective as you might hope. What happens for us is that many modules are maintained by an interest group disjoint from the core. By giving a representative of the interest group commit privileges, things get addressed pretty quickly and competently. However, this is a convenience for users of the module more than a way of kickstarting a development process (note that the interest group already exists). This requires a separate distribution of the standard library. Two points: recall the ElementTree thread. There were other plausible candidates. The XEmacs policy in such case is that they are all considered equally, and all are allowed to be distributed with the package distribution. In Python, this would conflict with TOOWTDI. Second, where the stdlib module is closely bound to the core, the maintainer ends up being the group of core developers. It can't be any other way, it seems to me. > - if there's no maintainer for a module, the *first* volunteer > can become so. I doubt this will work. It is usually the case that the first volunteer is acceptable, but it shouldn't be policy. > - *any* patch to stdlib which follows the proper guidelines (have > a test, don't break compatibility, etc.) *must* be applied *unless* > the maintainer objects in X days (if a maintainer > exists... otherwise it will just go in). This is an obviously bad idea. The stdlib needs to be deliberately pruned, not arbitrarily patched. From stephen at xemacs.org Tue Mar 6 07:58:41 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 06 Mar 2007 15:58:41 +0900 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> Message-ID: <87hcsyame6.fsf@uwakimon.sk.tsukuba.ac.jp> Giovanni Bajo writes: > On 05/03/2007 19.46, A.M. Kuchling wrote: > > > At PyCon, there was general agreement that exposing a read-only > > Bazaar/Mercurial/git/whatever version of the repository wouldn't be > > too much effort, and might make things easier for external people > > developing patches. > I really believe this is just a red herring, pushed by some SCM > wonk. The problem with patch submission has absolutely *nothing* to > do with tools. Of course it does. How important is a matter for individual judgment, of course. The *frustration level* with the acceptance process is certainly related to the annoyance of locally maintaining a patch in the face of a flow of upstream changes. The distributed SCMs make this *much* easier, and therefore can reduce the frustration level, at *zero* expense to the core developers (anybody with read access can maintain such a read-only repo). This is a good thing. It *is* important that the core sanction and support "official" mirror repos. This may or may not help the acceptance process to improve; I believe you are correct, that it will have little direct impact on the acceptance process. Nevertheless, life for third-party developers will become somewhat more pleasant, especially as they have a much easier way to exchange and refine patches. This last can feed back into the "review for review" bargain. From ronaldoussoren at mac.com Tue Mar 6 07:59:53 2007 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 6 Mar 2007 07:59:53 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <17900.26693.804101.291817@montanaro.dyndns.org> References: <20070305184614.GA19051@localhost.localdomain> <17900.26693.804101.291817@montanaro.dyndns.org> Message-ID: On 5 Mar, 2007, at 19:58, skip at pobox.com wrote: > > amk> Any ideas for fixing this problem? > > Not a one. > > amk> Tangentially related: > amk> At PyCon, there was general agreement that exposing a read- > only > amk> Bazaar/Mercurial/git/whatever version of the repository > wouldn't be > amk> too much effort, and might make things easier for external > people > amk> developing patches. > > I'm not much of a version control wonk. How would these help? > Can't the > folks who need such stuff do some sort of anonymous svn checkout? The version management systemens mentioned are distributed systems and would allow users to have local branches, which could mak development easier for them. They can already do this using SVK, which is a distributed version control system as well but uses SVN repositories to store its data. Ronald From phil at riverbankcomputing.co.uk Tue Mar 6 09:18:29 2007 From: phil at riverbankcomputing.co.uk (Phil Thompson) Date: Tue, 6 Mar 2007 08:18:29 +0000 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45ECFF37.6090901@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> Message-ID: <200703060818.29114.phil@riverbankcomputing.co.uk> On Tuesday 06 March 2007 5:42 am, Martin v. L?wis wrote: > Phil Thompson schrieb: > > 1. Don't suggest to people that, in order to get their patch reviewed, > > they should review other patches. The level of knowledge required to put > > together a patch is much less than that required to know if a patch is > > the right one. > > People don't *have* to review patches. They just can do that if they > want expedite review of their code. > > > 2. Publically identify the core developers and their areas of expertise > > and responsibility (ie. which parts of the source tree they "own"). > > I doubt this will help. Much of the code isn't owned by anybody > specifically. Those parts that are owned typically find their patches > reviewed and committed quickly (e.g. the tar file module, maintained by > Lars Gust?bel). Doesn't your last sentence completely contradict your first sentence? > > 4. Acceptance by core developers that only half the "job" is developing > > the core - the other half is mentoring potential future core developers. > > So what do you do with core developers that don't do their job? Fire them? Of course not, but this is a cultural issue not a technical one. The first step in changing a culture is to change the expectations. Phil From phil at riverbankcomputing.co.uk Tue Mar 6 09:34:06 2007 From: phil at riverbankcomputing.co.uk (Phil Thompson) Date: Tue, 6 Mar 2007 08:34:06 +0000 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45ED00FC.6050406@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <200703052116.48035.phil@riverbankcomputing.co.uk> <45ED00FC.6050406@v.loewis.de> Message-ID: <200703060834.06715.phil@riverbankcomputing.co.uk> On Tuesday 06 March 2007 5:49 am, Martin v. L?wis wrote: > Phil Thompson schrieb: > > I'm not sure what your point is. My point is that, if you want to > > encourage people to become core developers, they have to have a method of > > graduating through the ranks - learning (and being taught) as they go. To > > place a very high obstacle in their way right at the start is completely > > counter-productive. > > And please be assured that no such obstacle is in the submitters way. > Most patches are accepted without the submitter actually reviewing any > other patches. I'm glad to hear it - but I'm talking about the perception, not the fact. When occasionally submitters ask if their patch is going to be included, they will usually get a response suggesting they review other patches. That will only strengthen the perception. This discussion started because the feeling was expressed that it was difficult to get patches accepted and that new core developers were not being found. I would love to contribute more to the development of Python - I owe it a lot - but from where I stand (which is most definitely not where you stand) I can't see how to do that in a productive and rewarding way. Phil From dalke at dalkescientific.com Tue Mar 6 09:41:53 2007 From: dalke at dalkescientific.com (Andrew Dalke) Date: Tue, 6 Mar 2007 01:41:53 -0700 Subject: [Python-Dev] locals(), closures, and IronPython... In-Reply-To: References: <7AD436E4270DD54A94238001769C22276A624C7482@DF-GRTDANE-MSG.exchange.corp.microsoft.com> Message-ID: On 3/5/07, Guido van Rossum wrote: > I don't know too many good use cases for > locals() apart from "learning about the implementation" I think this > might be okay. Since I'm watching this list for any discussion on the traceback threads, I figured I would point out the most common use I know for locals() is in string interpolation when there are many local variables, eg: a = "spam" b = "egg" ... y = "foo" z = "bar" print fmtstr % locals() The next most is to deal with a large number of input parameters, as this from decimal.py: def __init__(self, prec=None, rounding=None, traps=None, flags=None, _rounding_decision=None, Emin=None, Emax=None, capitals=None, _clamp=0, _ignored_flags=None): ... for name, val in locals().items(): if val is None: setattr(self, name, _copy.copy(getattr(DefaultContext, name))) else: setattr(self, name, val) and this example based on a post of Alex Martelli's: def __init__(self, fee, fie, foo, fum): self.__dict__.update(locals()) del self.self In both cases they are shortcuts to "reduce boilerplate". I've more often used the first form in my code. If an inner local returned a superset of the items it returns now, I would not be concerned. I've rarely used the 2nd form in my code. The only way I can see there being a problem is if a function defines a class, which then uses the locals() trick, because >>> def blah(): ... a = 6; b = 7 ... class XYZ(object): ... def __init__(self): ... c = a ... print "in the class", locals() ... print "in the function", locals() ... XYZ() ... >>> blah() in the function {'a': 6, 'XYZ': , 'b': 7} in the class {'a': 6, 'c': 6, 'self': <__main__.XYZ object at 0x72ad0>} the code in the class's initializer will have more locals. I've never seen code like this (class defined in a function, with __init__ using locals()) and it's not something someone would write thinking it was a standard way of doing things. In both cases I'm not that bothered if it's implementation specific. Using locals has the feel of being too much like a trick to get around having to type so much. That's what editor macros are for :) Andrew dalke at dalkescientific.com From phil at riverbankcomputing.co.uk Tue Mar 6 09:48:04 2007 From: phil at riverbankcomputing.co.uk (Phil Thompson) Date: Tue, 6 Mar 2007 08:48:04 +0000 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45ED0371.5070108@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <45ED0371.5070108@v.loewis.de> Message-ID: <200703060848.04117.phil@riverbankcomputing.co.uk> On Tuesday 06 March 2007 6:00 am, Martin v. L?wis wrote: > Phil Thompson schrieb: > >>> Any ideas for fixing this problem? > >> > >> A better patch-tracker, better procedures for reviewing patches > >> surounding this new tracker, one or more proper dvcs's for people to > >> work off of. I'm not sure about 'identifying core developers' as we're > >> all volunteers, with dayjobs for the most part, and only a few people > >> seem to care enough about python as a whole. > > > > I don't think that that is true. I think a lot of people care, but many > > can't do anything about because the barrier to entry is too great. > > He was talking about the committers specifically who don't care about > Python as-a-whole, and I think this is true. But I also believe that > many contributors don't "care" about Python as-a-whole, in the sense > that they are uninterested in learning about implementation details of > libraries they will never use. What they do care about is the problems > they have, and they do contribute patches for them. > > >> While submitting patches is good, there's a lot more to it than just > >> submitting the 5-line code change to submit a bug/feature, and reviewing > >> takes a lot of time and effort. > > > > So there is something wrong there as well. > > > >> I don't think it's unreasonable to ask for > >> help from the submitters like we do, or ask them to write tests and docs > >> and such. > > > > Of course it's not unreasonable. I would expect to be told that a patch > > must have tests and docs before it will be finally accepted. However, > > before I add those things to the patch I would like some timely feedback > > from those with more experience that my patch is going in the right > > direction. > > This cannot work. It is very difficult to review a patch to fix a > presumed bug if there is no test case. You might not be able to > reproduce the patch without a test case at all - how could you then > know whether the patch actually fixes the bug? Please read what I said again. Yes, a patch must be reviewed before submission. Yes, a patch when submitted must include docs and test cases. I'm talking about the less formal process leading up to that point. The less formal process has a much lower barrier to entry, requires much less effort by the "reviewer", is the period during which the majority of the teaching happens, and will result in a better quality final patch that will require less effort to be put in to the final, formal review. > So I really think patches should be formally complete before being > submitted. This is an area were anybody can review: you don't need > to be an expert to see that no test cases are contributed to a > certain patch. > > If you really want to learn and help, review a few patches, to see > what kinds of problems you detect, and then post your findings to > python-dev. People then will comment on whether they agree with your > review, and what additional changes they like to see. Do you think this actually happens in practice? There is no point sticking to a process, however sensible, if it doesn't get used. Phil From nnorwitz at gmail.com Tue Mar 6 10:00:56 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 6 Mar 2007 01:00:56 -0800 Subject: [Python-Dev] Encouraging developers In-Reply-To: <200703060834.06715.phil@riverbankcomputing.co.uk> References: <20070305184614.GA19051@localhost.localdomain> <200703052116.48035.phil@riverbankcomputing.co.uk> <45ED00FC.6050406@v.loewis.de> <200703060834.06715.phil@riverbankcomputing.co.uk> Message-ID: On 3/6/07, Phil Thompson wrote: > On Tuesday 06 March 2007 5:49 am, Martin v. L?wis wrote: > > Phil Thompson schrieb: > > > I'm not sure what your point is. My point is that, if you want to > > > encourage people to become core developers, they have to have a method of > > > graduating through the ranks - learning (and being taught) as they go. To > > > place a very high obstacle in their way right at the start is completely > > > counter-productive. > > > > And please be assured that no such obstacle is in the submitters way. > > Most patches are accepted without the submitter actually reviewing any > > other patches. > > I'm glad to hear it - but I'm talking about the perception, not the fact. When > occasionally submitters ask if their patch is going to be included, they will > usually get a response suggesting they review other patches. That will only > strengthen the perception. > > This discussion started because the feeling was expressed that it was > difficult to get patches accepted and that new core developers were not being > found. I would love to contribute more to the development of Python - I owe > it a lot - but from where I stand (which is most definitely not where you > stand) I can't see how to do that in a productive and rewarding way. I recognize there is a big problem here. Each of us as individuals don't scale. So in order to get stuff done we need to be more distributed. This means distributing the workload (partially so we don't burn out). In order to do that we need to distribute the knowledge. That probably involves some changes. I understand it's really hard to get involved. It can be frustrating at times. But I think the best way is to find a mentor. Find an existing core developer that you relate to and/or have similar interests with. I've been trying to do this more. So here's my offer. Anyone who is serious about becoming a Python core developer, but doesn't know how to get involved can mail me. Privately, publicly, it doesn't matter to me. I will try to mentor you. Be prepared! I will demand submissions are high quality which at a minimum means: - it adds a new test, enhances an existing test or fixes a broken test - it has *all* the required documentation, including versionadded/versionchanged - most people think the feature is desirable - the code is maintainable and formatted according to the prevailing style - we follow the process (which can include improving the process if others agree) ie, there's no free lunch, unless you work at Google of course :-) It also means you are willing to hold other people up to equally high standards. Your contributions don't have to be code though. They can be doc, they can be PEPs, they can be the python.org website. It could even be helping out with Google's Summer of Code. The choice is yours. n From phil at riverbankcomputing.co.uk Tue Mar 6 10:06:22 2007 From: phil at riverbankcomputing.co.uk (Phil Thompson) Date: Tue, 6 Mar 2007 09:06:22 +0000 Subject: [Python-Dev] Encouraging developers In-Reply-To: <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> Message-ID: <200703060906.22762.phil@riverbankcomputing.co.uk> On Tuesday 06 March 2007 6:15 am, Raymond Hettinger wrote: > [Phil Thompson] > > > I think a lot of people care, but many can't > > do anything about because the barrier to entry is too great. > > Do you mean commit priviledges? ISTM, those tend to be > handed out readily to people who assert that they have good use for them. > Ask the Georg-bot how readily he was accepted and coached. IMO, > his acceptance was a model that all open source projects should aspire to. > > If you meant something else like knowing how to make a meaningful patch, > then you've got a point. It takes a while to learn your way around the > source tree and to learn the inter-relationships between the moving parts. > That is just the nature of the beast. I meant the second. While it may be the nature that doesn't mean that the situation can't be improved. > [MvL] > > >> While submitting patches is good, there's a lot more to it than just > >> submitting the 5-line code change to submit a bug/feature, and reviewing > >> takes a lot of time and effort. > > [Phil] > > > So there is something wrong there as well. > > I have not idea what you're getting at. Martin's comment seems > accurate to me. Unless it is a simple typo/doc fix, it takes > some time to assess whether the bug is real (some things are bugs > only in the eye of the submitter) and whether the given fix is the > right thing to do. > > Of course, automatic acceptance of patches would be a crummy idea. > There have been no shortage of patches complete with docs and tests > that were simply not the right thing to do. My point is simply that the effort required to review patches seems to be a problem. Perhaps the reasons for that need to be looked at and the process changed so that it is more effective. At the moment people just seem be saying "that's the way it is because that's the way it's got to be". > [Phil] > > > The process needs > > to keep people involved in it - at the moment submitting a patch is > > fire-and-forget. > > Such is the nature of a system of volunteers. If we had full-time people, > it could be a different story. IMO, given an 18 month release cycle, > it is perfectly acceptable for a patch to sit for a while until someone > with the relavant expertise can address it. Even with a tests and docs, > patch acceptance is far from automatic. That being said, I think history > has shown that important bugs get addressed and put into bug fix releases > without much loss of time. When Py2.5.1 goes out, I expect that all known, > important bugs will have been addressed and that's not bad. Then perhaps getting a full-time person should be taken seriously. Phil From martin at v.loewis.de Tue Mar 6 10:24:15 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 10:24:15 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <87irdeamyh.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <87irdeamyh.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <45ED333F.5080502@v.loewis.de> Stephen J. Turnbull schrieb: > An informal version of this process is how XEmacs identifies its > Reviewers (the title we give to those privileged to authorize commits > to all parts of XEmacs). People who care enough to make technical > comments on *others'* patches are rare, and we grab the competent ones > pretty quickly. My experience exactly. Besides Georg Brandl, I think this was also how Raymond Hettinger started his regular contributions to Python. > The mess is not "total", as Josiah Carlson points out. To the extent > that it is a mess, it is the consequence of a process similar to the > one you propose to institutionalize. It wasn't clear to me that this is the case, but I think you are exactly right. Those libraries that are now considered a mess had been ad-hoc contributions in many cases, with a single use case (namely the one of the original contributor). The contributor is not to blame, of course: he could only contribute what he has experience with. I wouldn't blame the committers who accepted the libraries, either - much of Python's value is in "libraries included". So to fix "the total mess", one has to replace the libraries with better ones, were available, learning from past experience to not accept libraries that had not been widely tested "in the field". > Second, where the stdlib module is closely bound to the core, the > maintainer ends up being the group of core developers. It can't be > any other way, it seems to me. It might be that individuals get designated maintainers: Guido maintains list and tuple, Tim maintaines dict, Raymond maintains set, I maintain configure.in. However, I doubt that would work in practice. Regards, Martin From thomas at python.org Tue Mar 6 10:26:50 2007 From: thomas at python.org (Thomas Wouters) Date: Tue, 6 Mar 2007 10:26:50 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> Message-ID: <9e804ac0703060126n18a8746lf84c0df80b72f995@mail.gmail.com> On 3/6/07, Neil Schemenauer wrote: > Using git-svn to track a SVN repository seems to work well. I'm not interested in setting up GIT myself, mostly because it doesn't solve any issues that other dvcs' don't solve, the on-disk repository is mighty big and it doesn't work very well on non-Linux systems (at least, not last I looked.) If you want to set it up and maintain it, though, that's fine by me. I would like to point out that while it takes only a few minutes to setup a new repository and start the conversion for any of the SCM's (not just distributed ones), that doesn't mean it's a no-brainer to set them up 'officially', and maintain them :) There's a lot more work in there, so be prepared to spend some time to get it to work right and reliable for everyone. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070306/8c318b89/attachment.htm From thomas at python.org Tue Mar 6 10:28:08 2007 From: thomas at python.org (Thomas Wouters) Date: Tue, 6 Mar 2007 10:28:08 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> References: <20070305184614.GA19051@localhost.localdomain> <9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> Message-ID: <9e804ac0703060128j2331bb7bv3a5431aa4396d04d@mail.gmail.com> On 3/6/07, Raymond Hettinger wrote: > [MvL] > >> While submitting patches is good, there's a lot more to it than just > >> submitting the 5-line code change to submit a bug/feature, and > reviewing > >> takes a lot of time and effort. That was incorrectly attributed; it was me, not Martin. (But thanks for agreeing :) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070306/23a67f13/attachment-0001.html From martin at v.loewis.de Tue Mar 6 10:31:50 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 10:31:50 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <200703060818.29114.phil@riverbankcomputing.co.uk> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <200703060818.29114.phil@riverbankcomputing.co.uk> Message-ID: <45ED3506.9040707@v.loewis.de> Phil Thompson schrieb: >>> 2. Publically identify the core developers and their areas of expertise >>> and responsibility (ie. which parts of the source tree they "own"). >> I doubt this will help. Much of the code isn't owned by anybody >> specifically. Those parts that are owned typically find their patches >> reviewed and committed quickly (e.g. the tar file module, maintained by >> Lars Gust?bel). > > Doesn't your last sentence completely contradict your first sentence? No (not sure how you are counting: there are three sentences): 1. Public identification will not help, because: 2. most code isn't in the responsibility of anybody (so publically identifying responsibilities would leave most code unassigned), and 3. for the code that has some responsible member, things are already fine (so public identification won't improve here) Maybe you meant to suggest "assign responsibilities to core developers, then identify them publically"; this is different from merely publically announcing already-assigned specific responsibilities. The latter won't work for the reasons discussed; the former won't work because these are volunteers, you can't assign anything to them. >>> 4. Acceptance by core developers that only half the "job" is developing >>> the core - the other half is mentoring potential future core developers. >> So what do you do with core developers that don't do their job? Fire them? > > Of course not, but this is a cultural issue not a technical one. The first > step in changing a culture is to change the expectations. I think the expectations of the users of Python have to adjust, then. This is free software, it has its own working principles that people need to get used to. In essence: if you want change, you need to execute it your own. Nobody will do it for you. Regards, Martin From martin at v.loewis.de Tue Mar 6 10:39:06 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 10:39:06 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <200703060834.06715.phil@riverbankcomputing.co.uk> References: <20070305184614.GA19051@localhost.localdomain> <200703052116.48035.phil@riverbankcomputing.co.uk> <45ED00FC.6050406@v.loewis.de> <200703060834.06715.phil@riverbankcomputing.co.uk> Message-ID: <45ED36BA.1080408@v.loewis.de> Phil Thompson schrieb: >> And please be assured that no such obstacle is in the submitters way. >> Most patches are accepted without the submitter actually reviewing any >> other patches. > > I'm glad to hear it - but I'm talking about the perception, not the fact. When > occasionally submitters ask if their patch is going to be included, they will > usually get a response suggesting they review other patches. That will only > strengthen the perception. That's because those who get their patches accepted quickly never complain. They are still the majority, though. In case of Titus Brown (who complained in his blog), I found that all of his 5 patches got integrated into Python, me committing four of them, and Georg committing one. Response time was between 3 days and 16 months. > This discussion started because the feeling was expressed that it was > difficult to get patches accepted and that new core developers were not being > found. I would love to contribute more to the development of Python - I owe > it a lot - but from where I stand (which is most definitely not where you > stand) I can't see how to do that in a productive and rewarding way. Not sure how long you have been contributing to free software: you will find, over time, that it is rewarding to get changes accepted even if the process takes several months. Patience is an important quality here. Regards, Martin From martin at v.loewis.de Tue Mar 6 10:52:48 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 10:52:48 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <200703060848.04117.phil@riverbankcomputing.co.uk> References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <45ED0371.5070108@v.loewis.de> <200703060848.04117.phil@riverbankcomputing.co.uk> Message-ID: <45ED39F0.2000907@v.loewis.de> Phil Thompson schrieb: >>> Of course it's not unreasonable. I would expect to be told that a patch >>> must have tests and docs before it will be finally accepted. However, >>> before I add those things to the patch I would like some timely feedback >>> from those with more experience that my patch is going in the right >>> direction. >> This cannot work. It is very difficult to review a patch to fix a >> presumed bug if there is no test case. You might not be able to >> reproduce the patch without a test case at all - how could you then >> know whether the patch actually fixes the bug? > > Please read what I said again. I did - I can't see where I misunderstood. You said you want feedback on the patch even if it doesn't have test and doc changes ("before I add those things"), and I said the only feedback you'll likely get is "add test cases and doc changes". > Yes, a patch must be reviewed before > submission. Yes, a patch when submitted must include docs and test cases. I'm > talking about the less formal process leading up to that point. The less > formal process has a much lower barrier to entry, requires much less effort > by the "reviewer", is the period during which the majority of the teaching > happens, and will result in a better quality final patch that will require > less effort to be put in to the final, formal review. Here I'm not sure what you are talking about. How would that process be executed? On python-dev? On the patches tracker? It often happens that people write a bug report, and I respond "yes, this is a bug, would you like to work on a patch?" They then either ask "should I do it this or that way?", and then I give my opinion. So this already happens - again, it's just that the people don't talk much about it. I can't see that the barrier at contributing is high. The number of patch submissions and bug reports proves the contrary. Literally hundreds of people contribute. >> If you really want to learn and help, review a few patches, to see >> what kinds of problems you detect, and then post your findings to >> python-dev. People then will comment on whether they agree with your >> review, and what additional changes they like to see. > > Do you think this actually happens in practice? I wasn't talking in general, I was talking specifically about you (Phil Thompson) here. If you really want to contribute in *maintaining* Python, you are more than welcome. Several current committers found their way into python-dev in the way you described, and nearly nobody was ever turned away. Regards, Martin From martin at v.loewis.de Tue Mar 6 10:57:58 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 10:57:58 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <200703060906.22762.phil@riverbankcomputing.co.uk> References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> <200703060906.22762.phil@riverbankcomputing.co.uk> Message-ID: <45ED3B26.5090605@v.loewis.de> Phil Thompson schrieb: > My point is simply that the effort required to review patches seems to be a > problem. Perhaps the reasons for that need to be looked at and the process > changed so that it is more effective. At the moment people just seem be > saying "that's the way it is because that's the way it's got to be". We have already changed the process a lot. These days patches are regularly turned away for not having tests and doc changes in them. Yet, there are no reviewers willing to contribute even this very straight-forward, easy-to-execute check. If the patch meets the formal criteria, the hard part (on the reviewers side) starts: they must understand the code being patched, the nature of the problem, and the patch itself. I don't see how this could be simplified, without neglecting quality. > Then perhaps getting a full-time person should be taken seriously. That's quite expensive, plus somebody would need to supervise that person. A part-time person would be less expensive, but still needs supervision. Regards, Martin From stephen at xemacs.org Tue Mar 6 11:12:46 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 06 Mar 2007 19:12:46 +0900 Subject: [Python-Dev] Encouraging developers In-Reply-To: <200703060818.29114.phil@riverbankcomputing.co.uk> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <200703060818.29114.phil@riverbankcomputing.co.uk> Message-ID: <87abyqadep.fsf@uwakimon.sk.tsukuba.ac.jp> Phil Thompson writes: > MvL wrote: > > I doubt this will help. Much of the code isn't owned by anybody > > specifically. Those parts that are owned typically find their patches > > reviewed and committed quickly (e.g. the tar file module, maintained by > > Lars Gust?bel). > Doesn't your last sentence completely contradict your first sentence? Not in view of the second one. Martin is saying that where the existing process looks like your suggestion, it already works great. It's not that this isn't well-known to the core developers! The problem is a lack of reviewers/module "owners". The existing review-for-review *offer* (not requirement!) proposes to increase the supply of reviewers by offering them "good and valuable consideration" (ie, a fast track for their own patches) in return. You may not wish to take advantage of that offer, but it addresses the underlying problem. The other proposals on the table amount to (a) the existing reviewers should work harder and (b) if patches aren't reviewed, then they should be presumed good enough to apply. I think it should be obvious how far they will get when restated in those practical terms. From nmm1 at cus.cam.ac.uk Tue Mar 6 11:19:04 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Tue, 06 Mar 2007 10:19:04 +0000 Subject: [Python-Dev] Access to bits for a PyLongObject Message-ID: =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= wrote: > Eric V. Smith schrieb: > > I'm working on PEP 3101, Advanced String Formatting. About the only > > built-in numeric formatting I have left to do is for converting a > > PyLongOjbect to binary. > > > > I need to know how to access the bits in a PyLong. > > I think it would be a major flaw in PEP 3101 if you really needed it. > The long int representation should be absolutely opaque - even the > fact that it is a sign+magnitude representation should be hidden. Well, it depends on the level for which PEP 3101 is intended. I had the same problem, and was pointed at _PyLong_AsByteArray, which was all I needed. In my case, going though a semi-generic formatter would not have been an acceptable interface (performance). However, if PEP 3101 is intended to be a higher level of formatting, then I agree with you. So I have nailed my colours firmly to the fence :-) Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From turnbull at sk.tsukuba.ac.jp Tue Mar 6 11:51:41 2007 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 06 Mar 2007 19:51:41 +0900 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45ED333F.5080502@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <87irdeamyh.fsf@uwakimon.sk.tsukuba.ac.jp> <45ED333F.5080502@v.loewis.de> Message-ID: <87649efxvm.fsf@uwakimon.sk.tsukuba.ac.jp> "Martin v. L?wis" writes: > Stephen J. Turnbull schrieb: > > Second, where the stdlib module is closely bound to the core, the > > maintainer ends up being the group of core developers. It can't be > > any other way, it seems to me. > > It might be that individuals get designated maintainers: Guido maintains > list and tuple, Tim maintaines dict, Raymond maintains set, I maintain > configure.in. However, I doubt that would work in practice. I was referring more to modules like "os" than to language features. My experience with XEmacs is that 3rd parties are generally surprised at how wide the range of packages that are considered to require the blessing of a core developer before messing with them. From p.f.moore at gmail.com Tue Mar 6 12:20:22 2007 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Mar 2007 11:20:22 +0000 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45ED07EE.4060402@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <45ED0353.2090201@scottdial.com> <45ED07EE.4060402@v.loewis.de> Message-ID: <79990c6b0703060320t23d4487rcc97ec69c669a8ee@mail.gmail.com> On 06/03/07, "Martin v. L?wis" wrote: > Scott Dial schrieb: > > While I understand that this tit-for-tat mechanism is meant to ensure > > participation, I believe in reality it doesn't, as the 400-some > > outstanding patches you referenced elswhere indicate. I can personally > > attest to having a patch that is over a year old with no "core > > developer" having any interest at all with the subject matter. And to be > > frank, nor did I really, but I saw a problem and was capable of solving > > it. My lack of caring about the patch means I am not going to beat > > people over the head to pay attention. This system is broken for someone > > like me (coder) that just wants to help out (non-coders). > > If you don't care that much about the patch, it's not broken. As I said > before, the number of unreviewed patches has been roughly stable for > some time now. If the patch is not really important, it may take two > years now to get it in, but eventually, it will (if you then still are > interested to work on it to complete it). Here's a random offer - let me know the patch number for your patch, and I'll review it. Note that I do *not* consider myself a core developer, I don't even have the tools these days to build Python easily - I certainly haven't done so for a while. The likelihood is that I don't know much about the subject area of your patch, either. As a final disclaimer, note that I have no commit privilege, so my review won't result in your patch actually being applied :-) I'll post the results of my review here, as an example of what a reviewer needs to look at. If someone wants to distil that into a set of "how to review a patch" guidelines, then that's great. More reviewers would be a huge benefit. I agree that "contributing" feels hard, and often the hard bit is gaining the attention of the committers. The 5-for-1 offers help this, and shouldn't be dismissed - it's just that the *other* ways involve people skills (and so are far harder!!!) Maybe we should emphasize (again) that reviewing patches is also contributing, and would be greatly appreciated. Paul. From martin at v.loewis.de Tue Mar 6 13:00:10 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 13:00:10 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <79990c6b0703060320t23d4487rcc97ec69c669a8ee@mail.gmail.com> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <45ED0353.2090201@scottdial.com> <45ED07EE.4060402@v.loewis.de> <79990c6b0703060320t23d4487rcc97ec69c669a8ee@mail.gmail.com> Message-ID: <45ED57CA.7090101@v.loewis.de> Paul Moore schrieb: >> Scott Dial schrieb: >>> While I understand that this tit-for-tat mechanism is meant to ensure >>> participation, I believe in reality it doesn't, as the 400-some >>> outstanding patches you referenced elswhere indicate. I can personally >>> attest to having a patch that is over a year old with no "core >>> developer" having any interest at all with the subject matter. > > Here's a random offer - let me know the patch number for your patch, > and I'll review it. Surprisingly (and I hope Scott can clarify that), I can't find anything. Assuming Scott's SF account is "geekmug", I don't see any open patches (1574068 was closed within 20 days by amk, last October). Regards, Martin From martin at v.loewis.de Tue Mar 6 13:36:03 2007 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 13:36:03 +0100 Subject: [Python-Dev] splitext('.cshrc') Message-ID: <45ED6033.2020807@v.loewis.de> #1115886 complains that in the file name '.cshrc', the entire file name is treated as an extension, with no root. #1462106 contains a patch for that, changing the behavior so that there will always be a root file name (and no extension if the file is just a dotfile). Should this be changed? Opinions? Regards, Martin From johann at rocholl.net Tue Mar 6 13:39:53 2007 From: johann at rocholl.net (Johann C. Rocholl) Date: Tue, 6 Mar 2007 12:39:53 +0000 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45ED6033.2020807@v.loewis.de> References: <45ED6033.2020807@v.loewis.de> Message-ID: <8233478f0703060439m26d0965axfe71e4b94dc24f69@mail.gmail.com> On 3/6/07, "Martin v. L?wis" wrote: > #1115886 complains that in the file name '.cshrc', the > entire file name is treated as an extension, with no > root. > > #1462106 contains a patch for that, changing the behavior > so that there will always be a root file name (and no > extension if the file is just a dotfile). > > Should this be changed? Opinions? +1 From phd at phd.pp.ru Tue Mar 6 13:53:56 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Tue, 6 Mar 2007 15:53:56 +0300 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45ED6033.2020807@v.loewis.de> References: <45ED6033.2020807@v.loewis.de> Message-ID: <20070306125356.GB22844@phd.pp.ru> On Tue, Mar 06, 2007 at 01:36:03PM +0100, "Martin v. L?wis" wrote: > #1115886 complains that in the file name '.cshrc', the > entire file name is treated as an extension, with no > root. > > #1462106 contains a patch for that, changing the behavior > so that there will always be a root file name (and no > extension if the file is just a dotfile). > > Should this be changed? Opinions? Yes. In .pythonrc.py .pythonrc is the root, and .py is the extension. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From jeremy at alum.mit.edu Tue Mar 6 13:54:10 2007 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Tue, 6 Mar 2007 07:54:10 -0500 Subject: [Python-Dev] locals(), closures, and IronPython... In-Reply-To: <7AD436E4270DD54A94238001769C22276A624C7574@DF-GRTDANE-MSG.exchange.corp.microsoft.com> References: <7AD436E4270DD54A94238001769C22276A624C7482@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <7AD436E4270DD54A94238001769C22276A624C7574@DF-GRTDANE-MSG.exchange.corp.microsoft.com> Message-ID: On 3/5/07, Dino Viehland wrote: > Thanks Guido. It might take some time (and someone may very well beat me to it if they care a lot) but I'll see if we can get the PEP started. Dino, One of the questions I was puzzling over was what locals() should return in a class scope. It's confusing in CPython, because the same dictionary is used to stored names that are destined to become class attributes and names that are used to created closures for methods with free variables. The implementation was causing the latter names to show up as class attributes when you called locals(). Terry is right that lifetime of variables is a key issue. It would probably be bad if all the local variables of a function were kept alive because a nested function used locals(). One idea was discussed at PyCon was having a different representation for an environment so that it would be easier to introspect on it but still possible to pass the environment to exec. One possibility is just a third dictionary--globals, locals, nonlocals. Another possibility is an object where you can ask about the scope of each variable but also extract a dictionary to pass to locals. Jeremy > > -----Original Message----- > From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido van Rossum > Sent: Monday, March 05, 2007 2:14 PM > To: Dino Viehland > Cc: python-dev at python.org > Subject: Re: [Python-Dev] locals(), closures, and IronPython... > > Jeremy Hylton has been asking questions about this too at the sprint > after PyCon. I'm tempted to accept that the exact behavior of locals() > is implementation-defined (IOW undefined :-) as long as it includes > the locals defined in the current scope; whether it also includes free > variables could be debatable. I don't know too many good use cases for > locals() apart from "learning about the implementation" I think this > might be okay. Though a PEP might be in order to get agreement between > users, developers and other implementation efforts (e.g. PyPy, > Jython). > > On 3/5/07, Dino Viehland wrote: > > > > > > > > > > def a(): > > > > x = 4 > > > > y = 2 > > > > def b(): > > > > print y, locals() > > > > print locals() > > > > b() > > > > > > > > a() > > > > > > > > in CPython prints: > > > > > > > > {'y': 2, 'x': 4, 'b': } > > > > 2 {'y': 2} > > > > > > > > I'm wondering if it's intentional that these don't print dictionaries w/ the > > same contents or if it's more an accident of the implementation. In other > > words would it be reasonable for IronPython to promote all of the locals of > > a into b's dictionary when both a and b call locals? > > > > > > > > We currently match CPython's behavior here - at least in what we display > > although possibly not in the lifetimes of objects. We're considering an > > internal change which might alter the behavior here though and end up > > displaying all the members. > > > > > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu > From g.brandl at gmx.net Tue Mar 6 13:40:04 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 06 Mar 2007 13:40:04 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45ED6033.2020807@v.loewis.de> References: <45ED6033.2020807@v.loewis.de> Message-ID: Martin v. L?wis schrieb: > #1115886 complains that in the file name '.cshrc', the > entire file name is treated as an extension, with no > root. > > #1462106 contains a patch for that, changing the behavior > so that there will always be a root file name (and no > extension if the file is just a dotfile). > > Should this be changed? Opinions? Since dotfiles can have extensions as well (e.g. ~/.fonts.conf), I'd say that the current behavior is wrong. Question is, is it worth possibly breaking compatibility... Georg From g.brandl at gmx.net Tue Mar 6 14:09:54 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 06 Mar 2007 14:09:54 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> References: <20070305184614.GA19051@localhost.localdomain><9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> Message-ID: Raymond Hettinger schrieb: > [Phil Thompson] >> I think a lot of people care, but many can't >> do anything about because the barrier to entry is too great. > > Do you mean commit priviledges? ISTM, those tend to be > handed out readily to people who assert that they have good use for them. > Ask the Georg-bot how readily he was accepted and coached. IMO, > his acceptance was a model that all open source projects should aspire to. Indeed. For me, it wasn't "hard" to get tracker rights. I reviewed some patches, commented on bugs, posted suggestions to python-dev etc. When I asked about tracker rights on python-dev, they were given to me. Then, it wasn't "hard" to get commit rights. I contributed some stuff, and after a while I asked about commit rights on python-dev, and they were given to me on condition that I still let a core dev review inteded changes. As far as I recall, there has been nearly no one who asked for commit rights recently, so why complain that the entry barrier is too great? Surely you cannot expect python-dev to got out and say "would you like to have commit privileges?"... Georg From jeremy at alum.mit.edu Tue Mar 6 14:42:00 2007 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Tue, 6 Mar 2007 08:42:00 -0500 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> <9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> Message-ID: On 3/6/07, Georg Brandl wrote: > Raymond Hettinger schrieb: > > [Phil Thompson] > >> I think a lot of people care, but many can't > >> do anything about because the barrier to entry is too great. > > > > Do you mean commit priviledges? ISTM, those tend to be > > handed out readily to people who assert that they have good use for them. > > Ask the Georg-bot how readily he was accepted and coached. IMO, > > his acceptance was a model that all open source projects should aspire to. > > Indeed. For me, it wasn't "hard" to get tracker rights. I reviewed some patches, > commented on bugs, posted suggestions to python-dev etc. When I asked about > tracker rights on python-dev, they were given to me. > Then, it wasn't "hard" to get commit rights. I contributed some stuff, and > after a while I asked about commit rights on python-dev, and they were given > to me on condition that I still let a core dev review inteded changes. > > As far as I recall, there has been nearly no one who asked for commit rights > recently, so why complain that the entry barrier is too great? Surely you > cannot expect python-dev to got out and say "would you like to have commit > privileges?"... You can ask whether we should have a plan for increasing the number of developers, actively seeking out new developers, and mentoring people who express interest. Would the code be better if we had more good developers working on it? Would we get more bugs fixed and patches closed? If so, it wouldn't hurt to have some deliberate strategy for bringing new developers in. I can easily imagine someone spending a lot of time mentoring and a little time coding, but having a bigger impact that someone who only wrote code. Jeremy From meine at informatik.uni-hamburg.de Tue Mar 6 14:44:52 2007 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Tue, 6 Mar 2007 14:44:52 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45ED6033.2020807@v.loewis.de> References: <45ED6033.2020807@v.loewis.de> Message-ID: <200703061444.52832.meine@informatik.uni-hamburg.de> Am Dienstag, 06. M?rz 2007 13:36 schrieb Martin v. L?wis: > #1115886 complains that in the file name '.cshrc', the > entire file name is treated as an extension, with no > root. > > #1462106 contains a patch for that, changing the behavior > so that there will always be a root file name (and no > extension if the file is just a dotfile). > > Should this be changed? Opinions? +1 The current behavior is clearly a bug, since a leading dot does not start an extension, but marks a file as "hidden". The latter is on UNIX, and while this is different on Windows, I cannot imagine that anyone would a) have dotfiles under that OS, or even then b) rely on the current behavior that their full filename counts as "extension". Greetings, Hans From eric+python-dev at trueblade.com Tue Mar 6 14:45:53 2007 From: eric+python-dev at trueblade.com (Eric V. Smith) Date: Tue, 06 Mar 2007 08:45:53 -0500 Subject: [Python-Dev] Access to bits for a PyLongObject In-Reply-To: <45ECF545.2080806@v.loewis.de> References: <45ECCA92.8090709@trueblade.com> <45ECF545.2080806@v.loewis.de> Message-ID: <45ED7091.3010705@trueblade.com> Martin v. L?wis wrote: > Eric V. Smith schrieb: >> I'm working on PEP 3101, Advanced String Formatting. About the only >> built-in numeric formatting I have left to do is for converting a >> PyLongOjbect to binary. >> >> I need to know how to access the bits in a PyLong. > > I think it would be a major flaw in PEP 3101 if you really needed it. > The long int representation should be absolutely opaque - even the > fact that it is a sign+magnitude representation should be hidden. > > Looking at the PEP, I see that a class can implement __format__. > Wouldn't it be appropriate if the long type implemented that? > Implementation-wise, I would expect that long_format already does the > bulk of what you need to do. Yes, I think that would be appropriate. However, it conflicts with the current implementation strategy, which is to make a stand-alone module until we can flush out all of the issues. Not that our current implementation should drive the correct decision, of course. Also, it would either mean duplicating lots of code from the int formatter, or having a formatter library that both can call. This is because __format__ must implement all formats, including padding, parenthesis for negatives, etc., not just the "missing" binary format. Not that that's necessarily bad, either. But see the next point. > OTOH, also look at _PyString_FormatLong. I think a solution would be to add 'b' to _PyString_FormatLong, which I'm already calling for hex, octal, and decimal formatting. Does that sound reasonable? It seems to me that if binary is useful enough for PEP 3101, it should generally be available in _PyString_FormatLong. The obvious implementation of this would require adding a nb_binary to PyNumberMethods. I'm not sure what the impact of that change would be, but it sounds really big and probably a show-stopper. Maybe a direct call to a binary formatter would be better. OTOH, this approach isn't as efficient as I'd like (for all formatting outputs, not just binary), because it has to build a string object and then copy data out of it. Having written all of this, I'm now thinking that Nick's suggestion of _PyLong_AsByteArray might be the way to go. I would use that for all of my formatting for longs. I think I can use my output buffer as the buffer for _PyLong_AsByteArray, since all formats (binary, decimal, octal, hex) are less "bit dense" than the byte array. As long as I read, format, and write the data in the correct order, I'd be okay. So even though I'd copy the data into my buffer, at least I wouldn't be allocating more memory or another object just to extract data from the long. Maybe I'm over-emphasizing performance, given the early status of the implementation. But I'd like PEP 3101 to be as efficient as possible, because once it's available I'll replace all of the '%' string formatting in my code with it. I think others will consider that as well. Thank you for your insights. I apologize for the length of this message, but as I believe Pascal said, I did not have time to make it shorter. Eric. From phil at riverbankcomputing.co.uk Tue Mar 6 14:59:39 2007 From: phil at riverbankcomputing.co.uk (Phil Thompson) Date: Tue, 6 Mar 2007 13:59:39 +0000 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> Message-ID: <200703061359.39528.phil@riverbankcomputing.co.uk> On Tuesday 06 March 2007 1:42 pm, Jeremy Hylton wrote: > On 3/6/07, Georg Brandl wrote: > > Raymond Hettinger schrieb: > > > [Phil Thompson] > > > > > >> I think a lot of people care, but many can't > > >> do anything about because the barrier to entry is too great. > > > > > > Do you mean commit priviledges? ISTM, those tend to be > > > handed out readily to people who assert that they have good use for > > > them. Ask the Georg-bot how readily he was accepted and coached. IMO, > > > his acceptance was a model that all open source projects should aspire > > > to. > > > > Indeed. For me, it wasn't "hard" to get tracker rights. I reviewed some > > patches, commented on bugs, posted suggestions to python-dev etc. When I > > asked about tracker rights on python-dev, they were given to me. > > Then, it wasn't "hard" to get commit rights. I contributed some stuff, > > and after a while I asked about commit rights on python-dev, and they > > were given to me on condition that I still let a core dev review inteded > > changes. > > > > As far as I recall, there has been nearly no one who asked for commit > > rights recently, so why complain that the entry barrier is too great? > > Surely you cannot expect python-dev to got out and say "would you like to > > have commit privileges?"... > > You can ask whether we should have a plan for increasing the number of > developers, actively seeking out new developers, and mentoring people > who express interest. Would the code be better if we had more good > developers working on it? Would we get more bugs fixed and patches > closed? If so, it wouldn't hurt to have some deliberate strategy for > bringing new developers in. I can easily imagine someone spending a > lot of time mentoring and a little time coding, but having a bigger > impact that someone who only wrote code. Thank you - that's exactly what I'm trying to say. Phil From phd at phd.pp.ru Tue Mar 6 15:00:16 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Tue, 6 Mar 2007 17:00:16 +0300 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <200703061444.52832.meine@informatik.uni-hamburg.de> References: <45ED6033.2020807@v.loewis.de> <200703061444.52832.meine@informatik.uni-hamburg.de> Message-ID: <20070306140016.GA3839@phd.pp.ru> On Tue, Mar 06, 2007 at 02:44:52PM +0100, Hans Meine wrote: > a leading dot does not start an > extension, but marks a file as "hidden". The latter is on UNIX, and while On Unix - I mean in the OS itself - there are no such things as "roots", "extensions" and "hidden files". All these are only conventions in the user programs. The current behaviour is not a bug in a strict sense, but it is inconvenient and hence should be changed. > this is different on Windows, I cannot imagine that anyone would > a) have dotfiles under that OS I have. (-: I often copy files from an ext3 partition to a FAT partition on my dual-booting desktop. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From fuzzyman at voidspace.org.uk Tue Mar 6 15:03:12 2007 From: fuzzyman at voidspace.org.uk (Fuzzyman) Date: Tue, 06 Mar 2007 14:03:12 +0000 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <20070306140016.GA3839@phd.pp.ru> References: <45ED6033.2020807@v.loewis.de> <200703061444.52832.meine@informatik.uni-hamburg.de> <20070306140016.GA3839@phd.pp.ru> Message-ID: <45ED74A0.2060701@voidspace.org.uk> Oleg Broytmann wrote: >[snip..] > >>this is different on Windows, I cannot imagine that anyone would >>a) have dotfiles under that OS >> >> > > > It is very common for cross platform programs to create configuration files which are dotfiles, whichever OS they are running on. Michael Foord From nas at arctrix.com Tue Mar 6 15:27:20 2007 From: nas at arctrix.com (Neil Schemenauer) Date: Tue, 6 Mar 2007 08:27:20 -0600 Subject: [Python-Dev] Encouraging developers In-Reply-To: <9e804ac0703060126n18a8746lf84c0df80b72f995@mail.gmail.com> References: <20070305184614.GA19051@localhost.localdomain> <9e804ac0703060126n18a8746lf84c0df80b72f995@mail.gmail.com> Message-ID: <20070306142720.GA31643@python.ca> On Tue, Mar 06, 2007 at 10:26:50AM +0100, Thomas Wouters wrote: > I'm not interested in setting up GIT myself, mostly because it > doesn't solve any issues that other dvcs' don't solve dvcs wars are the new editor wars. :-) > the on-disk repository is mighty big and it doesn't work very well > on non-Linux systems (at least, not last I looked.) Not true. The on-disk repository is now one of the more efficient ones. There is also now a MinGW port (i.e. native Windows). Neil From mlobol at gmail.com Tue Mar 6 15:28:32 2007 From: mlobol at gmail.com (Miguel Lobo) Date: Tue, 6 Mar 2007 15:28:32 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules Message-ID: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> Hi list, A month and a half ago, I submitted patch 1644818 to the CPython Patch Tracker: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1644818&group_id=5470 On several occassions I have been advised to mention the patch in this list, so here it is: The problem: Importing built-in submodule extensions (e.g. PyQt4.QtCore and PyQt4.QtGui in my case) does not work. By built-in I mean statically linked against the Python executable. The cause: find_module in import.c only checks the module name (e.g. QtCore) against the built-in list, which should contain the full name (e.g. PyQt4.QtCore) instead. Also, the above check is performed after the code to check if the parent module is frozen, which would have already exited in that case. Solution: By moving the "is_builtin()" check to earlier in find_module and using the "fullname" variable instead of "name", I can build PyQt4.QtCoreand PyQt4.QtGui into the interpreter and import and use them with no problem whatsoever, even if their parent module (PyQt4) is frozen. When I submitted the patch, I ran the regression tests and everything seemed Ok. Also, the patch is for Python-2.5, but it a month and a half ago it was applicable to the svn trunk with only a one-line offset. As I am completely new to CPython development, perhaps this problem has already been discussed and/or fixed I may have done something incorrectly. Please let me know if that is the case. Regards, Miguel Lobo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070306/ae613224/attachment.htm From ncoghlan at gmail.com Tue Mar 6 15:41:23 2007 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 07 Mar 2007 00:41:23 +1000 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain><9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> Message-ID: <45ED7D93.4010601@gmail.com> Georg Brandl wrote: > As far as I recall, there has been nearly no one who asked for commit rights > recently, so why complain that the entry barrier is too great? Surely you > cannot expect python-dev to got out and say "would you like to have commit > privileges?"... I think the number one suggestion I can make to anyone who is genuinely interested in contributing to the Python core is to lurk on python-dev for a while. It *does* require a reasonable time commitment (much more than the time required to fire a one-off patch at the SF tracker), but I've certainly found it to be a valuable learning experience (both in general and in relation to Python core development): - getting an idea of who's who (and what's what) in the Python world - getting an idea of what needs to be done in CPython development - seeing bugs and patches discussed (yes, it happens!) - learning various non-Python specific things ranging from good API design and doing object-oriented programming in C to the intricacies of binary floating point, Unicode and POSIX signal handling (etc, etc, ...) I believe my personal involvement in core development followed a similar trajectory to Georg's - lurked on python-dev for a while, began participating in discussions before too long (I'm not very good at lurking ;), helped out with the initial addition of the decimal module, got tracker privileges to help out with various bugs and patches, then eventually got commit privileges in order to update PEP 343. And I think this approach still works - it's just that it is mainly useful to people that are interested in Python core development in general, rather than those that are looking to get a specific bug fixed or patch accepted (although the latter can happen too - Lars was given commit privileges when it became clear that he was both willing and able to be the maintainer of the tarfile module). One thing that did happen though (which the messages from Jeremy & Phil reminded me of) is that I got a lot of direction, advice and assistance from Raymond when I was doing that initial work on improving the speed of the decimal module - I had the time available to run the relevant benchmarks repeatedly and try different things out, while Raymond had the experience needed to suggest possible avenues for optimisation (and when to abandon an approach as making the code too complicated to be effectively maintained). I don't know whether or not there is anything specific we can do to encourage that kind of coaching/mentoring activity, but I know it was a significant factor in my become more comfortable with making contributions. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From amk at amk.ca Tue Mar 6 15:41:30 2007 From: amk at amk.ca (A.M. Kuchling) Date: Tue, 6 Mar 2007 09:41:30 -0500 Subject: [Python-Dev] Encouraging developers In-Reply-To: <200703060906.22762.phil@riverbankcomputing.co.uk> References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> <200703060906.22762.phil@riverbankcomputing.co.uk> Message-ID: <20070306144130.GA5613@localhost.localdomain> On Tue, Mar 06, 2007 at 09:06:22AM +0000, Phil Thompson wrote: > My point is simply that the effort required to review patches seems to be a > problem. Perhaps the reasons for that need to be looked at and the process > changed so that it is more effective. At the moment people just seem be > saying "that's the way it is because that's the way it's got to be". Unfortunately I think the effort required is intrinsic to reviewing patches. Trivial or obviously correct patches get applied quickly, so the remaining bugs and patches are the ones that are hard to fix. For example, our oldest bug is http://www.python.org/sf/214033, opened 2000-09-11, and is that (x?)? is reported as an error by the SRE regex parser; the PCRE engine and Perl both accept it. Fixing it requires changing sre_parse, figuring out what to do (should it be equivalent to '(x?)' or to '(x)?', and then being very sure that no other patterns are broken by the change. I suspect that fixing this bug properly by researching the right answer, implementing it, and adding tests would take me a half-day's worth of work. If modifying sre_parse is very difficult, it could take longer. --amk From amk at amk.ca Tue Mar 6 15:53:35 2007 From: amk at amk.ca (A.M. Kuchling) Date: Tue, 6 Mar 2007 09:53:35 -0500 Subject: [Python-Dev] SVK (was: Encouraging developers) In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> <17900.26693.804101.291817@montanaro.dyndns.org> Message-ID: <20070306145335.GB5613@localhost.localdomain> On Tue, Mar 06, 2007 at 07:59:53AM +0100, Ronald Oussoren wrote: > development easier for them. They can already do this using SVK, > which is a distributed version control system as well but uses SVN > repositories to store its data. I'm happy to write up a wiki page describing how to use SVK to set up your own mirror of the Python SVN. However, trying it now, the initial setup seems very slow: SVK is retrieving each of 54165 revisions and it'll probably take over an hour to complete. I'm retrying it now by mirroring only the trunk to see if that's any faster, but this seems like an impediment -- if you want to develop a bugfix of sufficient complexity to need a DVCS, and the first step required takes over an hour, that may dissipate your forward motion. (Of course, I don't know how long a checkout of a hypothetical Bazaar repository would take; maybe it's not any faster.) --amk From martin at v.loewis.de Tue Mar 6 16:00:01 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 16:00:01 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <20070306125356.GB22844@phd.pp.ru> References: <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> Message-ID: <45ED81F1.502@v.loewis.de> Oleg Broytmann schrieb: >> Should this be changed? Opinions? > > Yes. In .pythonrc.py .pythonrc is the root, and .py is the extension. Ah, it would do that already: with multiple dots, the last one always provides the extension. However, for .pythonrc, it would conclude that .pythonrc is the extension. Regards, Martin From phd at phd.pp.ru Tue Mar 6 16:04:59 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Tue, 6 Mar 2007 18:04:59 +0300 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45ED81F1.502@v.loewis.de> References: <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> Message-ID: <20070306150459.GA8906@phd.pp.ru> On Tue, Mar 06, 2007 at 04:00:01PM +0100, "Martin v. L?wis" wrote: > > Yes. In .pythonrc.py .pythonrc is the root, and .py is the extension. > > Ah, it would do that already: with multiple dots, the last one always > provides the extension. Ah, sorry. I messed it with .split(). Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From martin at v.loewis.de Tue Mar 6 16:05:18 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 16:05:18 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> <9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> Message-ID: <45ED832E.9060501@v.loewis.de> Jeremy Hylton schrieb: > You can ask whether we should have a plan for increasing the number of > developers, actively seeking out new developers, and mentoring people > who express interest. Would the code be better if we had more good > developers working on it? Would we get more bugs fixed and patches > closed? Certainly, yes. > If so, it wouldn't hurt to have some deliberate strategy for > bringing new developers in. Well, it might hurt. I do have a strategy: I force people eager to get their patches included into reviewing, with the 5:1 deal. I do consider it a useful strategy, and feel sorry that nobody else was willing to adopt it. However, it seems that this also has hurt, because now some people believe this is the only way to get patches reviewed (but perhaps that's not too bad, because before they believed there is no way at all to get patches reviewed...). Regards, Martin From martin at v.loewis.de Tue Mar 6 16:07:16 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 16:07:16 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <20070306150459.GA8906@phd.pp.ru> References: <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> Message-ID: <45ED83A4.80604@v.loewis.de> Oleg Broytmann schrieb: > On Tue, Mar 06, 2007 at 04:00:01PM +0100, "Martin v. L?wis" wrote: >>> Yes. In .pythonrc.py .pythonrc is the root, and .py is the extension. >> Ah, it would do that already: with multiple dots, the last one always >> provides the extension. > > Ah, sorry. I messed it with .split(). Ok - now I'm confused: do you consider this behavior (splitext('.pythonrc') == ('', '.pythonrc')) correct or not? Regards, Martin From martin at v.loewis.de Tue Mar 6 16:10:42 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 16:10:42 +0100 Subject: [Python-Dev] git (Was: Encouraging developers) In-Reply-To: <20070306142720.GA31643@python.ca> References: <20070305184614.GA19051@localhost.localdomain> <9e804ac0703060126n18a8746lf84c0df80b72f995@mail.gmail.com> <20070306142720.GA31643@python.ca> Message-ID: <45ED8472.2060009@v.loewis.de> Neil Schemenauer schrieb: >> the on-disk repository is mighty big and it doesn't work very well >> on non-Linux systems (at least, not last I looked.) > > Not true. The on-disk repository is now one of the more efficient > ones. Which is a relative quality :-) Every time I update the Linux kernel sandbox I have, it takes many many minutes to run, and often aborts with a timeout. If this still makes git one of the more efficient dvcs systems, I don't want to see the other ones :-( Thinking of git always makes me not complain about the time subversion needs to produce a certain log message... Regards, Martin From martin at v.loewis.de Tue Mar 6 16:14:03 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 16:14:03 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45ED7D93.4010601@gmail.com> References: <20070305184614.GA19051@localhost.localdomain><9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> <45ED7D93.4010601@gmail.com> Message-ID: <45ED853B.9050700@v.loewis.de> Nick Coghlan schrieb: > One thing that did happen though (which the messages from Jeremy & Phil > reminded me of) is that I got a lot of direction, advice and assistance > from Raymond when I was doing that initial work on improving the speed > of the decimal module - I had the time available to run the relevant > benchmarks repeatedly and try different things out, while Raymond had > the experience needed to suggest possible avenues for optimisation (and > when to abandon an approach as making the code too complicated to be > effectively maintained). > > I don't know whether or not there is anything specific we can do to > encourage that kind of coaching/mentoring activity, but I know it was a > significant factor in my become more comfortable with making contributions. While there was no explicit management of a mentoring process, I think it so happened that committers always found a mentor. It so happened that some developer activated privileges for them (either himself, or requesting that this be done), and then certainly feels responsible for his 'student'. This automatically establishes a mentoring relationship. Regards, Martin From phd at phd.pp.ru Tue Mar 6 16:19:29 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Tue, 6 Mar 2007 18:19:29 +0300 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45ED83A4.80604@v.loewis.de> References: <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <45ED83A4.80604@v.loewis.de> Message-ID: <20070306151929.GD8906@phd.pp.ru> On Tue, Mar 06, 2007 at 04:07:16PM +0100, "Martin v. L?wis" wrote: > Oleg Broytmann schrieb: > > On Tue, Mar 06, 2007 at 04:00:01PM +0100, "Martin v. L?wis" wrote: > >>> Yes. In .pythonrc.py .pythonrc is the root, and .py is the extension. > >> Ah, it would do that already: with multiple dots, the last one always > >> provides the extension. > > > > Ah, sorry. I messed it with .split(). > > Ok - now I'm confused: do you consider this behavior > (splitext('.pythonrc') == ('', '.pythonrc')) correct > or not? I messed this in the sense that I have ran .split('.pythonrc.py'), got ('', '.pythonrc.py') and thought there is a problem. .split() works fine. Even .splitext() works fine with multiple dots: >>> os.path.splitext("/bin/.pythonrc.py") ('/bin/.pythonrc', '.py') but >>> os.path.splitext(".pythonrc") ('', '.pythonrc') and I think it should be ('.pythonrc', '') Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From martin at v.loewis.de Tue Mar 6 16:19:59 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 16:19:59 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070306144130.GA5613@localhost.localdomain> References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> <200703060906.22762.phil@riverbankcomputing.co.uk> <20070306144130.GA5613@localhost.localdomain> Message-ID: <45ED869F.8030706@v.loewis.de> A.M. Kuchling schrieb: > For example, our oldest bug is http://www.python.org/sf/214033, opened > 2000-09-11, and is that (x?)? is reported as an error by the SRE regex > parser; the PCRE engine and Perl both accept it. Fixing it requires > changing sre_parse, figuring out what to do (should it be equivalent > to '(x?)' or to '(x)?', and then being very sure that no other > patterns are broken by the change. I suspect that fixing this bug > properly by researching the right answer, implementing it, and adding > tests would take me a half-day's worth of work. If modifying > sre_parse is very difficult, it could take longer. I just applied a patch by Aaron Watters for HTMLParser, which was three years old. The patch wasn't contributed in unified diff, it hadn't test cases and documentation changes, and it changed the way HTMLParser does unescaping of references in HTML attributes. It was a fairly small change, yet it contained a bug, and a likely-to-occur boundary behavior (entity references for undefined entities). In testing, I found that HTMLParser (incorrectly) supports ' and decided to continue to support it for compatibility. All in all, it took me about one hour to review and apply this patch (again, even though it changed only 20 lines or so). I felt it wouldn't have been fair to Aaron to go back and ask for unified diffs etc now that the patch had been sitting there for several years. Yet, in all these years, nobody else commented that the patch was incomplete, let alone commenting on whether the feature was desirable. Regards, Martin From amk at amk.ca Tue Mar 6 16:22:58 2007 From: amk at amk.ca (A.M. Kuchling) Date: Tue, 6 Mar 2007 10:22:58 -0500 Subject: [Python-Dev] SVK In-Reply-To: <20070306145335.GB5613@localhost.localdomain> References: <20070305184614.GA19051@localhost.localdomain> <17900.26693.804101.291817@montanaro.dyndns.org> <20070306145335.GB5613@localhost.localdomain> Message-ID: <20070306152258.GA7895@localhost.localdomain> On Tue, Mar 06, 2007 at 09:53:35AM -0500, A.M. Kuchling wrote: > initial setup seems very slow: SVK is retrieving each of 54165 > revisions and it'll probably take over an hour to complete. It's even worse than that. Retrying with the trunk, SVK synced 500 revisions in about 10 minutes. We have 54165 revisions (not all trunk ones). That means it will take about 54165/500 * 10 minutes = 1080 minutes = 18 hours. Yikes! SVK has an option to skip revisions, the -s argument to sync, but it doesn't seem to work: amk at matterhorn:~/tmp$ svk mirror http://svn.python.org/projects/python/ //mirror/python Committed revision 1977. amk at matterhorn:~/tmp$ svk sync -s 54100 //mirror/python HTTP Path Not Found: REPORT request failed on '/projects/!svn/bc/1/python': '/projects/!svn/bc/1/python' path not found amk at matterhorn:~/tmp$ --amk From martin at v.loewis.de Tue Mar 6 16:34:50 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 16:34:50 +0100 Subject: [Python-Dev] Access to bits for a PyLongObject In-Reply-To: <45ED7091.3010705@trueblade.com> References: <45ECCA92.8090709@trueblade.com> <45ECF545.2080806@v.loewis.de> <45ED7091.3010705@trueblade.com> Message-ID: <45ED8A1A.60908@v.loewis.de> Eric V. Smith schrieb: > Also, it would either mean duplicating lots of code from the int > formatter, or having a formatter library that both can call. This is > because __format__ must implement all formats, including padding, > parenthesis for negatives, etc., not just the "missing" binary format. > Not that that's necessarily bad, either. But see the next point. Ah, I had missed the point that it's just binary formatting that you are concerned with (and then I missed that binary is "base 2", rather than "sequence of bits") >> OTOH, also look at _PyString_FormatLong. > > I think a solution would be to add 'b' to _PyString_FormatLong, which > I'm already calling for hex, octal, and decimal formatting. Does that > sound reasonable? It seems to me that if binary is useful enough for > PEP 3101, it should generally be available in _PyString_FormatLong. That sounds fine. > The obvious implementation of this would require adding a nb_binary to > PyNumberMethods. I'm not sure what the impact of that change would be, > but it sounds really big and probably a show-stopper. Maybe a direct > call to a binary formatter would be better. Sure, introducing _PyLong_Dual (or _PyLong_AsDualString) would be appropriate, that can then forward to long_format. > OTOH, this approach isn't as efficient as I'd like (for all formatting > outputs, not just binary), because it has to build a string object and > then copy data out of it. Ah, but that's a proof-of-concept implementation only, right? A "true" implementation should use __format__ (or whatever it's called). If *that* then isn't efficient, you should be worried (and consider introduction of a slot in the type object). > Having written all of this, I'm now thinking that Nick's suggestion of > _PyLong_AsByteArray might be the way to go. I would use that for all of > my formatting for longs. How would you do negative numbers, then? AsByteArray gives you two's complement. > Maybe I'm over-emphasizing performance, given the early status of the > implementation. Most definitely. > But I'd like PEP 3101 to be as efficient as possible, > because once it's available I'll replace all of the '%' string > formatting in my code with it. That is fine. However, don't trade efficiency for maintainability. Keep encapsulation of types, this is what OO is for. Modularize along with type boundaries. If that loses efficiency, come up with interfaces that still modularize in that way but are efficient. Don't "hack" to achieve performance. (Any other way I can formulate the same objective :-?) Regards, Martin From martin at v.loewis.de Tue Mar 6 16:35:31 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 16:35:31 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <20070306151929.GD8906@phd.pp.ru> References: <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <45ED83A4.80604@v.loewis.de> <20070306151929.GD8906@phd.pp.ru> Message-ID: <45ED8A43.8090400@v.loewis.de> Oleg Broytmann schrieb: >>>> os.path.splitext(".pythonrc") > ('', '.pythonrc') > > and I think it should be > > ('.pythonrc', '') Thanks, so it sounds like the patch should be accepted. Regards, Martin From martin at v.loewis.de Tue Mar 6 16:44:42 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 16:44:42 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> Message-ID: <45ED8C6A.30901@v.loewis.de> Miguel Lobo schrieb: > As I am completely new to CPython development, perhaps this problem has > already been discussed and/or fixed I may have done something > incorrectly. Please let me know if that is the case. I looked at it briefly. If I understand correctly, the proposed feature is fine, but lacks a test case. Regards, Martin From martin at v.loewis.de Tue Mar 6 16:48:48 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 16:48:48 +0100 Subject: [Python-Dev] SVK In-Reply-To: <20070306145335.GB5613@localhost.localdomain> References: <20070305184614.GA19051@localhost.localdomain> <17900.26693.804101.291817@montanaro.dyndns.org> <20070306145335.GB5613@localhost.localdomain> Message-ID: <45ED8D60.1080708@v.loewis.de> A.M. Kuchling schrieb: > I'm happy to write up a wiki page describing how to use SVK to set up > your own mirror of the Python SVN. However, trying it now, the > initial setup seems very slow: SVK is retrieving each of 54165 > revisions and it'll probably take over an hour to complete. If it helps, you can download a snapshot of the entire repository from http://svn.python.org/snapshots/projects-svn-tarball.tar.bz2 This is more than Python, so it depending on your link, it may or may not pay off. You can also try to run the svk conversion on dinsdale, if that helps, so we can provide daily initial svn repositories (if there is such a thing) for download. > (Of course, I don't know how long a checkout of a hypothetical Bazaar > repository would take; maybe it's not any faster.) From my experience with git and the Linux repository, an hour is "about right". People familiar with these tools probably learn to do initial checkouts over night. Regards, Martin From g.brandl at gmx.net Tue Mar 6 16:49:00 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 06 Mar 2007 16:49:00 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45ED853B.9050700@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain><9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> <45ED7D93.4010601@gmail.com> <45ED853B.9050700@v.loewis.de> Message-ID: Martin v. L?wis schrieb: > Nick Coghlan schrieb: >> One thing that did happen though (which the messages from Jeremy & Phil >> reminded me of) is that I got a lot of direction, advice and assistance >> from Raymond when I was doing that initial work on improving the speed >> of the decimal module - I had the time available to run the relevant >> benchmarks repeatedly and try different things out, while Raymond had >> the experience needed to suggest possible avenues for optimisation (and >> when to abandon an approach as making the code too complicated to be >> effectively maintained). >> >> I don't know whether or not there is anything specific we can do to >> encourage that kind of coaching/mentoring activity, but I know it was a >> significant factor in my become more comfortable with making contributions. > > While there was no explicit management of a mentoring process, I think > it so happened that committers always found a mentor. It so happened > that some developer activated privileges for them (either himself, or > requesting that this be done), and then certainly feels responsible > for his 'student'. This automatically establishes a mentoring > relationship. Perhaps we should really try to make *that* notion widely known out there, opposed to others like the alleged 5:1 requirement or that it is hard to get patches into the Python core :) A sort of announcement and some text on the website would surely help... Georg From pje at telecommunity.com Tue Mar 6 16:51:10 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 06 Mar 2007 10:51:10 -0500 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45ED83A4.80604@v.loewis.de> References: <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> Message-ID: <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> At 04:07 PM 3/6/2007 +0100, Martin v. L?wis wrote: >Oleg Broytmann schrieb: > > On Tue, Mar 06, 2007 at 04:00:01PM +0100, "Martin v. L?wis" wrote: > >>> Yes. In .pythonrc.py .pythonrc is the root, and .py is the extension. > >> Ah, it would do that already: with multiple dots, the last one always > >> provides the extension. > > > > Ah, sorry. I messed it with .split(). > >Ok - now I'm confused: do you consider this behavior >(splitext('.pythonrc') == ('', '.pythonrc')) correct >or not? I consider it correct, or at the least, don't think it should be changed, as it would make the behavior more difficult to reason about and introduce yet another thing to worry about when writing cross-version code. From aahz at pythoncraft.com Tue Mar 6 16:52:31 2007 From: aahz at pythoncraft.com (Aahz) Date: Tue, 6 Mar 2007 07:52:31 -0800 Subject: [Python-Dev] locals(), closures, and IronPython... In-Reply-To: References: <7AD436E4270DD54A94238001769C22276A624C7482@DF-GRTDANE-MSG.exchange.corp.microsoft.com> Message-ID: <20070306155230.GD4528@panix.com> On Tue, Mar 06, 2007, Andrew Dalke wrote: > On 3/5/07, Guido van Rossum wrote: >> >> I don't know too many good use cases for >> locals() apart from "learning about the implementation" I think this >> might be okay. > > Since I'm watching this list for any discussion on the traceback > threads, I figured I would point out the most common use I know > for locals() is in string interpolation when there are many local > variables, eg: > > a = "spam" > b = "egg" > ... > y = "foo" > z = "bar" > > print fmtstr % locals() I'll second this one. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "I disrespectfully agree." --SJM From aahz at pythoncraft.com Tue Mar 6 16:53:34 2007 From: aahz at pythoncraft.com (Aahz) Date: Tue, 6 Mar 2007 07:53:34 -0800 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> References: <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> Message-ID: <20070306155334.GE4528@panix.com> On Tue, Mar 06, 2007, Phillip J. Eby wrote: > At 04:07 PM 3/6/2007 +0100, Martin v. L?wis wrote: >> >>Ok - now I'm confused: do you consider this behavior >>(splitext('.pythonrc') == ('', '.pythonrc')) correct >>or not? > > I consider it correct, or at the least, don't think it should be changed, > as it would make the behavior more difficult to reason about and introduce > yet another thing to worry about when writing cross-version code. That's my primary concern, as well. I'm -0 WRT 2.x; +1 WRT 3.x. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "I disrespectfully agree." --SJM From g.brandl at gmx.net Tue Mar 6 16:54:03 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 06 Mar 2007 16:54:03 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070306144130.GA5613@localhost.localdomain> References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> <200703060906.22762.phil@riverbankcomputing.co.uk> <20070306144130.GA5613@localhost.localdomain> Message-ID: A.M. Kuchling schrieb: > On Tue, Mar 06, 2007 at 09:06:22AM +0000, Phil Thompson wrote: >> My point is simply that the effort required to review patches seems to be a >> problem. Perhaps the reasons for that need to be looked at and the process >> changed so that it is more effective. At the moment people just seem be >> saying "that's the way it is because that's the way it's got to be". > > Unfortunately I think the effort required is intrinsic to reviewing > patches. Trivial or obviously correct patches get applied quickly, so > the remaining bugs and patches are the ones that are hard to fix. Exactly. I often find myself looking at a patch or bug and turn away thinking "I would need hours to review and check that in, while the developer who originally wrote the code might perhaps do it in much less time". It's not easy to apply patches to code that was written by others and that should stay as compatible as possible for all cases not covered by the patch. Georg From barry at python.org Tue Mar 6 17:00:50 2007 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Mar 2007 11:00:50 -0500 Subject: [Python-Dev] SVK In-Reply-To: <45ED8D60.1080708@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <17900.26693.804101.291817@montanaro.dyndns.org> <20070306145335.GB5613@localhost.localdomain> <45ED8D60.1080708@v.loewis.de> Message-ID: <4EA5964E-C0E5-4E6E-8B38-41242E627281@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mar 6, 2007, at 10:48 AM, Martin v. L?wis wrote: >> (Of course, I don't know how long a checkout of a hypothetical Bazaar >> repository would take; maybe it's not any faster.) > > From my experience with git and the Linux repository, an hour is > "about right". People familiar with these tools probably learn > to do initial checkouts over night. I can't speak to a python-on-bzr, but for another project, doing the initial branch of a huge and deep history took maybe an hour, hour- and-a-half over the network. I know that bzr is trying to improve this. However, by checking out into a repository, once all that history was on my local machine, branching was trivially fast. And if your pushing changes into a remote repository that's up-to-date, that will be very fast too. So yes, there's some initial overhead but with bzr done right, for most subsequent work, there's no perceptible lag due to the scm. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRe2QM3EjvBPtnXfVAQIaSAP/SAoPvqTQZgfG8EDZB6J2hgggAUgWlYun xCtYYtqgnFq0afBOHP1/K1OuD5SFsHghjrLykJjia6TNRC2MpRblk/44Xc5syXCJ aCCJ7t1CLZWMjLjyEpPIcpwBNC6DNfOQYZc28b5BabeIHsF3FqsZb4sIKsSU+dK2 gigs3rjbHRE= =z/vM -----END PGP SIGNATURE----- From nick.bastin at gmail.com Tue Mar 6 17:24:36 2007 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Tue, 6 Mar 2007 11:24:36 -0500 Subject: [Python-Dev] Subversion checkout hanging? Message-ID: <66d0a6e10703060824l695f4b21g9337ba576dbce765@mail.gmail.com> I've had to blast my windows machine, as one is apparently required to do on occasion, and I'm trying to set up subversion again. I saved my private key file, and I can use plink -T to connect and I get: ( success ( 1 2 ( ANONYMOUS EXTERNAL ) ( edit-pipeline ) ) ) and that seems correct, and jives with the FAQ at least. I've also edited my %APPDATA%/Subversion/config file, and I know it was the right one, because I screwed it up at first and it didn't work at all.. :-) However, now I'm just getting a hang when I execute: svn checkout svn+ssh://pythondev at svn.python.org/python/trunk I've only let it go for about 5 minutes so far, so maybe it's thinking about something, but I suspect it isn't... Anyone have any idea what I've done wrong? -- Nick From stephen at xemacs.org Tue Mar 6 17:50:25 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 07 Mar 2007 01:50:25 +0900 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> <9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> Message-ID: <87wt1ue2pa.fsf@uwakimon.sk.tsukuba.ac.jp> George Brandl writes: >> As far as I recall, there has been nearly no one who asked for >> commit rights recently, so why complain that the entry barrier is >> too great? Surely you cannot expect python-dev to got out and say >> "would you like to have commit privileges?"... Why not? It depends on how far out "out" is, but I was surprised how much effect we (at XEmacs) got by simply asking people who contributed a couple of patches if they would like to take on tracking + patch flow management for their own patches in return for direct access to the repository. Giving out authority to approve commits is another matter, but as long as the new people are willing to participate in the mechanics it's generally a management win to have more committers. The answer is more often than not "no, but *thanks for asking*." Merely asking creates an atmosphere of openness and trust. That's a lot, when simply asking that developers of patches consider third parties' use cases is attacked as a "high barrier" to participation. And of course it's the most common path to greater authority. From ilya at bluefir.net Tue Mar 6 17:49:00 2007 From: ilya at bluefir.net (Ilya Sandler) Date: Tue, 6 Mar 2007 08:49:00 -0800 (PST) Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <200703061444.52832.meine@informatik.uni-hamburg.de> References: <45ED6033.2020807@v.loewis.de> <200703061444.52832.meine@informatik.uni-hamburg.de> Message-ID: On Tue, 6 Mar 2007, Hans Meine wrote: > Am Dienstag, 06. M?rz 2007 13:36 schrieb Martin v. L?wis: > > #1115886 complains that in the file name '.cshrc', the > > entire file name is treated as an extension, with no > > root. > > The current behavior is clearly a bug, since a leading dot does not start an > extension, but marks a file as "hidden". It's not at all clear that current behaviour should be considered a bug . e.g, I think it's reasonable to expect that splitext( a+"." + b) == (a, .b ) for any a,b which have no dots in them... The patch will break this assumption for empty a So, I'd classify dot files as a border case (either behaviour could be viewed as wrong/correct).. In which case backward compatibility should be the primary consideration IMHO Ilya > Greetings, > Hans > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ilya%40bluefir.net > From dinov at exchange.microsoft.com Tue Mar 6 17:57:36 2007 From: dinov at exchange.microsoft.com (Dino Viehland) Date: Tue, 6 Mar 2007 08:57:36 -0800 Subject: [Python-Dev] locals(), closures, and IronPython... In-Reply-To: References: <7AD436E4270DD54A94238001769C22276A624C7482@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <7AD436E4270DD54A94238001769C22276A624C7574@DF-GRTDANE-MSG.exchange.corp.microsoft.com> Message-ID: <7AD436E4270DD54A94238001769C22276A624C77C3@DF-GRTDANE-MSG.exchange.corp.microsoft.com> The lifetime issue is bad - unfortunately we have the same issue in v1.x we just don't show you the names/values. That alone should (and hopefully will) drive us to clean this up but right now we'll only be worse in that we are explicit about keeping the dictionaries alive. Classes are interesting for us as well... Our model is that we chain all of our dictionaries together including the classes in FunctionEnvironment's (just an optimized dictionary, usually only containing the variables used in the closure) and ultimately that chains out to the globals dictionary. The change that exposes all of the locals is to drop the explicit list of keys in the outer scopes. The classes dictionaries get mixed in here as well and so now when doing a fully dynamic name-lookup we need to know to skip the classes which is strange. How would "nonlocals" be used in the method? Wouldn't you need a new opcode which would resolve to a non-local variable or add it explicitly into the LEGB search? -----Original Message----- From: jhylton at gmail.com [mailto:jhylton at gmail.com] On Behalf Of Jeremy Hylton Sent: Tuesday, March 06, 2007 4:54 AM To: Dino Viehland Cc: Guido van Rossum; python-dev at python.org Subject: Re: [Python-Dev] locals(), closures, and IronPython... On 3/5/07, Dino Viehland wrote: > Thanks Guido. It might take some time (and someone may very well beat me to it if they care a lot) but I'll see if we can get the PEP started. Dino, One of the questions I was puzzling over was what locals() should return in a class scope. It's confusing in CPython, because the same dictionary is used to stored names that are destined to become class attributes and names that are used to created closures for methods with free variables. The implementation was causing the latter names to show up as class attributes when you called locals(). Terry is right that lifetime of variables is a key issue. It would probably be bad if all the local variables of a function were kept alive because a nested function used locals(). One idea was discussed at PyCon was having a different representation for an environment so that it would be easier to introspect on it but still possible to pass the environment to exec. One possibility is just a third dictionary--globals, locals, nonlocals. Another possibility is an object where you can ask about the scope of each variable but also extract a dictionary to pass to locals. Jeremy > > -----Original Message----- > From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido van Rossum > Sent: Monday, March 05, 2007 2:14 PM > To: Dino Viehland > Cc: python-dev at python.org > Subject: Re: [Python-Dev] locals(), closures, and IronPython... > > Jeremy Hylton has been asking questions about this too at the sprint > after PyCon. I'm tempted to accept that the exact behavior of locals() > is implementation-defined (IOW undefined :-) as long as it includes > the locals defined in the current scope; whether it also includes free > variables could be debatable. I don't know too many good use cases for > locals() apart from "learning about the implementation" I think this > might be okay. Though a PEP might be in order to get agreement between > users, developers and other implementation efforts (e.g. PyPy, > Jython). > > On 3/5/07, Dino Viehland wrote: > > > > > > > > > > def a(): > > > > x = 4 > > > > y = 2 > > > > def b(): > > > > print y, locals() > > > > print locals() > > > > b() > > > > > > > > a() > > > > > > > > in CPython prints: > > > > > > > > {'y': 2, 'x': 4, 'b': } > > > > 2 {'y': 2} > > > > > > > > I'm wondering if it's intentional that these don't print dictionaries w/ the > > same contents or if it's more an accident of the implementation. In other > > words would it be reasonable for IronPython to promote all of the locals of > > a into b's dictionary when both a and b call locals? > > > > > > > > We currently match CPython's behavior here - at least in what we display > > although possibly not in the lifetimes of objects. We're considering an > > internal change which might alter the behavior here though and end up > > displaying all the members. > > > > > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu > From phd at phd.pp.ru Tue Mar 6 18:05:10 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Tue, 6 Mar 2007 20:05:10 +0300 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: References: <45ED6033.2020807@v.loewis.de> <200703061444.52832.meine@informatik.uni-hamburg.de> Message-ID: <20070306170509.GA18237@phd.pp.ru> On Tue, Mar 06, 2007 at 08:49:00AM -0800, Ilya Sandler wrote: > I think it's reasonable to expect that > > splitext( a+"." + b) == (a, .b ) > > for any a,b which have no dots in them... Except for an empty 'a', in what case 'b' is the name, not the extension. Well, 'a' cannot be empty because it's the name, but 'b' can be empty. For ".pythonrc.py" ".pythonrc" is the name and ".py" is the extension. For ".pythonrc" ".pythonrc" is the name and there is no extension. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From dustin at v.igoro.us Tue Mar 6 18:22:49 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Tue, 6 Mar 2007 11:22:49 -0600 Subject: [Python-Dev] Encouraging developers In-Reply-To: <87wt1ue2pa.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20070305184614.GA19051@localhost.localdomain> <9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> <87wt1ue2pa.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20070306172249.GE11462@v.igoro.us> On Wed, Mar 07, 2007 at 01:50:25AM +0900, Stephen J. Turnbull wrote: > Why not? It depends on how far out "out" is, but I was surprised how > much effect we (at XEmacs) got by simply asking people who contributed > a couple of patches if they would like to take on tracking + patch > flow management for their own patches in return for direct access to > the repository. I've been thinking about how other projects handle this. One method is to appoint "maintainers" of specific pieces of functionality. The term "maintainers" was used earlier, but referring only to core developers (those with commit access). I have a somewhat different proposal: In summary, create a layer of volunteer, non-committing maintainers for specific modules who agree to do in-depth analysis of patches for their areas of expertise, and pass well-formed, reviewed patches along to committers. Every part of Python gets a maintainer, annotated in the comments atop the file. A basic process is established for promotion/demotion of maintainers. New patches for a module get sent to that module's maintainer, who checks for well-formedness, functionality, and appropriateness to the module. The *maintainer* can then refer successful patches to the core developers, who can just skim the patch and check that the unit test pass. The core of the interpreter would be implicitly maintained by the core developers, while each module or package of the stdlib is assigned to a specific maintainer (or several, if more are willing). New modules are initially assigned to their author, while existing modules with no apparent maintainer are assigned to a "maintainer-needed" pseudo-maintainer. Patches to maintainer-less modules would languish, unless the submitter stepped up as maintainer, or yelled loudly enough that the core devs processed the pach. I think this would have several advantages: - maintainers can do the basic screening that takes so long and is no fun for core developers - being a maintainer can be a stepping-stone to becoming a full developer, for those who wish to join - patch authors have an advocate "inside the system" Speaking personally, I don't want to be a core developer, but I would be happy to maintain a half-dozen stdlib modules. This is loosely based on the Gentoo project's idea of maintainers (although in Gentoo maintainers must be full developers). Dustin From gustavotabares at gmail.com Tue Mar 6 18:43:55 2007 From: gustavotabares at gmail.com (Gustavo Tabares) Date: Tue, 6 Mar 2007 12:43:55 -0500 Subject: [Python-Dev] PCBuild8 In-Reply-To: <45EC8856.1060305@v.loewis.de> References: <87lkibd1od.fsf@valverde.peloton> <45EC5EED.1050507@v.loewis.de> <45EC7CBA.5050301@abaqus.com> <45EC8856.1060305@v.loewis.de> Message-ID: <1871c95a0703060943w45c91159j25f73f641502e30e@mail.gmail.com> I ran into the same problem and I'm pretty sure this cleared it up: http://mail.python.org/pipermail/python-dev/2006-August/068369.html Good luck, Gustavo On 3/5/07, "Martin v. L?wis" wrote: > > Andrew MacKeith schrieb: > > Is there a scheduled date for the release of Python-2.5.1 ? > > There was a scheduled date, but it then interfered with the schedule > of the people executing it, so there is none at the moment. I think > we will release 2.5.1 in April. > > > I presume that the PCBuild8 directory should be used when building in 64 > > bit on Win64. > > No. The official binaries both for Itanium and AMD64 (which are both > "in 64 bit on Win64") are created with the PCbuild directory. See > PCbuild/readme.txt. > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/gustavotabares%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070306/1a6edd3c/attachment.html From tlesher at gmail.com Tue Mar 6 18:44:32 2007 From: tlesher at gmail.com (Tim Lesher) Date: Tue, 6 Mar 2007 12:44:32 -0500 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <200703061444.52832.meine@informatik.uni-hamburg.de> References: <45ED6033.2020807@v.loewis.de> <200703061444.52832.meine@informatik.uni-hamburg.de> Message-ID: <9613db600703060944v17c4c372n151a58aeb801a853@mail.gmail.com> On 3/6/07, Hans Meine wrote: > The current behavior is clearly a bug, since a leading dot does not start an > extension, but marks a file as "hidden". The latter is on UNIX, and while > this is different on Windows, I cannot imagine that anyone would > a) have dotfiles under that OS, or even then > b) rely on the current behavior that their full filename counts as > "extension". FWIW, all of the "standard" Windows functions from the Microsoft CRT (_splitpath) to the Shell API (PathRemoveExtension) to the CLR (System.IO.Path.*) believe that ".cshrc" is the extension of the filename ".cshrc". I'm not sure if that's an argument for or against the patch, though. -- Tim Lesher From ilya at bluefir.net Tue Mar 6 18:54:44 2007 From: ilya at bluefir.net (Ilya Sandler) Date: Tue, 6 Mar 2007 09:54:44 -0800 (PST) Subject: [Python-Dev] Encouraging developers In-Reply-To: <45ED869F.8030706@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> <200703060906.22762.phil@riverbankcomputing.co.uk> <20070306144130.GA5613@localhost.localdomain> <45ED869F.8030706@v.loewis.de> Message-ID: On Tue, 6 Mar 2007, [ISO-8859-1] "Martin v. L?wis" wrote: > Yet, in all these years, nobody else commented that the patch was incomplete, > let alone commenting on whether the feature was desirable. Which actually brings up another point: in many cases even a simple comment by a core developer: "yes this feature is desirable" might be of considerable value as it's likely to increase chances that some other developer will decide to spend time on the patch.... Similarly, some bug reports are for border cases. A confirmation by a core developer: "yes, that needs fixing" might encourage someone else to submit a patch... I'd also suggest that request for test cases/docs comes after (or together with) suggestion that a feature is desirable in the first place. Ilya > Regards, > Martin > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ilya%40bluefir.net > From g.brandl at gmx.net Tue Mar 6 19:00:18 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 06 Mar 2007 19:00:18 +0100 Subject: [Python-Dev] Subversion checkout hanging? In-Reply-To: <66d0a6e10703060824l695f4b21g9337ba576dbce765@mail.gmail.com> References: <66d0a6e10703060824l695f4b21g9337ba576dbce765@mail.gmail.com> Message-ID: Nicholas Bastin schrieb: > I've had to blast my windows machine, as one is apparently required to > do on occasion, and I'm trying to set up subversion again. I saved my > private key file, and I can use plink -T to connect and I get: > > ( success ( 1 2 ( ANONYMOUS EXTERNAL ) ( edit-pipeline ) ) ) > > and that seems correct, and jives with the FAQ at least. I've also > edited my %APPDATA%/Subversion/config file, and I know it was the > right one, because I screwed it up at first and it didn't work at > all.. :-) > > However, now I'm just getting a hang when I execute: > > svn checkout svn+ssh://pythondev at svn.python.org/python/trunk > > I've only let it go for about 5 minutes so far, so maybe it's thinking > about something, but I suspect it isn't... > > Anyone have any idea what I've done wrong? You could try to do ssh -vv pythondev at svn.python.org and see if the debug messages mean anything to you. Georg From martin at v.loewis.de Tue Mar 6 19:24:51 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 19:24:51 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> References: <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> Message-ID: <45EDB1F3.6060801@v.loewis.de> Phillip J. Eby schrieb: > I consider it correct, or at the least, don't think it should be > changed, as it would make the behavior more difficult to reason about > and introduce yet another thing to worry about when writing > cross-version code. Now it's becoming difficult: several people in favor, some opposed... I'll wait a bit longer, but will still likely commit it, unless opposition gets stronger: If the current behavior is incorrect (in the sense that it contradicts wide-spread intuition), then an application worrying about this detail should surely make the 2.6 behavior also appear in 2.5 and earlier. I'm not sure what people actually use splitext for: I guess there are two applications: a) given a list of file names, give me all those belonging to a hard-coded list of extensions (e.g. .py, .pyc, .c, .h). These won't break, since they likely won't search for "all files ending in .bash_profile" - there is only one per directory, and if the want it, they use the entire filename. b) given a list of file names, classify them for display (the way the Windows explorer works, and similar file managers). They use MIME databases and the like, and if they are unix-ish, they probably reject the current splitext implementation already as incorrect, and have work-arounds. As these files now show up with "no extension", I rather expect that the work-around won't trigger, and the default behavior will be the correct one. Regards, Martin From martin at v.loewis.de Tue Mar 6 19:32:32 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 19:32:32 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> <200703060906.22762.phil@riverbankcomputing.co.uk> <20070306144130.GA5613@localhost.localdomain> <45ED869F.8030706@v.loewis.de> Message-ID: <45EDB3C0.3030604@v.loewis.de> Ilya Sandler schrieb: > I'd also suggest that request for test cases/docs comes after > (or together with) suggestion that a feature is desirable in the first > place. It depends. I was going through some old patches today, and came across one that added a class to heapq. I couldn't tell (even after reading the code) what precisely all implications are, so I was unable to proceed review beyond "please provide documentation saying what this is for". I would then be able to tell: a) whether it really does what it promises to do, and b) whether this is desirable to have I remember many occasions where a patch was rejected and never reconsidered (even after discussion on python-dev) because the submitter failed to clearly specify what the intention of the change was, as it turned out that the code didn't implement the intention, and people reviewing rejected it because they thought it was meant to do something else (namely, to do what it actually did). Regards, Martin From nick.bastin at gmail.com Tue Mar 6 19:46:36 2007 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Tue, 6 Mar 2007 13:46:36 -0500 Subject: [Python-Dev] Subversion checkout hanging? In-Reply-To: References: <66d0a6e10703060824l695f4b21g9337ba576dbce765@mail.gmail.com> Message-ID: <66d0a6e10703061046n74516047jd6a59a3db3756336@mail.gmail.com> On 3/6/07, Georg Brandl wrote: > You could try to do > > ssh -vv pythondev at svn.python.org > > and see if the debug messages mean anything to you. My problem is that SSH works fine if you just try to do that (well, with plink). It's subversion that doesn't seem to be working. -- Nick From nick.bastin at gmail.com Tue Mar 6 19:51:51 2007 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Tue, 6 Mar 2007 13:51:51 -0500 Subject: [Python-Dev] Subversion checkout hanging? In-Reply-To: <66d0a6e10703061046n74516047jd6a59a3db3756336@mail.gmail.com> References: <66d0a6e10703060824l695f4b21g9337ba576dbce765@mail.gmail.com> <66d0a6e10703061046n74516047jd6a59a3db3756336@mail.gmail.com> Message-ID: <66d0a6e10703061051v43216488lcb3f107ff28cfbdc@mail.gmail.com> I've fixed it. It appears that there was something wrong with Pageant, and removing my key and readding it solved the problem. The lack of any debugging info from subversion was very helpful in solving this problem.. :-) Thanks for the help from those who responded. -- Nick From pje at telecommunity.com Tue Mar 6 20:03:20 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 06 Mar 2007 14:03:20 -0500 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45EDB1F3.6060801@v.loewis.de> References: <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> At 07:24 PM 3/6/2007 +0100, Martin v. L?wis wrote: >given a list of file names, classify them for display (the > way the Windows explorer works, and similar file managers). > They use MIME databases and the like, and if they are unix-ish, > they probably reject the current splitext implementation already > as incorrect, and have work-arounds. I know I've written code like this that *depends* on the current behavior. It's *useful* to classify e.g. .svn directories or .*rc files by their "extension", so I'm honestly baffled by the idea of wanting to treat such files as *not* having an extension (as opposed to a possibly-unrecognized one). From jason.orendorff at gmail.com Tue Mar 6 20:02:12 2007 From: jason.orendorff at gmail.com (Jason Orendorff) Date: Tue, 6 Mar 2007 14:02:12 -0500 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070305184614.GA19051@localhost.localdomain> References: <20070305184614.GA19051@localhost.localdomain> Message-ID: On 3/5/07, A.M. Kuchling wrote: > Any ideas for fixing this problem? The current developer FAQ says: 2.4 How can I become a developer? There's only one way to become a developer, and that's through the School of Hard Knocks. http://mail.python.org/pipermail/python-dev/2002-September/028725.html That's a little glib. And maybe inaccurate. That message (by Raymond Hettinger and probably not originally intended to be the first thing developers-aspirant see) seems at odds with Martin's characterization, in this thread, of Raymond's own experience. I would submit a doc patch, but what's the use. ;) I should be explicit-- I greatly admire the python-dev community and its processes. I don't get the feeling much happens in private e-mail. Quite the opposite: it feels like important decisions are regularly made on python-dev. I don't think it's hard to contribute. I don't think the stdlib is a huge mess of brokenness. And I don't think the community is either. -j From skip at pobox.com Tue Mar 6 20:03:39 2007 From: skip at pobox.com (skip at pobox.com) Date: Tue, 6 Mar 2007 13:03:39 -0600 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45ED853B.9050700@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> <45ED7D93.4010601@gmail.com> <45ED853B.9050700@v.loewis.de> Message-ID: <17901.47883.902814.735062@montanaro.dyndns.org> Nick> I don't know whether or not there is anything specific we can do Nick> to encourage that kind of coaching/mentoring activity, but I know Nick> it was a significant factor in my become more comfortable with Nick> making contributions. Martin> While there was no explicit management of a mentoring process, I Martin> think it so happened that committers always found a mentor. Could the Summer of Code be used as a vehicle to match up current developers with potentially new ones? The svn sandbox (or a branch) could serve as a place for developers to get their feet wet. Perhaps Raymond can comment on whether he thinks that makes sense based upon his experience mentoring the Decimal-in-C module. Skip From nick.bastin at gmail.com Tue Mar 6 20:08:19 2007 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Tue, 6 Mar 2007 14:08:19 -0500 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> References: <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <45EDB1F3.6060801@v.loewis.de> <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> Message-ID: <66d0a6e10703061108x1efac5fdpa66d90326432f405@mail.gmail.com> On 3/6/07, Phillip J. Eby wrote: > At 07:24 PM 3/6/2007 +0100, Martin v. L?wis wrote: > >given a list of file names, classify them for display (the > > way the Windows explorer works, and similar file managers). > > They use MIME databases and the like, and if they are unix-ish, > > they probably reject the current splitext implementation already > > as incorrect, and have work-arounds. > > I know I've written code like this that *depends* on the current > behavior. It's *useful* to classify e.g. .svn directories or .*rc files by > their "extension", so I'm honestly baffled by the idea of wanting to treat > such files as *not* having an extension (as opposed to a > possibly-unrecognized one). My argument would be that the file is not 'unnamed', with an extension of 'cshrc'. The file is actually called 'cshrc', and the '.' is metadata that is attached to tell the shell to hide the file. Assuming that we want ot be ignorant of shell semantics (and I think we do), then the file is called '.cshrc', and it has no extension. The notion of an unnamed file with an extension I think would be very odd to most people. +1 to changing the behaviour to return .cshrc as the filename, with no extension. -- Nick From tjreedy at udel.edu Tue Mar 6 20:17:12 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 6 Mar 2007 14:17:12 -0500 Subject: [Python-Dev] Encouraging developers References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de><200703060818.29114.phil@riverbankcomputing.co.uk> <45ED3506.9040707@v.loewis.de> Message-ID: ""Martin v. L?wis"" wrote in message news:45ED3506.9040707 at v.loewis.de... >1. Public identification will not help, because: >2. most code isn't in the responsibility of anybody (so publically > identifying responsibilities would leave most code unassigned), and >3. for the code that has some responsible member, things are already > fine (so public identification won't improve here) If I were looking for an 'ophan' (python-coded) module to adopt, then such a list would help me choose. It would also be helpful if the new tracker system could produce a list of module-specific open items sorted by module, since that would indicate modules needing attention, and I could look for a batch that were unassigned. Terry Jan Reedy From dave at boost-consulting.com Tue Mar 6 20:20:56 2007 From: dave at boost-consulting.com (David Abrahams) Date: Tue, 06 Mar 2007 14:20:56 -0500 Subject: [Python-Dev] Finding the python library binaries (and docs) Message-ID: <87ps7m89gn.fsf@valverde.peloton> I'm trying to find the Python library binaries associated with a given python executable. If I look at the help for sys.prefix and sys.exec_prefix import sys; help(sys) I see: prefix -- prefix used to find the Python library exec_prefix -- prefix used to find the machine-specific Python library This text is somewhat ambiguous, but from the fact that exec_prefix refers to the "machine-specific" Python library I'm led to guess that prefix's "the Python library" refers to the *.py* files that form most of the Python standard library, and "the machine-specific Python library" perhaps refers to the library binaries such as libpython25.a. However, neither of those interpretations seems to correspond to reality. My guess is that these constants correspond to the --prefix and --exec_prefix options to the configure script invocation that was used to build Python. What I'm really looking for, I think, is the directory corresponding to the --libdir option to configure, but I don't see that in sys. If I look at "configure --help," it claims the default libdir is EPREFIX/lib, so I suppose I could look in sys.exec_prefix+'/lib', but when I look at real installations I find only the shared libraries in that location. For the static libraries, I have to look in EPREFIX/lib/python$(version)/config/, and in fact there is a symlink there to the shared libs. So: 1. I think the documentation for sys and configure both need some updating 2. I'd like to know if there's an officially correct procedure for finding the library binaries associated with a Python executable. Thanks in advance, -- Dave Abrahams Boost Consulting www.boost-consulting.com From pje at telecommunity.com Tue Mar 6 20:23:39 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 06 Mar 2007 14:23:39 -0500 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <66d0a6e10703061108x1efac5fdpa66d90326432f405@mail.gmail.co m> References: <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <45EDB1F3.6060801@v.loewis.de> <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070306142130.02d30008@sparrow.telecommunity.com> At 02:08 PM 3/6/2007 -0500, Nicholas Bastin wrote: >The notion of an unnamed file with an extension I think would be very >odd to most people. Clearly, we all think that "most" people are like ourselves. :) I think that for someone with a Windows/DOS background, that's *exactly* what .cshrc looks like! (IIRC in DOS, *every* file has an extension, such that 'foo' and 'foo.' refer to the same file.) From johann at rocholl.net Tue Mar 6 20:24:43 2007 From: johann at rocholl.net (Johann C. Rocholl) Date: Tue, 6 Mar 2007 20:24:43 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> References: <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <45EDB1F3.6060801@v.loewis.de> <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> Message-ID: <8233478f0703061124o1fcc28aajbe81a1eb92cb5c2a@mail.gmail.com> On 3/6/07, Phillip J. Eby wrote: > It's *useful* to classify e.g. .svn directories or .*rc files by their "extension" I respectfully disagree. When trying to find directories named .svn or files named .bashrc, I do >>> filename in ('.svn', '.bashrc') because I don't expect filenames like foo.svn or bar.bashrc -- have you ever seen one? These are not "extensions" for me. From skip at pobox.com Tue Mar 6 20:51:41 2007 From: skip at pobox.com (skip at pobox.com) Date: Tue, 6 Mar 2007 13:51:41 -0600 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070306172249.GE11462@v.igoro.us> References: <20070305184614.GA19051@localhost.localdomain> <9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> <87wt1ue2pa.fsf@uwakimon.sk.tsukuba.ac.jp> <20070306172249.GE11462@v.igoro.us> Message-ID: <17901.50765.434570.799628@montanaro.dyndns.org> dustin> In summary, create a layer of volunteer, non-committing dustin> maintainers for specific modules who agree to do in-depth dustin> analysis of patches for their areas of expertise, and pass dustin> well-formed, reviewed patches along to committers. One problem with this sort of system is that it's difficult for many people to commit the personal resources necessary over a long period of time. Life often gets in the way. Consider the rather simple task of fielding submissions to the Python job board. I think three or four different people who have taken care of that task over the last two or three years. In each case the transition to a new person was a bit bumpy because life sort of jumped up an bit the current maintainer in the butt, leaving a scramble to find a new person to take over that role. Now consider how simple that is compared with, say, being the Python XML guru. Let's say Fred Drake takes on that role. (I'm not picking on Fred. My brain just associates him with XML-in-Python, rightly or wrongly.) Things go swimmingly for awhile, then he gets a huge load dumped on him at work, his wife has a baby and the family moves to Austin, TX to be close to his aging parents. After moving to Austin it takes a month to get a properly functioning Internet connection because the phone company is so lame. I think it's understandable that his level of committment to XML-in-Python might be reduced. Other events in his life might take over to such an extent that he forgets to (or can't easily) let anyone know. It's not until someone happens to notice (maybe Fred, maybe Martin, maybe nobody for quite awhile) that there are a bunch of XML-related bug reports and patches piling up that the team starts looking for someone new to assume that role. Skip From amk at amk.ca Tue Mar 6 20:52:03 2007 From: amk at amk.ca (A.M. Kuchling) Date: Tue, 6 Mar 2007 14:52:03 -0500 Subject: [Python-Dev] Encouraging developers In-Reply-To: <17901.47883.902814.735062@montanaro.dyndns.org> References: <20070305184614.GA19051@localhost.localdomain> <9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> <45ED7D93.4010601@gmail.com> <45ED853B.9050700@v.loewis.de> <17901.47883.902814.735062@montanaro.dyndns.org> Message-ID: <20070306195203.GA16676@localhost.localdomain> On Tue, Mar 06, 2007 at 01:03:39PM -0600, skip at pobox.com wrote: > Could the Summer of Code be used as a vehicle to match up current developers > with potentially new ones? The svn sandbox (or a branch) could serve as a > place for developers to get their feet wet. Perhaps Raymond can comment on > whether he thinks that makes sense based upon his experience mentoring the > Decimal-in-C module. Gregory Johnson, who did the rewrite of mailbox.py in the 2005 SoC, did a very good job; the module went into 2.5 and, judging by the few bug reports that have come in, gained some users pretty quickly. He also vanished after SoC was over, which is unfortunate, but not due to anything in my mentoring (I hope!). --amk From martin at v.loewis.de Tue Mar 6 20:52:51 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 20:52:51 +0100 Subject: [Python-Dev] Subversion checkout hanging? In-Reply-To: <66d0a6e10703060824l695f4b21g9337ba576dbce765@mail.gmail.com> References: <66d0a6e10703060824l695f4b21g9337ba576dbce765@mail.gmail.com> Message-ID: <45EDC693.2010909@v.loewis.de> Nicholas Bastin schrieb: > I've had to blast my windows machine, as one is apparently required to > do on occasion, and I'm trying to set up subversion again. I saved my > private key file, and I can use plink -T to connect and I get: > > ( success ( 1 2 ( ANONYMOUS EXTERNAL ) ( edit-pipeline ) ) ) > > and that seems correct, and jives with the FAQ at least. I've also > edited my %APPDATA%/Subversion/config file, and I know it was the > right one, because I screwed it up at first and it didn't work at > all.. :-) > > However, now I'm just getting a hang when I execute: > > svn checkout svn+ssh://pythondev at svn.python.org/python/trunk > > I've only let it go for about 5 minutes so far, so maybe it's thinking > about something, but I suspect it isn't... > > Anyone have any idea what I've done wrong? I would verify that you were indeed passing pythondev at svn.python.org to plink. Regards, Martin From tjreedy at udel.edu Tue Mar 6 20:55:57 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 6 Mar 2007 14:55:57 -0500 Subject: [Python-Dev] splitext('.cshrc') References: <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de><20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de><20070306150459.GA8906@phd.pp.ru> <45ED83A4.80604@v.loewis.de> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> Message-ID: "Phillip J. Eby" wrote in message news:5.1.1.6.0.20070306104954.028deec0 at sparrow.telecommunity.com... >I consider it correct, or at the least, don't think it should be changed, >as it would make the behavior more difficult to reason about and introduce >yet another thing to worry about when writing cross-version code. Windows did not allow .xxx as a filename in my attempts, so this case seems irrelevant there. tjr From dustin at v.igoro.us Tue Mar 6 21:09:57 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Tue, 6 Mar 2007 14:09:57 -0600 Subject: [Python-Dev] Encouraging developers In-Reply-To: <17901.50765.434570.799628@montanaro.dyndns.org> References: <20070305184614.GA19051@localhost.localdomain> <9e804ac0703051338k50222fa2jcf926d8c1058305b@mail.gmail.com> <200703052326.46667.phil@riverbankcomputing.co.uk> <005c01c75fb6$dab881c0$6800a8c0@RaymondLaptop1> <87wt1ue2pa.fsf@uwakimon.sk.tsukuba.ac.jp> <20070306172249.GE11462@v.igoro.us> <17901.50765.434570.799628@montanaro.dyndns.org> Message-ID: <20070306200957.GK11462@v.igoro.us> On Tue, Mar 06, 2007 at 01:51:41PM -0600, skip at pobox.com wrote: > > dustin> In summary, create a layer of volunteer, non-committing > dustin> maintainers for specific modules who agree to do in-depth > dustin> analysis of patches for their areas of expertise, and pass > dustin> well-formed, reviewed patches along to committers. > > One problem with this sort of system is that it's difficult for many people > to commit the personal resources necessary over a long period of time. Life > often gets in the way. This is *definitely* the core problem with this system, and has plagued every project to use a variant of it (including many small projects with only one developer who takes months to respond to email). I think one *advantage* of this system would be that, with patch submitters having a specific person to whom their patches should be addressed, non-responsiveness on that person's part would be detected and brought to the community's attention more quickly. It would help a great deal to have a very formalized system in place for promoting/demoting maintainers -- email templates with filterable subject lines and specific addresses to send them to, specific expected response times, etc. As someone else said in another thread, we all think that everyone thinks like us (I think that's tautological?). My thinking is that a lot of people like me would love to have a small "corner of Python" for which they are responsible. Dustin From rrr at ronadam.com Tue Mar 6 21:38:25 2007 From: rrr at ronadam.com (Ron Adam) Date: Tue, 06 Mar 2007 14:38:25 -0600 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> <200703052116.48035.phil@riverbankcomputing.co.uk> <45ED00FC.6050406@v.loewis.de> <200703060834.06715.phil@riverbankcomputing.co.uk> Message-ID: <45EDD141.20605@ronadam.com> Neal Norwitz wrote: > I recognize there is a big problem here. Each of us as individuals > don't scale. So in order to get stuff done we need to be more > distributed. This means distributing the workload (partially so we > don't burn out). In order to do that we need to distribute the > knowledge. That probably involves some changes. Neil with deep insight states... """In order to do that we need to distribute the knowledge.""" + 1000 I'm looking forward to a new tracker and hope it manages single projects... (patches and bugs) better. It would be great if we could search for items based on possibly the following conditions. patch_complete patch_reviewed patch_approved test_complete test_reviewed test_approved doc_complete doc_reviewed doc_approved For new features: pep_completed pep_reviewed pep_approved Finally: (after all the above applicable conditions are true) accept_reject (python-dev (or BDFL) approval) [*] *Note: Rejections could be done sooner if it obviously a bad idea. In the case of bugs, the tests would probably come first in order to verify and reproduce the specific bug. What something like this would do is effectively distribute knowledge as you suggest. For example someone good at writing docs could do a search for all patches that do not have doc's or need docs reviewed and contribute in that way. The same for someone interested in writing and reviewing tests. It would also be easy for python core developers to get a list of items that are completed *and* are reviewed and then go through those items that are up for final approval on py-dev on a regular schedule. If it's determined there still needs to be work on any one item, they can clear the specific flags, (needs better tests, clear all test flags, with a suggestion of action), Then when it is fixed and re-reviewed it will come up for final approval again when the next final patch review period comes around. (or sooner if a core developer wants to push it though.) > I understand it's really hard to get involved. It can be frustrating > at times. But I think the best way is to find a mentor. Find an > existing core developer that you relate to and/or have similar > interests with. I've been trying to do this more. > > So here's my offer. Anyone who is serious about becoming a Python > core developer, but doesn't know how to get involved can mail me. > Privately, publicly, it doesn't matter to me. I will try to mentor > you. Cool! Thanks. > Be prepared! I will demand submissions are high quality which at a > minimum means: > > - it adds a new test, enhances an existing test or fixes a broken test > - it has *all* the required documentation, including > versionadded/versionchanged > - most people think the feature is desirable > - the code is maintainable and formatted according to the prevailing style > - we follow the process (which can include improving the process if > others agree) > ie, there's no free lunch, unless you work at Google of course :-) > > It also means you are willing to hold other people up to equally high standards. I wouldn't expect less. > Your contributions don't have to be code though. They can be doc, > they can be PEPs, they can be the python.org website. It could even > be helping out with Google's Summer of Code. The choice is yours. I wonder if a tracker could also have patch requests. Similar to bugs, except it's more of a todo list. Oh, never mind there is a feature request group already. Could that possibly be extended to other areas? _Ron From mlobol at gmail.com Tue Mar 6 21:40:25 2007 From: mlobol at gmail.com (Miguel Lobo) Date: Tue, 6 Mar 2007 21:40:25 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <45ED8C6A.30901@v.loewis.de> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45ED8C6A.30901@v.loewis.de> Message-ID: <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> > > > As I am completely new to CPython development, perhaps this problem has > > already been discussed and/or fixed I may have done something > > incorrectly. Please let me know if that is the case. > > I looked at it briefly. If I understand correctly, the proposed feature > is fine, but lacks a test case. > I have now added a test case. Please let me know if anything else is needed. Thanks, Miguel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070306/abd46e82/attachment.htm From mail at timgolden.me.uk Tue Mar 6 21:18:11 2007 From: mail at timgolden.me.uk (Tim Golden) Date: Tue, 06 Mar 2007 20:18:11 +0000 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: References: <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de><20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de><20070306150459.GA8906@phd.pp.ru> <45ED83A4.80604@v.loewis.de> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> Message-ID: <45EDCC83.9040900@timgolden.me.uk> Terry Reedy wrote: > "Phillip J. Eby" wrote in message > news:5.1.1.6.0.20070306104954.028deec0 at sparrow.telecommunity.com... >> I consider it correct, or at the least, don't think it should be changed, >> as it would make the behavior more difficult to reason about and introduce >> yet another thing to worry about when writing cross-version code. > > Windows did not allow .xxx as a filename in my attempts, so this case seems > irrelevant there. Windows is a bit funny there. You can't create a ".xxx" filename in Explorer, but you can from Console/the API: import os f = open (".xxx", "w") f.close () os.path.isfile (".xxx") # True Not that this helps anything, really, but just to make the point :) TJG From jimjjewett at gmail.com Tue Mar 6 21:53:43 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 6 Mar 2007 15:53:43 -0500 Subject: [Python-Dev] how can I review? [was: Encouraging developers] Message-ID: The 5:1 patch review is a good idea -- but what is the procedure for reviewing a patch? I often comment on patches. Does this count as a review? Would anyone know if it did? If I were going through five at the same time, and I had a sixth to push, I could post here. Normally, I just make a comment on the SF tracker. As far as I know, that makes zero difference to any committer (at least) until they have already decided to look at the issue themselves. At best, I am shaving off a round of back-and-forth, if there would have been one. Sometimes all I say is "What about case X"? The patch isn't ready to be committed yet. It might be that comments are enough, but it isn't ready. I wouldn't want the fact fact that I commented to grab a committer's attention. Sometimes the patch is good, or they deal with all issues.[1] At that point, I ... stop commenting. I don't know of any way (except personal email) to indicate that it was reviewed, let alone approved. [1] Well, all that I noticed at the time -- I don't have a checklist, and there isn't a standard review form. One option would be a designated wiki page listing who reviewed patches when and whether they are ready -- but it seems sort of heavyweight, like it ought to be part of the tracker. I do like Dustin's suggestion (http://mail.python.org/pipermail/python-dev/2007-March/071598.html) If some committers are interested (and tell me how they want the review notification), I would be happy to pre-filter some stdlib patches. If there are several volunteers wanting to split the work, I would be willing to organize the split however the others prefer. -jJ From facundo at taniquetil.com.ar Tue Mar 6 21:56:38 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Tue, 6 Mar 2007 20:56:38 +0000 (UTC) Subject: [Python-Dev] Adding socket timeout to urllib2 References: Message-ID: Guido van Rossum wrote: >> - I'll modify urlopen for it to accept a socket_timeout parameter, >> default to None > > I'd call it timeout. There can't really be much ambiguity can there? Yes and no. For example, if I do a ``urllib2.urlopen("ftp://ftp.myhome.com.ar/blah.txt", timeout=10)``, the timeout is about the socket or about the file transfer? > I say your next step should be to submit a patch, perhaps > incorporating the work that Skip has already done. Ok. > You could also reduce the amount of work to be done initially by only > handling http. At this point I expect that'll cover 99% of all uses. Me too, I suggested ftplib for being cleaner... Thanks. -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From martin at v.loewis.de Tue Mar 6 22:01:38 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 22:01:38 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> References: <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> Message-ID: <45EDD6B2.7060006@v.loewis.de> Phillip J. Eby schrieb: > I know I've written code like this that *depends* on the current > behavior. It's *useful* to classify e.g. .svn directories or .*rc files > by their "extension", so I'm honestly baffled by the idea of wanting to > treat such files as *not* having an extension (as opposed to a > possibly-unrecognized one). I never considered it an extension. Ask 10 people around you to see what a leading dot on Unix in a file name means, and I would be suprised if more than one answered "it separates the file name from the extension". Most of them likely include "hidden file" in their answer, and the rest (of those who know anything about it) will say "dotfiles, not displayed by ls, and not included in globbing". It really is a *different* dot from the one that separates the file name from its extension. I also fail to see why it is useful to invoke splitext on a file name to find out whether it is .svn - I would rather check the file name itself. It's unfortunate, of course, that people apparently relied on this behavior (is the code you are referring to publicly available? If so, where?) Regards, Martin From pje at telecommunity.com Tue Mar 6 22:28:28 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 06 Mar 2007 16:28:28 -0500 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: References: <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <45ED83A4.80604@v.loewis.de> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070306162651.02b6c768@sparrow.telecommunity.com> At 02:55 PM 3/6/2007 -0500, Terry Reedy wrote: >"Phillip J. Eby" wrote in message >news:5.1.1.6.0.20070306104954.028deec0 at sparrow.telecommunity.com... > >I consider it correct, or at the least, don't think it should be changed, > >as it would make the behavior more difficult to reason about and introduce > >yet another thing to worry about when writing cross-version code. > >Windows did not allow .xxx as a filename in my attempts, so this case seems >irrelevant there. Huh? .xyz files work fine on Windows. From martin at v.loewis.de Tue Mar 6 22:27:42 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 22:27:42 +0100 Subject: [Python-Dev] Finding the python library binaries (and docs) In-Reply-To: <87ps7m89gn.fsf@valverde.peloton> References: <87ps7m89gn.fsf@valverde.peloton> Message-ID: <45EDDCCE.6080403@v.loewis.de> David Abrahams schrieb: > I'm trying to find the Python library binaries associated with a given > python executable. This really isn't a python-dev question; please use python-list (news:comp.lang.python) instead. Please take a look at sys.path. > 1. I think the documentation for sys and configure both need some > updating Would you like to work on a patch? This information can be readily obtained from the Python source code. > 2. I'd like to know if there's an officially correct procedure for > finding the library binaries associated with a Python executable. Yes (although I'm not sure what a "library binary" is). Regards, Martin From skip at pobox.com Tue Mar 6 22:40:25 2007 From: skip at pobox.com (skip at pobox.com) Date: Tue, 6 Mar 2007 15:40:25 -0600 Subject: [Python-Dev] how can I review? [was: Encouraging developers] In-Reply-To: References: Message-ID: <17901.57289.617333.258887@montanaro.dyndns.org> Jim> The 5:1 patch review is a good idea -- but what is the procedure Jim> for reviewing a patch? Jim> I often comment on patches. Does this count as a review? Would Jim> anyone know if it did? I believe "review" can mean a few things: * Comments. Reviewing the code does it look reasonable to you given your experience in the space the patch is playing in? Is it missing anything (test cases, documentation, platform dependencies)? * Apply it in your sandbox and try it out. Does the Python test suite pass? Does it work with your applications? * Extend it. If it's missing platform dependencies, test cases or documentation and you can supply (any of) them, feel free to do so. Open a new patch and add a comment to the current tracker item containing a reference to it. (In SourceForge at least you won't be able to attach a file to a tracker item you didn't create or don't own. YMMV once the Rapture occurs and we get to RoundUp nirvana.) Generally, once you've reviewed the five, post a note here referencing them and also referring to the item you would like reviewed. (Personally, if you take a tracker item from "clearly can't be committed as is" to "this is good to go" I think the five review bar should be lowered.) Jim> Sometimes the patch is good, or they deal with all issues.[1] At Jim> that point, I ... stop commenting. "Works for me" and "looks good to me" are also valid comments. ;-) Skip From larry at hastings.org Tue Mar 6 22:43:19 2007 From: larry at hastings.org (Larry Hastings) Date: Tue, 06 Mar 2007 13:43:19 -0800 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45ED83A4.80604@v.loewis.de> References: <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <45ED83A4.80604@v.loewis.de> Message-ID: <45EDE077.2060003@hastings.org> Martin v. L?wis wrote: > Ok - now I'm confused: do you consider this behavior > (splitext('.pythonrc') == ('', '.pythonrc')) correct > or not? > +1 on the behavior. However, the patch is special-casing a leading dot; it would still fail on splitext(".."). If we're gonna fix the bug, I'd rather we fixed it dead. What follows is a cross-platform splitext(). It passes all the tests I threw at it, also attached. If there is interest, I would be happy to submit it as an alternate patch. If runtime speed is paramount, I could lose the portability, and instead port my patch into all three *path.py files natively (dropping os.sep and os.altsep). Hope this helps, /larry/ ---------- import os def splitext(f): sepIndex = f.rfind(os.sep) if os.altsep: altsepIndex = f.rfind(os.altsep) sepIndex = max(sepIndex, altsepIndex) dotIndex = f.rfind(".") if dotIndex > sepIndex: # skip all leading dots filenameIndex = sepIndex + 1 while filenameIndex < dotIndex: if f[filenameIndex] != '.': return f[:dotIndex], f[dotIndex:] filenameIndex += 1 return f, '' def splitextTest(path, filename, ext): assert splitext(path) == (filename, ext) assert splitext(os.sep + path) == (os.sep + filename, ext) assert splitext("abc" + os.sep + path) == ("abc" + os.sep + filename, ext) assert splitext("abc.def" + os.sep + path) == ("abc.def" + os.sep + filename, ext) if os.altsep: assert splitext(os.altsep + path) == (os.altsep + filename, ext) assert splitext("abc" + os.altsep + path) == ("abc" + os.altsep + filename, ext) assert splitext("abc.def" + os.altsep + path) == ("abc.def" + os.altsep + filename, ext) splitextTest("foo.bar", "foo", ".bar") splitextTest("foo.boo.bar", "foo.boo", ".bar") splitextTest("foo.boo.biff.bar", "foo.boo.biff", ".bar") splitextTest(".csh.rc", ".csh", ".rc") splitextTest("nodots", "nodots", "") splitextTest(".cshrc", ".cshrc", "") splitextTest("...manydots", "...manydots", "") splitextTest("...manydots.ext", "...manydots", ".ext") splitextTest(".", ".", "") splitextTest("..", "..", "") splitextTest("........", "........", "") // -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070306/4848d482/attachment-0001.html From guido at python.org Tue Mar 6 22:47:21 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Mar 2007 13:47:21 -0800 Subject: [Python-Dev] Adding socket timeout to urllib2 In-Reply-To: References: Message-ID: On 3/6/07, Facundo Batista wrote: > Guido van Rossum wrote: > > >> - I'll modify urlopen for it to accept a socket_timeout parameter, > >> default to None > > > > I'd call it timeout. There can't really be much ambiguity can there? > > Yes and no. For example, if I do a > ``urllib2.urlopen("ftp://ftp.myhome.com.ar/blah.txt", timeout=10)``, the > timeout is about the socket or about the file transfer? Think of it this way. "Timeout" doesn't mean the whole thing needs to be completed in 10 secs. It means that over 10 secs of no activity causes it to be aborted. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Tue Mar 6 22:51:00 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 22:51:00 +0100 Subject: [Python-Dev] how can I review? [was: Encouraging developers] In-Reply-To: References: Message-ID: <45EDE244.7080404@v.loewis.de> Jim Jewett schrieb: > The 5:1 patch review is a good idea -- but what is the procedure for > reviewing a patch? > > I often comment on patches. Does this count as a review? Sure. Ideally, a review should bring the patch to an "accept-or-reject" state, i.e. it should lead to a recommendation to the committer. An explanation should include the reasoning done (eg. for accept: useful feature, complete implementation, for reject: patch is incomplete, has undesirable side effects). However, in that 5:1 deal, any kind of review was good in the past, even though it doesn't lead to a decision. > Would anyone know if it did? I often see your reviews (thanks!), but I don't keep track. > If I were going through five at the same time, and I had a sixth to > push, I could post here. Normally, I just make a comment on the SF > tracker. As far as I know, that makes zero difference to any > committer (at least) until they have already decided to look at the > issue themselves. At best, I am shaving off a round of > back-and-forth, if there would have been one. Indeed. That is already useful, although it might help more if you stated a recommendation (I know you do in many cases). If you have a list of patches that you think are ready for decision, please post the list here, listing clear accepts and clear rejects. For uncertain cases, you can try to start a discussion; make your lay out pros and cons and explain what side you are leaning to. > Sometimes all I say is "What about case X"? The patch isn't ready to > be committed yet. It might be that comments are enough, but it isn't > ready. I wouldn't want the fact fact that I commented to grab a > committer's attention. If you think the patch is not complete, state so: "It is not complete because it doesn't support case X". If the submitter doesn't respond, you can consider revising it yourself, if you think the patch is important, or recommend rejection (if you think it isn't that relevant). > Sometimes the patch is good, or they deal with all issues.[1] At that > point, I ... stop commenting. I don't know of any way (except > personal email) to indicate that it was reviewed, let alone approved. For the record, state your position in a comment. It is important to record what position people have on patches they reviewed. If they merely review, and then don't communicate their findings, it is wasted time. > One option would be a designated wiki page listing who reviewed > patches when and whether they are ready -- but it seems sort of > heavyweight, like it ought to be part of the tracker. I agree. Maybe we should add some "reviewed by" field to the roundup tracker where reviewers can record that they brought the patch to a state ready for committing/final rejection (I think a second check is still needed, as the case of splitext shows: you were in favor of rejecting it because of the incompatibility, but it looks like the majority of users is in favor of accepting because they consider the current behavior buggy). > If some committers are interested (and tell me how they want the > review notification), I would be happy to pre-filter some stdlib > patches. If there are several volunteers wanting to split the work, I > would be willing to organize the split however the others prefer. Not sure how you are organizing this work: If you have completed a review (i.e. so that just some technical action would need to be taken), feel free to post to python-dev (ideally with a python.org/sf/number link so people can easily follow your analysis), with a subject like "recommend for rejection/acceptance". The tricky ones are really the incomplete ones: if neither the submitter nor you yourself feel like completing the patch, posting to comp.lang.python might also reveal some contributors. Regards, Martin From martin at v.loewis.de Tue Mar 6 22:53:29 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 22:53:29 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45EDE077.2060003@hastings.org> References: <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <45ED83A4.80604@v.loewis.de> <45EDE077.2060003@hastings.org> Message-ID: <45EDE2D9.4030703@v.loewis.de> Larry Hastings schrieb: > Hope this helps, Indeed it does! After all this discussion, a documentation clarification is certainly in order, but I can work that out myself. Thanks, Martin From dave at boost-consulting.com Tue Mar 6 22:56:36 2007 From: dave at boost-consulting.com (David Abrahams) Date: Tue, 06 Mar 2007 16:56:36 -0500 Subject: [Python-Dev] Finding the python library binaries (and docs) References: <87ps7m89gn.fsf@valverde.peloton> <45EDDCCE.6080403@v.loewis.de> Message-ID: <873b4i8297.fsf@valverde.peloton> on Tue Mar 06 2007, "Martin v. L?wis" wrote: > David Abrahams schrieb: >> I'm trying to find the Python library binaries associated with a given >> python executable. > > This really isn't a python-dev question; please use python-list > (news:comp.lang.python) instead. I wrestled with the right list for this one and determined that only the python devs would know the answers. Also I intended to propose that the information I'm looking for be added to sys as a standard attribute. > Please take a look at sys.path. No help at all; that is the module search path (i.e. for .py files), not for the Python library binaries. >> 1. I think the documentation for sys and configure both need some >> updating > > Would you like to work on a patch? This information can be readily > obtained from the Python source code. I'll consider it, once we get the original intention cleared up. There are lots of ways to interpret what the code actually does, and documentation that merely transcribes the code's logic will not be very useful. >> 2. I'd like to know if there's an officially correct procedure for >> finding the library binaries associated with a Python executable. > > Yes (although I'm not sure what a "library binary" is). I gave one example in my post: libpython25.a -- Dave Abrahams Boost Consulting www.boost-consulting.com From martin at v.loewis.de Tue Mar 6 23:02:33 2007 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 23:02:33 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de><200703060818.29114.phil@riverbankcomputing.co.uk> <45ED3506.9040707@v.loewis.de> Message-ID: <45EDE4F9.2020902@v.loewis.de> Terry Reedy schrieb: > It would also be helpful if the new tracker system could produce a list of > module-specific open items sorted by module, since that would indicate > modules needing attention, and I could look for a batch that were > unassigned. The new tracker will initially have the same "category" as the current one, but it will have the ability to group by them (and perhaps even count them). Of course, many bugs don't have a category set, so some volunteer would first need to go through all open bugs and categorize them. If, for "Modules", you want a more fine-grained classification, it would be possible to add new categories, or add another field "affected modules" (multi-list, I assume). If there is a volunteer that would like to go through all bug reports and classify them according this finer category, I can work with the roundup maintainers to add that (although it will likely only happen after the switch to roundup - it is easy to extend the schema with additional fields if it is known exactly what they are). There is also a generic "keywords" field in the roundup tracker - perhaps the "affected modules" could become keywords. HTH, Martin From pje at telecommunity.com Tue Mar 6 22:39:33 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 06 Mar 2007 16:39:33 -0500 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45EDD6B2.7060006@v.loewis.de> References: <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070306162907.02b7e2a0@sparrow.telecommunity.com> At 10:01 PM 3/6/2007 +0100, Martin v. L?wis wrote: >It's unfortunate, of course, that people apparently relied on >this behavior I was going to say it's the *documented* behavior, but I see that the documentation is actually such that it could be interpreted either way. However, since it's not documented more specifically, it seems perfectly reasonable to rely on the implementation's behavior to resolve the ambiguity. >(is the code you are referring to publicly available? No, and I don't know if it's still being used, although it probably is. But it might also still be running on Python 2.2, and I'm not sure if any of its actual uses are actually affected by such dotted files, or whether the code will ever be upgraded to more recent Python versions. I haven't worked at the company in question for a few years now. From martin at v.loewis.de Tue Mar 6 23:12:43 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 06 Mar 2007 23:12:43 +0100 Subject: [Python-Dev] Finding the python library binaries (and docs) In-Reply-To: <873b4i8297.fsf@valverde.peloton> References: <87ps7m89gn.fsf@valverde.peloton> <45EDDCCE.6080403@v.loewis.de> <873b4i8297.fsf@valverde.peloton> Message-ID: <45EDE75B.1010205@v.loewis.de> David Abrahams schrieb: > on Tue Mar 06 2007, "Martin v. L?wis" wrote: >> David Abrahams schrieb: >>> I'm trying to find the Python library binaries associated with a given >>> python executable. >> This really isn't a python-dev question; please use python-list >> (news:comp.lang.python) instead. > > I wrestled with the right list for this one and determined that only > the python devs would know the answers. That absolutely cannot be the case. Python is open source, you have *everything* you need to answer this question. >>> 2. I'd like to know if there's an officially correct procedure for >>> finding the library binaries associated with a Python executable. >> Yes (although I'm not sure what a "library binary" is). > > I gave one example in my post: libpython25.a Ah, ok. If you want to know where that is installed (officially), check out what "make install" does. Also, ask yourself whether you know a Python module that should know how to find it. distutils and freeze come to mind. HTH, Martin From guido at python.org Tue Mar 6 23:17:50 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Mar 2007 14:17:50 -0800 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> Message-ID: On 3/5/07, Facundo Batista wrote: > Thomas Wouters wrote: > > > developers and people who develop their own software. I would like to hear > > from people who have concrete doubts about this upgrade path. I don't mean > > Disclaimer: I'm not involved in Py3k, and not even tried it once. And > don't know the details of the tool to "transform" Py2 to Py3. > > Having said that, the *only* fear I have, is how safe it will be. And > here comes the explanation. > > I'm now in an enviroment where we rely on Python, and I advocates on it > everyday. We use 2.3, and 2.4, some servers we're about to deploy have > 2.5, and we'll use the lastest Py2.x everytime we deploy a new machine. > > But, as we have always running behind our ToDo, we don't have time to > say "Ok, this server runs 2.3, how can I start using 2.5?". Too many > servers, too many applications, too many > undocumented-and-nobody-knows-about-it applications. And they all are > applications running 7x24. I know the feeling. Google still uses Python 2.2 for most apps. We're supporting 2.4 as well, and hope to have the last apps migrated to 2.4 in a year or so. *Then* we'll start supporting a higher version -- probably 2.6 by then. > And Py3k is around the corner, and I even heard some guys saying "If I'd > spent time taking care that this app runs ok after changing a bit of > it, I'll wait to Python 3000". In your kind of env it's clearly too early to think about 3.0. You shouldn't expect to be switching until a year after 3.0 is released, probably. > So, the enviroment is explained, now I go with "how safe it will be". > I'd love to know that there'll be a tool that tells me, after running it > over my application, Your app is "ready for migration"'. I do *not* care > if it takes a lot of work to make the proper changes, but I need to feel > confident that after running the last changed version during a week in, > say, Py2.7, and no warnings appear, and the "verification tool" say > green light, I can start running it in Py3k. And look at it a week more. > And say "life is great", :) You need comperhensive unit tests too. Then: *If* you get the green light from 2.6's py3k warning mode, *and* the conversion tool produces syntactically correct code without issuing warnings, *and* your unit tests all pass, *then* I expect you'll be in about the same situation as for any version upgrade in such an environment (e.g. 2.4 -> 2.5). I.e. you need to do a big integration test and once that passes you should be set for a stress-free transition. I think that's the best we can hope for. I plan to do it this way at Google too. > Anyway, I know how hard is it, and I regret not having the time I'd love > to have to help you. All I can do is thank you. > > Thank you very much! You're welcome! > > One thing in particular I wonder about is the warning about mixing tabs and > > spaces. Should it be in category 2) (on by default) or 3) (still off by > > default, but part of -Wpy3k)? > > For me, I'd love to 2.6 start warning "you're mixing tab and spaces, > shame yourself!". (The English say "shame on you" :-) While I'd like to do this too I don't know how many folks there are out there who have no way to suppress the warning (because they're end users of a script someone else wrote for them). I think it may be more of a development warning, so I think category is appropriate. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Tue Mar 6 23:18:01 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Mar 2007 23:18:01 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <5.1.1.6.0.20070306162907.02b7e2a0@sparrow.telecommunity.com> References: <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <5.1.1.6.0.20070306162907.02b7e2a0@sparrow.telecommunity.com> Message-ID: <45EDE899.9020709@v.loewis.de> Phillip J. Eby schrieb: > At 10:01 PM 3/6/2007 +0100, Martin v. L?wis wrote: >> It's unfortunate, of course, that people apparently relied on >> this behavior > > I was going to say it's the *documented* behavior, but I see that the > documentation is actually such that it could be interpreted either way. > > However, since it's not documented more specifically, it seems perfectly > reasonable to rely on the implementation's behavior to resolve the ambiguity. Sure, it is an incompatible change, no doubt. However, incompatible changes are "ok" for feature releases (not so fo bugfix releases). So this being an incompatible change alone is not a reason to reject the patch. Significant breakage in many applications might be, but I don't expect that for this change (which is really tiny). Regards, Martin From tjreedy at udel.edu Tue Mar 6 23:17:48 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 6 Mar 2007 17:17:48 -0500 Subject: [Python-Dev] splitext('.cshrc') References: <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de><20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de><20070306150459.GA8906@phd.pp.ru> <45ED83A4.80604@v.loewis.de><5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <5.1.1.6.0.20070306162651.02b6c768@sparrow.telecommunity.com> Message-ID: "Phillip J. Eby" wrote in message news:5.1.1.6.0.20070306162651.02b6c768 at sparrow.telecommunity.com... | >Windows did not allow .xxx as a filename in my attempts, so this case seems | >irrelevant there. | | Huh? .xyz files work fine on Windows. Tim G. explained that Explorer, which I tried, is for whatever reason stricter that other interfaces. tjr From guido at python.org Tue Mar 6 23:22:39 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Mar 2007 14:22:39 -0800 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: <20070228062840.17852.238152490.divmod.quotient.727@ohm> References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> <20070228062840.17852.238152490.divmod.quotient.727@ohm> Message-ID: On 2/27/07, glyph at divmod.com wrote: > 2to3 should take great care _tell_ you when it fails. One concern I have is that the source translation may subtly alter the *semantics* of unit test code, so that the tests are no longer effective and do not provide adequate coverage. I think this is an extremely unlikely edge case, but a very dangerous one in the event that it does happen. I don't know of any bugs in 2to3 that will do this at the moment, but then it's hardly the final release. Actually, this did happen with the iterkeys -> keys translation. A few unittests were reduced to pure nonsense by this. Most of those failed, but it was definitely a hassle finding out what went wrong (in part because I was too greedy and checked in a bunch of converted code without any review at all). Lesson learned I would say. Fortunately, this is a case where 2.6's Py3k warning mode should compensate. > Also, the limitations of 2to3 should be very well documented, so that when it does tell you how it has failed, it's clear to the developer what changes to make to the *2.6 source* to cause it to translate correctly, not how to fix up the translated 3.0 code. I'm hoping Collin will continue his excellent work on 2to3. Hopefully he'll get help from others in writing docs aimed at teaching the c.l.py crowd how to use it and what to expect. > Thanks *very* much to all the python core developers and community members who have been working to make this happen, especially to Neal, Thomas, and Guido for listening intently to all of our concerns at PyCon. You're welcome. I believe I even got a beer out of it. ;-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Tue Mar 6 23:30:02 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 07 Mar 2007 11:30:02 +1300 Subject: [Python-Dev] locals(), closures, and IronPython... In-Reply-To: References: <7AD436E4270DD54A94238001769C22276A624C7482@DF-GRTDANE-MSG.exchange.corp.microsoft.com> Message-ID: <45EDEB6A.1000409@canterbury.ac.nz> Andrew Dalke wrote: > def __init__(self, prec=None, rounding=None, > traps=None, flags=None, > _rounding_decision=None, > Emin=None, Emax=None, > capitals=None, _clamp=0, > _ignored_flags=None): > ... > for name, val in locals().items(): > if val is None: > setattr(self, name, _copy.copy(getattr(DefaultContext, name))) > else: > setattr(self, name, val) Hmmm. For things like that, maybe what you really want is to be able to give the ** arg a default value: def __init__(self, **kwds = dict(rec=None, rounding=None, traps=None, flags=None, _rounding_decision=None, Emin=None, Emax=None, capitals=None, _clamp=0, _ignored_flags=None)) for name, val in kwds: ... Although you can get a similar effect now by doing def __init__(self, **kwds): args = dict(prec=None, rounding=None, traps=None, flags=None, _rounding_decision=None, Emin=None, Emax=None, capitals=None, _clamp=0, _ignored_flags=None) args.update(kwds) for name, value in args: ... So, no need for locals() here. -- Greg From tjreedy at udel.edu Tue Mar 6 23:49:24 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 6 Mar 2007 17:49:24 -0500 Subject: [Python-Dev] Encouraging developers References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de><200703060818.29114.phil@riverbankcomputing.co.uk> <45ED3506.9040707@v.loewis.de> <45EDE4F9.2020902@v.loewis.de> Message-ID: ""Martin v. L?wis"" wrote in message news:45EDE4F9.2020902 at v.loewis.de... | Terry Reedy schrieb: | > It would also be helpful if the new tracker system could produce a list of | > module-specific open items sorted by module, since that would indicate | > modules needing attention, and I could look for a batch that were | > unassigned. | | The new tracker will initially have the same "category" as the current | one, but it will have the ability to group by them (and perhaps even | count them). | | Of course, many bugs don't have a category set, so some volunteer would | first need to go through all open bugs and categorize them. I'll take a look at the categories on SF and see if any are unclear to me. | If, for "Modules", you want a more fine-grained classification, it | would be possible to add new categories, or add another field | "affected modules" (multi-list, I assume). I guess making each module a category would be too fine... | | If there is a volunteer that would like to go through all bug | reports and classify them according this finer category, I can | work with the roundup maintainers to add that (although it | will likely only happen after the switch to roundup - it is | easy to extend the schema with additional fields if it is | known exactly what they are). | | There is also a generic "keywords" field in the roundup | tracker - perhaps the "affected modules" could become keywords. I am presuming that it will be easier to write scripts to search and manipulate on the new tracker (whether to run on the tracker site or elsewhere). If so, then the information just needs to be recorded somewhere. An existing keywords field sounds good enough. Terry Jan Reedy From barry at python.org Wed Mar 7 00:10:06 2007 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Mar 2007 18:10:06 -0500 Subject: [Python-Dev] PEP 364, Transitioning to the Py3K standard library Message-ID: <6EBA9F60-ECBF-426B-AC60-E1044AEDE938@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Here's a new PEP that's the outgrowth of work Brett Cannon and I did at PyCon. Basically we were looking for a way to allow for forward compatibility with PEP 3108, which describes a reorganization of the standard library. PEP 364 is for Python 2.x -- it doesn't say what the reorganized standard library should look like (that's still for PEP 3108 to decide), but it provides a mechanism whereby for renamed modules, both the old and new module names can live side-by-side. Interestingly enough, I was able to remove the one-off hack in the email package for its PEP 8-compliant renaming by using the code in the reference implementation for PEP 364. Here's the text of the PEP, which should be available at http:// www.python.org/dev/peps/pep-0364/ shortly. - -Barry PEP: 364 Title: Transitioning to the Py3K Standard Library Version: $Revision$ Last-Modified: $Date$ Author: Barry A. Warsaw Status: Active Type: Standards Track Content-Type: text/x-rst Created: 01-Mar-2007 Python-Version: 2.6 Post-History: Abstract ======== PEP 3108 describes the reorganization of the Python standard library for the Python 3.0 release [1]_. This PEP describes a mechanism for transitioning from the Python 2.x standard library to the Python 3.0 standard library. This transition will allow and encourage Python programmers to use the new Python 3.0 library names starting with Python 2.6, while maintaining the old names for backward compatibility. In this way, a Python programmer will be able to write forward compatible code without sacrificing interoperability with existing Python programs. Rationale ========= PEP 3108 presents a rationale for Python standard library (stdlib) reorganization. The reader is encouraged to consult that PEP for details about why and how the library will be reorganized. Should PEP 3108 be accepted in part or in whole, then it is advantageous to allow Python programmers to begin the transition to the new stdlib module names in Python 2.x, so that they can write forward compatible code starting with Python 2.6. Note that PEP 3108 proposes to remove some "silly old stuff", i.e. modules that are no longer useful or necessary. The PEP you are reading does not address this because there are no forward compatibility issues for modules that are to be removed, except to stop using such modules. This PEP concerns only the mechanism by which mappings from old stdlib names to new stdlib names are maintained. Please consult PEP 3108 for all specific module renaming proposals. Specifically see the section titled ``Modules to Rename`` for guidelines on the old name to new name mappings. The few examples in this PEP are given for illustrative purposes only and should not be used for specific renaming recommendations. Supported Renamings =================== There are at least 4 use cases explicitly supported by this PEP: - - Simple top-level package name renamings, such as ``StringIO`` to ``stringio``; - - Sub-package renamings where the package name may or may not be renamed, such as ``email.MIMEText`` to ``email.mime.text``; - - Extension module renaming, such as ``cStringIO`` to ``cstringio``; - - Third party renaming of any of the above. Two use cases supported by this PEP include renaming simple top-level modules, such as ``StringIO``, as well as modules within packages, such as ``email.MIMEText``. In the former case, PEP 3108 currently recommends ``StringIO`` be renamed to ``stringio``, following PEP 8 recommendations [2]_. In the latter case, the email 4.0 package distributed with Python 2.5 already renamed ``email.MIMEText`` to ``email.mime.text``, although it did so in a one-off, uniquely hackish way inside the email package. The mechanism described in this PEP is general enough to handle all module renamings, obviating the need for the Python 2.5 hack (except for backward compatibility with earlier Python versions). An additional use case is to support the renaming of C extension modules. As long as the new name for the C module is importable, it can be remapped to the new name. E.g. ``cStringIO`` renamed to ``cstringio``. Third party package renaming is also supported, via several public interfaces accessible by any Python module. Remappings are not performed recursively. .mv files ========= Remapping files are called ``.mv`` files; the suffix was chosen to be evocative of the Unix mv(1) command. An ``.mv`` file is a simple line-oriented text file. All blank lines and lines that start with a # are ignored. All other lines must contain two whitespace separated fields. The first field is the old module name, and the second field is the new module name. Both module names must be specified using their full dotted-path names. Here is an example ``.mv`` file from Python 2.6:: # Map the various string i/o libraries to their new names StringIO stringio cStringIO cstringio ``.mv`` files can appear anywhere in the file system, and there is a programmatic interface provided to parse them, and register the remappings inside them. By default, when Python starts up, all the ``.mv`` files in the ``oldlib`` package are read, and their remappings are automatically registered. This is where all the module remappings should be specified for top-level Python 2.x standard library modules. Implementation Specification ============================ This section provides the full specification for how module renamings in Python 2.x are implemented. The central mechanism relies on various import hooks as described in PEP 302 [3]_. Specifically ``sys.path_importer_cache``, ``sys.path``, and ``sys.meta_path`` are all employed to provide the necessary functionality. When Python's import machinery is initialized, the oldlib package is imported. Inside oldlib there is a class called ``OldStdlibLoader``. This class implements the PEP 302 interface and is automatically instantiated, with zero arguments. The constructor reads all the ``.mv`` files from the oldlib package directory, automatically registering all the remappings found in those ``.mv`` files. This is how the Python 2.x standard library is remapped. The OldStdlibLoader class should not be instantiated by other Python modules. Instead, you can access the global OldStdlibLoader instance via the ``sys.stdlib_remapper`` instance. Use this instance if you want programmatic access to the remapping machinery. One important implementation detail: as needed by the PEP 302 API, a magic string is added to sys.path, and module __path__ attributes in order to hook in our remapping loader. This magic string is currently ```` and some changes were necessary to Python's site.py file in order to treat all sys.path entries starting with ``<`` as special. Specifically, no attempt is made to make them absolute file names (since they aren't file names at all). In order for the remapping import hooks to work, the module or package must be physically located under its new name. This is because the import hooks catch only modules that are not already imported, and cannot be imported by Python's built-in import rules. Thus, if a module has been moved, say from Lib/StringIO.py to Lib/stringio.py, and the former's ``.pyc`` file has been removed, then without the remapper, this would fail:: import StringIO Instead, with the remapper, this failing import will be caught, the old name will be looked up in the registered remappings, and in this case, the new name ``stringio`` will be found. The remapper then attempts to import the new name, and if that succeeds, it binds the resulting module into sys.modules, under both the old and new names. Thus, the above import will result in entries in sys.modules for 'StringIO' and 'stringio', and both will point to the exact same module object. Note that no way to disable the remapping machinery is proposed, short of moving all the ``.mv`` files away or programmatically removing them in some custom start up code. In Python 3.0, the remappings will be eliminated, leaving only the "new" names. Programmatic Interface ====================== Several methods are added to the ``sys.stdlib_remapper`` object, which third party packages can use to register their own remappings. Note however that in all cases, there is one and only one mapping from an old name to a new name. If two ``.mv`` files contain different mappings for an old name, or if a programmatic call is made with an old name that is already remapped, the previous mapping is lost. This will not affect any already imported modules. The following methods are available on the ``sys.stdlib_remapper`` object: - - ``read_mv_file(filename)`` -- Read the given file and register all remappings found in the file. - - ``read_directory_mv_files(dirname, suffix='.mv')`` -- List the given directory, reading all files in that directory that have the matching suffix (``.mv`` by default). For each parsed file, register all the remappings found in that file. - - ``set_mapping(oldname, newname)`` -- Register a new mapping from an old module name to a new module name. Both must be the full dotted-path name to the module. newname may be ``None`` in which case any existing mapping for oldname will be removed (it is not an error if there is no existing mapping). - - ``get_mapping(oldname, default=None)`` -- Return any registered newname for the given oldname. If there is no registered remapping, default is returned. Open Issues =========== - Should there be a command line switch and/or environment variable to disable all remappings? - Should remappings occur recursively? - Should we automatically parse package directories for .mv files when the package's __init__.py is loaded? This would allow packages to easily include .mv files for their own remappings. Compare what the email package currently has to do if we place its ``.mv`` file in the email package instead of in the oldlib package:: # Expose old names import os, sys sys.stdlib_remapper.read_directory_mv_files(os.path.dirname (__file__)) I think we should automatically read a package's directory for any ``.mv`` files it might contain. Reference Implementation ======================== A reference implementation, in the form of a patch against the current (as of this writing) state of the Python 2.6 svn trunk, is available as SourceForge patch #1675334 [4]_. Note that this patch includes a rename of ``cStringIO`` to ``cstringio``, but this is primarily for illustrative and unit testing purposes. Should the patch be accepted, we might want to split this change off into other PEP 3108 changes. References ========== .. [1] PEP 3108, Standard Library Reorganization, Cannon (http://www.python.org/dev/peps/pep-3108) .. [2] PEP 8, Style Guide for Python Code, GvR, Warsaw (http://www.python.org/dev/peps/pep-0008) .. [3] PEP 302, New Import Hooks, JvR, Moore (http://www.python.org/dev/peps/pep-0302) .. [4] Reference implementation on SourceForge (https://sourceforge.net/tracker/index.php? func=detail&aid=1675334&group_id=5470&atid=305470) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRe30z3EjvBPtnXfVAQJkVwP9Fv5NgvyBJZH5/mU0H73dPJnGTLLOP0F/ MxGfrRR8qMdHITOSs6bNUZGtwDWwG5A1Aw+ksxqDaYOL0sdR7gTHAZqaH1aCGn1H n/m4wSOzQE3NgBFF+Dv8j1A+8C+l6CWhc/ZMlHRRdtxqSYQCjae8XaxHaO94E50Q ktyPb4LhOdg= =Vd2Z -----END PGP SIGNATURE----- From terry at jon.es Wed Mar 7 00:16:54 2007 From: terry at jon.es (Terry Jones) Date: Wed, 7 Mar 2007 00:16:54 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: Your message at 16:39:33 on Tuesday, 6 March 2007 References: <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <5.1.1.6.0.20070306162907.02b7e2a0@sparrow.telecommunity.com> Message-ID: <17901.63078.595584.18386@terry-jones-computer.local> I think there are various good arguments that the current behavior of splitext isn't optimal. But..... these don't feel strong enough to me to break existing code or to force people who happen to be in the know to go hunt down and review old code etc. I don't see the point in doing that, just to fix something that everyone seems to agree is quite marginal. There are about 500 hits for splitext\(lang:python at code.google.com. Many of these blindly use something like newname = splitext(filename)[0] + '.bak' You could convincingly argue that these would be "fixed" if the current behavior were changed. In fact, from looking at 30 examples or so, I suspect much more code would be fixed than broken. But changing the behavior of programs doesn't seem like a good idea - even if you can claim to have fixed them. (This reminds me of SUN adopting a newer malloc in the mid-90s - it had better internal fragmentation behavior for many requests, and thereby broke a bunch of working (but in fact buggy) code that was inadvertently relying on the slop in the older malloc). I do think the behavior can be improved, and that it should be fixed, but at a place where other incompatible changes will also be being made, and where people are already going to have to do serious code review. To me that argues for fixing it in Py3k. Arguing that it's just a minor change, and so therefore it can go straight in, seems to also lend support to the suggestion that it can wait. Terry From greg.ewing at canterbury.ac.nz Wed Mar 7 00:18:49 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 07 Mar 2007 12:18:49 +1300 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <9613db600703060944v17c4c372n151a58aeb801a853@mail.gmail.com> References: <45ED6033.2020807@v.loewis.de> <200703061444.52832.meine@informatik.uni-hamburg.de> <9613db600703060944v17c4c372n151a58aeb801a853@mail.gmail.com> Message-ID: <45EDF6D9.3010205@canterbury.ac.nz> Tim Lesher wrote: > FWIW, all of the "standard" Windows functions from the Microsoft CRT > (_splitpath) to the Shell API (PathRemoveExtension) to the CLR > (System.IO.Path.*) believe that ".cshrc" is the extension of the > filename ".cshrc". > > I'm not sure if that's an argument for or against the patch, though. It *might* be an argument for making the behaviour platform-dependent. -- Greg From mike.klaas at gmail.com Wed Mar 7 00:23:11 2007 From: mike.klaas at gmail.com (Mike Klaas) Date: Tue, 6 Mar 2007 15:23:11 -0800 Subject: [Python-Dev] locals(), closures, and IronPython... In-Reply-To: <45EDEB6A.1000409@canterbury.ac.nz> References: <7AD436E4270DD54A94238001769C22276A624C7482@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <45EDEB6A.1000409@canterbury.ac.nz> Message-ID: <3d2ce8cb0703061523j438783eeu5eb4df719cb3840@mail.gmail.com> On 3/6/07, Greg Ewing wrote: > Although you can get a similar effect now by doing > > def __init__(self, **kwds): > args = dict(prec=None, rounding=None, > traps=None, flags=None, > _rounding_decision=None, > Emin=None, Emax=None, > capitals=None, _clamp=0, > _ignored_flags=None) > args.update(kwds) > for name, value in args: > ... > > So, no need for locals() here. Yes, that is the obvious approach. But it is painful to abandon the introspectable signature. There's nothing quite like running help(func) and getting *args, **kwargs as the documented parameter list. -Mike From glyph at divmod.com Wed Mar 7 00:39:46 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Tue, 06 Mar 2007 23:39:46 -0000 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45EDE899.9020709@v.loewis.de> References: <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <5.1.1.6.0.20070306162907.02b7e2a0@sparrow.telecommunity.com> <45EDE899.9020709@v.loewis.de> Message-ID: <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> On 10:18 pm, martin at v.loewis.de wrote: >Phillip J. Eby schrieb: >>At 10:01 PM 3/6/2007 +0100, Martin v. L?wis wrote: >>>It's unfortunate, of course, that people apparently relied on >>>this behavior >> >>I was going to say it's the *documented* behavior, but I see that the >>documentation is actually such that it could be interpreted either >>way. >> >>However, since it's not documented more specifically, it seems >>perfectly >>reasonable to rely on the implementation's behavior to resolve the >>ambiguity. Despite the generally quite good documentation, I've learned about how quite a lot of how the standard library works by messing around at the interactive interpreter. This is one cost of the "incompatibility" - having to re-train developers who thought they knew how something worked, or are continuing to write new software while experimenting with older VMs. >Sure, it is an incompatible change, no doubt. However, incompatible >changes are "ok" for feature releases (not so fo bugfix releases). Incompatible changes may be *acceptable* for feature releases, but that doesn't mean they are desirable. The cost of incompatibility should be considered for every change. This cost is particularly bad when the incompatibility is of the "silent breakage" variety - the change being discussed here would not be the sort of thing that one could, say, produce a warning about or gently deprecate; anything relying on the old behavior would suddenly be incorrect, and any software wishing to straddle the 2.5<->2.6 boundary would be better off just implementing their own splitext() than relying on the stdlib. >So this being an incompatible change alone is not a reason to reject >the patch. Significant breakage in many applications might be, but >I don't expect that for this change (which is really tiny). Software is a chaotic system. The size of the change is unrelated to how badly it might break things. More to the point, we know the cost, what's the benefit? Is there any sort of bug that it is likely to prevent in *new* code? It clearly isn't going to help any old code. It seems there are people who see it both ways, and I haven't seen anything compelling to indicate that either behavior is particularly less surprising in the edge case. In cases like this, historical context should be considered, even for a major overhaul like 3.0. Of course, if the newly proposed semantics provided a solution to a real problem or common error, compatibility might be determined to be a less important issue. The use-cases being discussed here would be better served by having new APIs that do particular things and don't change existing semantics, though. For example, a "guess_mime_type(path)" function which could examine a path and figure out its mime type based on its extension (among other things). -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070306/dab58ede/attachment.html From glyph at divmod.com Wed Mar 7 00:48:49 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Tue, 06 Mar 2007 23:48:49 -0000 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> <20070228062840.17852.238152490.divmod.quotient.727@ohm> Message-ID: <20070306234849.6305.1580733783.divmod.xquotient.44@joule.divmod.com> On 10:22 pm, guido at python.org wrote: >On 2/27/07, glyph at divmod.com wrote: >>2to3 should take great care _tell_ you when it fails. One concern I >>have is that the source translation may subtly alter the *semantics* >>of unit test code, so that the tests are no longer effective and do >>not provide adequate coverage. >Actually, this did happen with the iterkeys -> keys translation. A few >unittests were reduced to pure nonsense by this. Most of those failed, >but it was definitely a hassle finding out what went wrong (in part >because I was too greedy and checked in a bunch of converted code >without any review at all). Lesson learned I would say. Fortunately, >this is a case where 2.6's Py3k warning mode should compensate. Was this due to a bug that can be fixed, or an inherent problem? I could imagine, thanks to Python's dynamism, there would be edge cases that are impossible to detect completely reliably. If that's the case it would be good to at least have pedantic warnings from 2to3 alerting the user to places where translation could possibly be doing something semantically dodgy. Ideally before jumping the chasm with Twisted I'd like to make sure that all of that sort of warning was gone, in _addition_ to a clean test run. >>Also, the limitations of 2to3 should be very well documented, so that >>when it does tell you how it has failed, it's clear to the developer >>what changes to make to the *2.6 source* to cause it to translate >>correctly, not how to fix up the translated 3.0 code. > >I'm hoping Collin will continue his excellent work on 2to3. Hopefully >he'll get help from others in writing docs aimed at teaching the >c.l.py crowd how to use it and what to expect. I'm sure he'll hear from me if anything goes wrong with it :). >>Thanks *very* much to all the python core developers and community >>members who have been working to make this happen, especially to Neal, >>Thomas, and Guido for listening intently to all of our concerns at >>PyCon. > >You're welcome. I believe I even got a beer out of it. ;-) Well deserved! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070306/f589c1a7/attachment.htm From brett at python.org Wed Mar 7 01:09:58 2007 From: brett at python.org (Brett Cannon) Date: Tue, 6 Mar 2007 16:09:58 -0800 Subject: [Python-Dev] PEP 364, Transitioning to the Py3K standard library In-Reply-To: <6EBA9F60-ECBF-426B-AC60-E1044AEDE938@python.org> References: <6EBA9F60-ECBF-426B-AC60-E1044AEDE938@python.org> Message-ID: On 3/6/07, Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Here's a new PEP that's the outgrowth of work Brett Cannon and I did > at PyCon. Basically we were looking for a way to allow for forward > compatibility with PEP 3108, which describes a reorganization of the > standard library. PEP 364 is for Python 2.x -- it doesn't say what > the reorganized standard library should look like (that's still for > PEP 3108 to decide), but it provides a mechanism whereby for renamed > modules, both the old and new module names can live side-by-side. > > Interestingly enough, I was able to remove the one-off hack in the > email package for its PEP 8-compliant renaming by using the code in > the reference implementation for PEP 364. > > Here's the text of the PEP, which should be available at http:// > www.python.org/dev/peps/pep-0364/ shortly. > > - -Barry > > PEP: 364 > Title: Transitioning to the Py3K Standard Library > Version: $Revision$ > Last-Modified: $Date$ > Author: Barry A. Warsaw > Status: Active > Type: Standards Track > Content-Type: text/x-rst > Created: 01-Mar-2007 > Python-Version: 2.6 > Post-History: > > > Abstract > ======== > > PEP 3108 describes the reorganization of the Python standard library > for the Python 3.0 release [1]_. This PEP describes a > mechanism for transitioning from the Python 2.x standard library to > the Python 3.0 standard library. This transition will allow and > encourage Python programmers to use the new Python 3.0 library names > starting with Python 2.6, while maintaining the old names for backward > compatibility. In this way, a Python programmer will be able to write > forward compatible code without sacrificing interoperability with > existing Python programs. > > > Rationale > ========= > > PEP 3108 presents a rationale for Python standard library (stdlib) > reorganization. The reader is encouraged to consult that PEP for > details about why and how the library will be reorganized. Should > PEP 3108 be accepted in part or in whole, then it is advantageous to > allow Python programmers to begin the transition to the new stdlib > module names in Python 2.x, so that they can write forward compatible > code starting with Python 2.6. > > Note that PEP 3108 proposes to remove some "silly old stuff", > i.e. modules that are no longer useful or necessary. The PEP you are > reading does not address this because there are no forward > compatibility issues for modules that are to be removed, except to > stop using such modules. > > This PEP concerns only the mechanism by which mappings from old stdlib > names to new stdlib names are maintained. Please consult PEP 3108 for > all specific module renaming proposals. Specifically see the section > titled ``Modules to Rename`` for guidelines on the old name to new > name mappings. The few examples in this PEP are given for > illustrative purposes only and should not be used for specific > renaming recommendations. > > > Supported Renamings > =================== > > There are at least 4 use cases explicitly supported by this PEP: > > - - Simple top-level package name renamings, such as ``StringIO`` to > ``stringio``; > > - - Sub-package renamings where the package name may or may not be > renamed, such as ``email.MIMEText`` to ``email.mime.text``; > > - - Extension module renaming, such as ``cStringIO`` to ``cstringio``; > This feels a little misleading. The renaming is not restricted to extension modules but also works with any top-level module. > - - Third party renaming of any of the above. > > Two use cases supported by this PEP include renaming simple top-level > modules, such as ``StringIO``, as well as modules within packages, > such as ``email.MIMEText``. > > In the former case, PEP 3108 currently recommends ``StringIO`` be > renamed to ``stringio``, following PEP 8 recommendations [2]_. > > In the latter case, the email 4.0 package distributed with Python 2.5 > already renamed ``email.MIMEText`` to ``email.mime.text``, although it > did so in a one-off, uniquely hackish way inside the email package. > The mechanism described in this PEP is general enough to handle all > module renamings, obviating the need for the Python 2.5 hack (except > for backward compatibility with earlier Python versions). > > An additional use case is to support the renaming of C extension > modules. As long as the new name for the C module is importable, it > can be remapped to the new name. E.g. ``cStringIO`` renamed to > ``cstringio``. > > Third party package renaming is also supported, via several public > interfaces accessible by any Python module. > I guess a .pth file could install the mappings for the third-party modules. > Remappings are not performed recursively. > > > .mv files > ========= > > Remapping files are called ``.mv`` files; the suffix was chosen to be > evocative of the Unix mv(1) command. An ``.mv`` file is a simple > line-oriented text file. All blank lines and lines that start with a > # are ignored. All other lines must contain two whitespace separated > fields. The first field is the old module name, and the second field > is the new module name. Both module names must be specified using > their full dotted-path names. Here is an example ``.mv`` file from > Python 2.6:: > > # Map the various string i/o libraries to their new names > StringIO stringio > cStringIO cstringio > > ``.mv`` files can appear anywhere in the file system, and there is a > programmatic interface provided to parse them, and register the > remappings inside them. By default, when Python starts up, all the > ``.mv`` files in the ``oldlib`` package are read, and their remappings > are automatically registered. This is where all the module remappings > should be specified for top-level Python 2.x standard library modules. > I personally prefer to just have this info in a dict with a specified format for the dict. I think that should at least be an Open Issue or address why a new file format was chosen over that solution (don't remember what you said at PyCon =). > > Implementation Specification > ============================ > > This section provides the full specification for how module renamings > in Python 2.x are implemented. The central mechanism relies on > various import hooks as described in PEP 302 [3]_. Specifically > ``sys.path_importer_cache``, ``sys.path``, and ``sys.meta_path`` are > all employed to provide the necessary functionality. > > When Python's import machinery is initialized, the oldlib package is > imported. Inside oldlib there is a class called ``OldStdlibLoader``. > This class implements the PEP 302 interface and is automatically > instantiated, with zero arguments. The constructor reads all the > ``.mv`` files from the oldlib package directory, automatically > registering all the remappings found in those ``.mv`` files. This is > how the Python 2.x standard library is remapped. > > The OldStdlibLoader class should not be instantiated by other Python > modules. Instead, you can access the global OldStdlibLoader instance > via the ``sys.stdlib_remapper`` instance. Use this instance if you want > programmatic access to the remapping machinery. > > One important implementation detail: as needed by the PEP 302 API, a > magic string is added to sys.path, and module __path__ attributes in > order to hook in our remapping loader. This magic string is currently > ```` and some changes were necessary to Python's site.py file > in order to treat all sys.path entries starting with ``<`` as > special. Specifically, no attempt is made to make them absolute file > names (since they aren't file names at all). > Probably should mention that sys.path_importer_cache is then set for the magic string. > In order for the remapping import hooks to work, the module or package > must be physically located under its new name. This is because the > import hooks catch only modules that are not already imported, and > cannot be imported by Python's built-in import rules. Thus, if a > module has been moved, say from Lib/StringIO.py to Lib/stringio.py, > and the former's ``.pyc`` file has been removed, then without the > remapper, this would fail:: > > import StringIO > > Instead, with the remapper, this failing import will be caught, the > old name will be looked up in the registered remappings, and in this > case, the new name ``stringio`` will be found. The remapper then > attempts to import the new name, and if that succeeds, it binds the > resulting module into sys.modules, under both the old and new names. > Thus, the above import will result in entries in sys.modules for > 'StringIO' and 'stringio', and both will point to the exact same > module object. > > Note that no way to disable the remapping machinery is proposed, short > of moving all the ``.mv`` files away or programmatically removing them > in some custom start up code. In Python 3.0, the remappings will be > eliminated, leaving only the "new" names. > There is no mention of what the sys.meta_path importer/loader is used for. > > Programmatic Interface > ====================== > > Several methods are added to the ``sys.stdlib_remapper`` object, which > third party packages can use to register their own remappings. Is this really necessary? It might be helpful if this use of mapping old to new names gets picked up and used heavily, but otherwise this might be overkill. You can easily get the object from sys.path_importer_cache as long as you know the magic string used on sys.path. > Note > however that in all cases, there is one and only one mapping from an > old name to a new name. If two ``.mv`` files contain different > mappings for an old name, or if a programmatic call is made with an > old name that is already remapped, the previous mapping is lost. This > will not affect any already imported modules. > > The following methods are available on the ``sys.stdlib_remapper`` > object: > > - - ``read_mv_file(filename)`` -- Read the given file and register all > remappings found in the file. > > - - ``read_directory_mv_files(dirname, suffix='.mv')`` -- List the given > directory, reading all files in that directory that have the > matching suffix (``.mv`` by default). For each parsed file, > register all the remappings found in that file. > > - - ``set_mapping(oldname, newname)`` -- Register a new mapping from an > old module name to a new module name. Both must be the full > dotted-path name to the module. newname may be ``None`` in which > case any existing mapping for oldname will be removed (it is not an > error if there is no existing mapping). > > - - ``get_mapping(oldname, default=None)`` -- Return any registered > newname for the given oldname. If there is no registered remapping, > default is returned. > If you are going to have the object accessible from outside sys.path_importer_cache I would also add an attribute that contains the magic string used on sys.path. Thanks for writing this up, Barry! I am just glad our idea for this actually worked in the end. =) -Brett From guido at python.org Wed Mar 7 01:13:43 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Mar 2007 16:13:43 -0800 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: <20070306234849.6305.1580733783.divmod.xquotient.44@joule.divmod.com> References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> <20070228062840.17852.238152490.divmod.quotient.727@ohm> <20070306234849.6305.1580733783.divmod.xquotient.44@joule.divmod.com> Message-ID: On 3/6/07, glyph at divmod.com wrote: > On 10:22 pm, guido at python.org wrote: > >On 2/27/07, glyph at divmod.com wrote: > >>2to3 should take great care _tell_ you when it fails. One concern I have > >>is that the source translation may subtly alter the *semantics* of unit > >>test code, so that the tests are no longer effective and do not provide > >>adequate coverage. > > >Actually, this did happen with the iterkeys -> keys translation. A few > >unittests were reduced to pure nonsense by this. Most of those failed, > >but it was definitely a hassle finding out what went wrong (in part > >because I was too greedy and checked in a bunch of converted code > >without any review at all). Lesson learned I would say. Fortunately, > >this is a case where 2.6's Py3k warning mode should compensate. > > Was this due to a bug that can be fixed, or an inherent problem? Uh-huh. ;-) I wrote two versions of the dict views refactoring. One that turns d.keys() into list(d.keys()) and d.iterkeys() into iter(d.keys()). This one is pretty robust except if you have classes that emulate 2.x-style dicts. But it generates ugly code. So I have a "light-weight" version that leaves d.keys() alone, while turning d.iterkeys() into d.keys(). This generates prettier code but more buggy. I probably should have used the heavy-duty one instead. > I could imagine, thanks to Python's dynamism, there would be edge cases that > are impossible to detect completely reliably. If that's the case it would > be good to at least have pedantic warnings from 2to3 alerting the user to > places where translation could possibly be doing something semantically > dodgy. Right, this is my hope for 2.6. > Ideally before jumping the chasm with Twisted I'd like to make sure that all > of that sort of warning was gone, in _addition_ to a clean test run. Right. You need both. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From anthony at interlink.com.au Wed Mar 7 01:21:26 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed, 7 Mar 2007 11:21:26 +1100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45ED333F.5080502@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <87irdeamyh.fsf@uwakimon.sk.tsukuba.ac.jp> <45ED333F.5080502@v.loewis.de> Message-ID: <200703071121.26854.anthony@interlink.com.au> On Tuesday 06 March 2007 20:24, Martin v. L?wis wrote: > It might be that individuals get designated maintainers: Guido > maintains list and tuple, Tim maintaines dict, Raymond maintains > set, I maintain configure.in. However, I doubt that would work in > practice. That approach would simply give us many many single points of failure. For instance, you're maintaining a particular module but at the time an important bug/patch comes up, you're off on holidays, or busy, or the like. Sure, you could say "this person is primary, and this person is backup" but hell, there's a lot of different components that make up Python. That would be a maintenance and management nightmare. -- Anthony Baxter It's never too late to have a happy childhood. From dave at boost-consulting.com Wed Mar 7 01:13:28 2007 From: dave at boost-consulting.com (David Abrahams) Date: Tue, 06 Mar 2007 19:13:28 -0500 Subject: [Python-Dev] Finding the python library binaries (and docs) In-Reply-To: <45EDE75B.1010205@v.loewis.de> (Martin v. =?utf-8?Q?L=C3=B6wi?= =?utf-8?Q?s's?= message of "Tue\, 06 Mar 2007 23\:12\:43 +0100") References: <87ps7m89gn.fsf@valverde.peloton> <45EDDCCE.6080403@v.loewis.de> <873b4i8297.fsf@valverde.peloton> <45EDE75B.1010205@v.loewis.de> Message-ID: <87odn57vx3.fsf@valverde.peloton> on Tue Mar 06 2007, "Martin v. L?wis" wrote: > David Abrahams schrieb: >> on Tue Mar 06 2007, "Martin v. L?wis" wrote: >>> David Abrahams schrieb: >>>> I'm trying to find the Python library binaries associated with a given >>>> python executable. >>> This really isn't a python-dev question; please use python-list >>> (news:comp.lang.python) instead. >> >> I wrestled with the right list for this one and determined that only >> the python devs would know the answers. > > That absolutely cannot be the case. Python is open source, you have > *everything* you need to answer this question. That assumes this is one of those questions to which "use the source" is applicable. I think answering it requires some understanding of intention that's not explicit in the source. 'Course, it may be that the answer is "nobody knows." >>>> 2. I'd like to know if there's an officially correct procedure for >>>> finding the library binaries associated with a Python executable. >>> Yes (although I'm not sure what a "library binary" is). >> >> I gave one example in my post: libpython25.a > > Ah, ok. If you want to know where that is installed (officially), > check out what "make install" does. Also, ask yourself whether you know > a Python module that should know how to find it. distutils No help there. It has at best a haphazard method of looking for libraries, and never looks in a /config subdirectory on linux AFAICT. Also AFAICT the results are not tied to a particular Python executable. Looking at some code that works most of the time because the authors tried to deduce intention by looking at the Python source and existing installations isn't much better than trying to deduce intention by looking at the Python source and existing installations myself. > and freeze Not part of Python? -- Dave Abrahams Boost Consulting www.boost-consulting.com From skip at pobox.com Wed Mar 7 01:51:01 2007 From: skip at pobox.com (skip at pobox.com) Date: Tue, 6 Mar 2007 18:51:01 -0600 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45EDE4F9.2020902@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <200703060818.29114.phil@riverbankcomputing.co.uk> <45ED3506.9040707@v.loewis.de> <45EDE4F9.2020902@v.loewis.de> Message-ID: <17902.3189.985931.546009@montanaro.dyndns.org> Martin> If, for "Modules", you want a more fine-grained classification, Martin> it would be possible to add new categories, or add another field Martin> "affected modules" (multi-list, I assume). Why not add some tag capability to the new tracker (maybe the generic keywords field you mentioned would suffice)? People could attach whatever tags seem appropriate. Limiting the tags to a (nearly) fixed set of categories or the names of modules seems limiting. Given a set of tags you could also do the tag cloud thing and be more buzzword compliant at the same time. ;-) Skip From jimjjewett at gmail.com Wed Mar 7 01:58:27 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 6 Mar 2007 19:58:27 -0500 Subject: [Python-Dev] how can I review? [was: Encouraging developers] In-Reply-To: <45EDE244.7080404@v.loewis.de> References: <45EDE244.7080404@v.loewis.de> Message-ID: I guess I should have been less specific. :D It has been quite a while since I worried about my own patches going stale; I just want to know how my review time can be more useful. Once a committer has already decided to look at a patch, comments may make the next step easier. But is there anyway to flag an issue as either ready for that look, or already judged not-yet-ready? On 3/6/07, "Martin v. L?wis" wrote: > If you have a list of patches that you think are ready for > decision, please post the list here, listing clear accepts and > clear rejects. ... > ... If you have completed > a review (i.e. so that just some technical action would need to > be taken), feel free to post to python-dev (ideally with a I had assumed that doing this patch-at-a-time would quickly get annoying. Is batching several patches together part of what you want? (If so, I would personally still rather that the tracker or at least a wiki page did that grouping.) > I agree. Maybe we should add some "reviewed by" field to the roundup > tracker where reviewers can record that they brought the patch > to a state ready for committing/final rejection Yes, please. > (I think a second > check is still needed, as the case of splitext shows: you were in > favor of rejecting it because of the incompatibility, ... (I think that was actually John J. Lee; I did just add a note that it should no longer be considered bugfix candidate for 2.4) -jJ From barry at python.org Wed Mar 7 02:08:22 2007 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Mar 2007 20:08:22 -0500 Subject: [Python-Dev] Encouraging developers In-Reply-To: <17902.3189.985931.546009@montanaro.dyndns.org> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <200703060818.29114.phil@riverbankcomputing.co.uk> <45ED3506.9040707@v.loewis.de> <45EDE4F9.2020902@v.loewis.de> <17902.3189.985931.546009@montanaro.dyndns.org> Message-ID: <060C1D8F-6492-4153-ADBB-C4A16C8CE710@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mar 6, 2007, at 7:51 PM, skip at pobox.com wrote: > Why not add some tag capability to the new tracker (maybe the generic > keywords field you mentioned would suffice)? People could attach > whatever > tags seem appropriate. Limiting the tags to a (nearly) fixed set of > categories or the names of modules seems limiting. Given a set of > tags you > could also do the tag cloud thing and be more buzzword compliant at > the same > time. ;-) Jira had a way of automatically assigning certain categories to certain people. I think the term was project leader or some such. Of course, this didn't mean that someone else couldn't fix the bug or that the bug couldn't be reassigned to someone else, but at least it initially gives all issues to the person nominally responsible for it. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRe4Qh3EjvBPtnXfVAQIzegP/dzWv3DGAKy5p5nRR/aWU90DVQJnw/WfK 0PPHlNCDhgixnQvjLP/0B8feeeUXHW3dwvodFkYrtxTHze6airkJLl9bypxMGgRo Q7ckj+51cXxySHLj9+5FEkdFS9gxrpWf6+toZoGSVZ08Wa2Pz6v2Pa3Oiu5kgvZ4 LAKfdvbFDh8= =U1zJ -----END PGP SIGNATURE----- From rhamph at gmail.com Wed Mar 7 02:10:35 2007 From: rhamph at gmail.com (Adam Olsen) Date: Tue, 6 Mar 2007 18:10:35 -0700 Subject: [Python-Dev] Adding socket timeout to urllib2 In-Reply-To: References: Message-ID: On 3/6/07, Guido van Rossum wrote: > On 3/6/07, Facundo Batista wrote: > > Guido van Rossum wrote: > > > > >> - I'll modify urlopen for it to accept a socket_timeout parameter, > > >> default to None > > > > > > I'd call it timeout. There can't really be much ambiguity can there? > > > > Yes and no. For example, if I do a > > ``urllib2.urlopen("ftp://ftp.myhome.com.ar/blah.txt", timeout=10)``, the > > timeout is about the socket or about the file transfer? > > Think of it this way. "Timeout" doesn't mean the whole thing needs to > be completed in 10 secs. It means that over 10 secs of no activity > causes it to be aborted. IOW, It's an idle timeout. -- Adam Olsen, aka Rhamphoryncus From guido at python.org Wed Mar 7 02:15:28 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Mar 2007 17:15:28 -0800 Subject: [Python-Dev] Adding socket timeout to urllib2 In-Reply-To: References: Message-ID: On 3/6/07, Adam Olsen wrote: > On 3/6/07, Guido van Rossum wrote: > > On 3/6/07, Facundo Batista wrote: > > > Guido van Rossum wrote: > > > > > > >> - I'll modify urlopen for it to accept a socket_timeout parameter, > > > >> default to None > > > > > > > > I'd call it timeout. There can't really be much ambiguity can there? > > > > > > Yes and no. For example, if I do a > > > ``urllib2.urlopen("ftp://ftp.myhome.com.ar/blah.txt", timeout=10)``, the > > > timeout is about the socket or about the file transfer? > > > > Think of it this way. "Timeout" doesn't mean the whole thing needs to > > be completed in 10 secs. It means that over 10 secs of no activity > > causes it to be aborted. > > IOW, It's an idle timeout. That's not in wikipedia. :-) It's the only timeout that is available to us, realistically; the socket module calls it timeout everywhere. So I think that should be a good name for it. The argument name doesn't need to server as complete documentation. I don't expect we'll ever see another kind of timeout added to this same API, and if we do, we'll just have to pick a different name for it. ;-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Wed Mar 7 03:05:59 2007 From: skip at pobox.com (skip at pobox.com) Date: Tue, 6 Mar 2007 20:05:59 -0600 Subject: [Python-Dev] Adding socket timeout to urllib2 In-Reply-To: References: Message-ID: <17902.7687.851296.385591@montanaro.dyndns.org> >> Think of it this way. "Timeout" doesn't mean the whole thing needs to >> be completed in 10 secs. It means that over 10 secs of no activity >> causes it to be aborted. Adam> IOW, It's an idle timeout. Not quite. It's a timeout when you are waiting for some sort of response. If you make a connection to an ftp server to send files the connection shouldn't be aborted if you take more than 10 seconds to prepare the file you want to upload. OTOH, if you send the file and don't get an acknowledgement back for 10 seconds, then you get a TimeoutError. Skip From rhamph at gmail.com Wed Mar 7 03:21:24 2007 From: rhamph at gmail.com (Adam Olsen) Date: Tue, 6 Mar 2007 19:21:24 -0700 Subject: [Python-Dev] Adding socket timeout to urllib2 In-Reply-To: References: Message-ID: On 3/6/07, Guido van Rossum wrote: > On 3/6/07, Adam Olsen wrote: > > IOW, It's an idle timeout. > > That's not in wikipedia. :-) I know, I checked before posting. ;) > It's the only timeout that is available to us, realistically; the > socket module calls it timeout everywhere. So I think that should be a > good name for it. The argument name doesn't need to server as complete > documentation. I don't expect we'll ever see another kind of timeout > added to this same API, and if we do, we'll just have to pick a > different name for it. ;-) On 3/6/07, skip at pobox.com wrote: > Not quite. It's a timeout when you are waiting for some sort of response. > If you make a connection to an ftp server to send files the connection > shouldn't be aborted if you take more than 10 seconds to prepare the file > you want to upload. OTOH, if you send the file and don't get an > acknowledgement back for 10 seconds, then you get a TimeoutError. I think calling it "timeout" in the API is fine. The documentation can then clarify that it's an idle timeout, except it only applies when blocked in a network operation. -- Adam Olsen, aka Rhamphoryncus From guido at python.org Wed Mar 7 03:32:48 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Mar 2007 18:32:48 -0800 Subject: [Python-Dev] Adding socket timeout to urllib2 In-Reply-To: References: Message-ID: On 3/6/07, Adam Olsen wrote: > I think calling it "timeout" in the API is fine. The documentation > can then clarify that it's an idle timeout, except it only applies > when blocked in a network operation. Since "idel timeout" is not a commonly understood term it would be even better if it was explained without using it. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Wed Mar 7 03:59:10 2007 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Mar 2007 21:59:10 -0500 Subject: [Python-Dev] PEP 364, Transitioning to the Py3K standard library In-Reply-To: References: <6EBA9F60-ECBF-426B-AC60-E1044AEDE938@python.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >> Supported Renamings >> =================== >> >> There are at least 4 use cases explicitly supported by this PEP: >> >> - - Simple top-level package name renamings, such as ``StringIO`` to >> ``stringio``; >> >> - - Sub-package renamings where the package name may or may not be >> renamed, such as ``email.MIMEText`` to ``email.mime.text``; >> >> - - Extension module renaming, such as ``cStringIO`` to >> ``cstringio``; >> > > This feels a little misleading. The renaming is not restricted to > extension modules but also works with any top-level module. Possibly. It's just a use case that illustrates the solution isn't tied to pure-Python modules. But I'm not wedded to making this distinction in the PEP. >> Third party package renaming is also supported, via several public >> interfaces accessible by any Python module. >> > > I guess a .pth file could install the mappings for the third-party > modules. How would that work? .pth files don't execute arbitrary Python code, but it seems like that would be necessary to make this work. OTOH, your suggestion does make me think that it might be possible to piggyback the module name mappings onto .pth file processing. It already understand the 'import' command, so why not a 'rename' command? >> ``.mv`` files can appear anywhere in the file system, and there is a >> programmatic interface provided to parse them, and register the >> remappings inside them. By default, when Python starts up, all the >> ``.mv`` files in the ``oldlib`` package are read, and their >> remappings >> are automatically registered. This is where all the module >> remappings >> should be specified for top-level Python 2.x standard library >> modules. >> > > I personally prefer to just have this info in a dict with a specified > format for the dict. I think that should at least be an Open Issue or > address why a new file format was chosen over that solution (don't > remember what you said at PyCon =). Think about what that big dictionary would look like. It would essentially be like a text file with mappings one per line, but you'd have a lot of extra goo on each line (quotes, colons, and commas). I thought it was important to decentralize the mappings and then to put them in the simplest format that would serve the purpose. > Probably should mention that sys.path_importer_cache is then set for > the magic string. Yep. > There is no mention of what the sys.meta_path importer/loader is > used for. Good point. >> Programmatic Interface >> ====================== >> >> Several methods are added to the ``sys.stdlib_remapper`` object, >> which >> third party packages can use to register their own remappings. > > Is this really necessary? It might be helpful if this use of mapping > old to new names gets picked up and used heavily, but otherwise this > might be overkill. You can easily get the object from > sys.path_importer_cache as long as you know the magic string used on > sys.path. So, I was thinking that third party packages could use the same mechanism to provide their own remappings to PEP 8 names. Certainly if that's something that third party packages want to do, it would be better to use some common Python API to do that rather than have them bake their own (or worse, avoid PEP 8-ifying their module names because of backward compatibility concerns). If we agree that's a use case we should support, then either we support this programmatically or we don't expose the API and do something automatic when the package is imported. I'm of two different minds on this. Probably the way I'd rather see this work is for a package to include a package.mv file in its package directory, and that file would get automatically loaded when the package was first imported. If people really don't like the separate data file, and really want to just use a Python dictionary, then the way to keep the decentralization would be to reserve a __*__ symbol in the package namespace that contained this remapping. Something like __renames__ = {...}. > If you are going to have the object accessible from outside > sys.path_importer_cache I would also add an attribute that contains > the magic string used on sys.path. Of course, that magic string is also available via oldlib._magic. I tend to think it won't be necessary, but if people really want it, I can see putting the magic string as an attribute on the OldStdlibLoader object, e.g. sys.stdlib_remapper.importer_cache_key. > Thanks for writing this up, Barry! I am just glad our idea for this > actually worked in the end. =) Indeed! I'm glad we got a chance to work on this together! - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRe4qj3EjvBPtnXfVAQJUvQP/Qe+rEoeZqn4OrnPg69aiW+m+5aF5TYLJ f6YXrvp+QGCcVbfPcwBmjIAlSBNB8MOobWDD9g5A3IIBVM+2+m4y/xKtTtr/c7mr bpPu6NZbbp2KaYiMVA6brbv9TQvjOP+eLft/hAEveKXZd1MhcdkhaEOX3uWS1u2X UeCEbEDj2AA= =VEHl -----END PGP SIGNATURE----- From skip at pobox.com Wed Mar 7 04:07:12 2007 From: skip at pobox.com (skip at pobox.com) Date: Tue, 6 Mar 2007 21:07:12 -0600 Subject: [Python-Dev] Adding socket timeout to urllib2 In-Reply-To: References: Message-ID: <17902.11360.91118.646113@montanaro.dyndns.org> Guido> Since "idel timeout" is not a commonly understood term it would Guido> be even better if it was explained without using it. I think it's commonly understood, but it doesn't mean what the socket timeout is used for. It's how long a connection can be idle (the client doesn't make a request of the server) before the server will close the connection. Skip From skip at pobox.com Wed Mar 7 04:16:34 2007 From: skip at pobox.com (skip at pobox.com) Date: Tue, 6 Mar 2007 21:16:34 -0600 Subject: [Python-Dev] PEP 364, Transitioning to the Py3K standard library In-Reply-To: References: <6EBA9F60-ECBF-426B-AC60-E1044AEDE938@python.org> Message-ID: <17902.11922.341345.349583@montanaro.dyndns.org> Barry, Does the proposed renaming include any restructuring (e.g. making hierarchies out of all or part of the stdlib where none existed before)? It wasn't obvious to me. For example, might there be a restructuring of the entire stdlib? Skip From skip at pobox.com Wed Mar 7 04:18:02 2007 From: skip at pobox.com (skip at pobox.com) Date: Tue, 6 Mar 2007 21:18:02 -0600 Subject: [Python-Dev] Adding socket timeout to urllib2 In-Reply-To: <6a36e7290703061915l4f67059dj76146edbb0903072@mail.gmail.com> References: <17902.11360.91118.646113@montanaro.dyndns.org> <6a36e7290703061915l4f67059dj76146edbb0903072@mail.gmail.com> Message-ID: <17902.12010.766128.422770@montanaro.dyndns.org> Bob> What does idle timeout have to do with urllib2 or any IO layer for Bob> that matter? I've only seen it as a very high level server-only Bob> feature... Nothing at all. I believe Adam just applied that term incorrectly to the socket timeout parameter. Skip From bob at redivi.com Wed Mar 7 04:15:24 2007 From: bob at redivi.com (Bob Ippolito) Date: Tue, 6 Mar 2007 19:15:24 -0800 Subject: [Python-Dev] Adding socket timeout to urllib2 In-Reply-To: <17902.11360.91118.646113@montanaro.dyndns.org> References: <17902.11360.91118.646113@montanaro.dyndns.org> Message-ID: <6a36e7290703061915l4f67059dj76146edbb0903072@mail.gmail.com> On 3/6/07, skip at pobox.com wrote: > > Guido> Since "idel timeout" is not a commonly understood term it would > Guido> be even better if it was explained without using it. > > I think it's commonly understood, but it doesn't mean what the socket > timeout is used for. It's how long a connection can be idle (the client > doesn't make a request of the server) before the server will close the > connection. > What does idle timeout have to do with urllib2 or any IO layer for that matter? I've only seen it as a very high level server-only feature... -bob From nilton.volpato at gmail.com Wed Mar 7 06:21:54 2007 From: nilton.volpato at gmail.com (Nilton Volpato) Date: Wed, 7 Mar 2007 02:21:54 -0300 Subject: [Python-Dev] Lack of sequential decompression in the zipfile module In-Reply-To: <5728eb900702161153t27f98db2ub06ec892b6039519@mail.gmail.com> References: <5728eb900702161153t27f98db2ub06ec892b6039519@mail.gmail.com> Message-ID: <27fef5640703062121k3785fc84w4396bba9322b590c@mail.gmail.com> Hi Derek, On 2/16/07, Derek Shockey wrote: > Though I am an avid Python programmer, I've never forayed into the area of > developing Python itself, so I'm not exactly sure how all this works. > > I was confused (and somewhat disturbed) to discover recently that the > zipfile module offers only one-shot decompression of files, accessible only > via the read() method. It is my understanding that the module will handle > files of up to 4 GB in size, and the idea of decompressing 4 GB directly > into memory makes me a little queasy. Other related modules (zlib, tarfile, > gzip, bzip2) all offer sequential decompression, but this does not seem to > be the case for zipfile (even though the underlying zlib makes it easy to > do). Not so easy, in fact. Unless you open only one zip member file at a time. If you open many member files concurrently how does file cache will work? Or how many seeks you will have to do if you read from one member file and from other alternatingly? Do you have a file-like interface or just read in chunks? Or, if you need to open more than one member file for writing in the same zip file, then this is not possible at all. > Since I was writing a script to work with potentially very large zipped > files, I took it upon myself to write an extract() method for zipfile, which > is essentially an adaption of the read() method modeled after tarfile's > extract(). I feel that this is something that should really be provided in > the zipfile module to make it more usable. I'm wondering if this has been > discussed before, or if anyone has ever viewed this as a problem. I can post > the code I wrote as a patch, though I'm not sure if my file IO handling is > as robust as it needs to be for the stdlib. I'd appreciate any insight into > the issue or direction on where I might proceed from here so as to fix what > I see as a significant problem. My Google Summer of Code project was just about this, and I implemented a lot of nice features. These features include: file-like access to zip member files (which solves your problem, and also provides a real file-like interface including .read(), .readline(), etc); support for BZIP2 compression; support for removing a member file; support for encrypting/decrypting member files. The project is hosted at sourceforge [http://ziparchive.sf.net]. You can take a look, and try it. I'm planning to make a new and improved release perfecting the API and doing some code refactoring. I really think that this improved version will be better than all other zip libraries in every aspect, including number of implemented features, speed/efficiency, and being easy to use. I think the time I will take to do this is roughly directly proportional to the amount of feedback (and help) I receive, since I alone can't think about all the needs of such a library. Also, if anyone would like to help developing, that you be great! I have some local code I'm working in, but I can commit this to an svn branch if anyone would like to see/help. Thanks, -- Nilton From nnorwitz at gmail.com Wed Mar 7 07:12:23 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 6 Mar 2007 22:12:23 -0800 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45EDD141.20605@ronadam.com> References: <20070305184614.GA19051@localhost.localdomain> <200703052116.48035.phil@riverbankcomputing.co.uk> <45ED00FC.6050406@v.loewis.de> <200703060834.06715.phil@riverbankcomputing.co.uk> <45EDD141.20605@ronadam.com> Message-ID: On 3/6/07, Ron Adam wrote: > Neal Norwitz wrote: > > I'm looking forward to a new tracker and hope it manages single projects... > (patches and bugs) better. It would be great if we could search for items > based on possibly the following conditions. The best place to discuss these things are on tracker-discuss. I think everything is pretty much done, we are waiting for someone to help write doc (Brett knows more about this). We are also waiting until after 2.5.1 is out before we switch over so there aren't unforeseen issues with the release. Hopefully 2.5.1 will be out in a month. > *Note: Rejections could be done sooner if it obviously a bad idea. Agreed. Note: I review nearly all traffic on patches through the mailing list. If I see someone say this patch should be rejected and I agree, I will do it immediately. I rarely do that because I don't see those comments. Mostly I just close duplicates that happen from time to time. > In the case of bugs, the tests would probably come first in order to verify > and reproduce the specific bug. Yup. Also write a test case if one doesn't exist. That will greatly speed debugging. Adding helpful information about what platform the problem occurs on can speed things up too. Writing a NEWS entry would also speed things up. This may sound kinda stupid, but if enough people help out, it might reduce the time it takes to fix a bug from 30 minutes to 15 minutes. If I only have 15 minutes to work on a problem, it's the difference between it getting fixed or remaining unresolved. > What something like this would do is effectively distribute knowledge as > you suggest. For example someone good at writing docs could do a search > for all patches that do not have doc's or need docs reviewed and contribute > in that way. The same for someone interested in writing and reviewing tests. Yes, every little bit helps! > I wonder if a tracker could also have patch requests. Similar to bugs, > except it's more of a todo list. Oh, never mind there is a feature request > group already. Could that possibly be extended to other areas? I don't know what you mean, can you clarify? n From nnorwitz at gmail.com Wed Mar 7 07:33:10 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 6 Mar 2007 22:33:10 -0800 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <200703060818.29114.phil@riverbankcomputing.co.uk> <45ED3506.9040707@v.loewis.de> Message-ID: On 3/6/07, Terry Reedy wrote: > > ""Martin v. L?wis"" wrote in message > news:45ED3506.9040707 at v.loewis.de... > >1. Public identification will not help, because: > >2. most code isn't in the responsibility of anybody (so publically > > identifying responsibilities would leave most code unassigned), and > >3. for the code that has some responsible member, things are already > > fine (so public identification won't improve here) > > If I were looking for an 'ophan' (python-coded) module to adopt, > then such a list would help me choose. os.listdir(os.path.dirname(os.__file__)) Does that help? :-) It's not really that far off, though. :-( Larger modules that have been contributed recently like subprocess and tarfile are well maintained. Most other modules are less so. Most of the modules don't have problems. However, many of the web-centric modules have outstanding issues, as does sre. I think some of the shell utility modules (shutil, path handling) have issues wrt portability. There are also many problems that only occur on one platform (most often Windows, AIX, or HP-UX). I won't touch these sorts of problems unless I can properly verify the problem and the fix. Generally problems in these areas don't have tests which make fixing them without access to the platform too costly IMO. asyncore has a lot of outstanding issues IIRC. Also many issues are documentation. I think it would be best for people to pick a module they are interested in and knowledgeable about. We could create a wiki that would be for informational use. If it works well, maybe we could formalize it. Start something, announce it to python-list or python-dev, and try it. If it works, we'll adopt it. > It would also be helpful if the new tracker system could produce a list of > module-specific open items sorted by module, since that would indicate > modules needing attention, and I could look for a batch that were > unassigned. Agreed. That's why I wanted keywords to be supported (which I believe they are). If we can slice and dice the issues up into categories, it will be easy to figure out where we need to spend more time. It also can be a great incentive to drop 5 issues to 0. Going from 977 to 972, just isn't as satisfying. n From rrr at ronadam.com Wed Mar 7 07:48:30 2007 From: rrr at ronadam.com (Ron Adam) Date: Wed, 07 Mar 2007 00:48:30 -0600 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> <200703052116.48035.phil@riverbankcomputing.co.uk> <45ED00FC.6050406@v.loewis.de> <200703060834.06715.phil@riverbankcomputing.co.uk> <45EDD141.20605@ronadam.com> Message-ID: <45EE603E.1000807@ronadam.com> Neal Norwitz wrote: > On 3/6/07, Ron Adam wrote: >> Neal Norwitz wrote: >> >> I'm looking forward to a new tracker and hope it manages single >> projects... >> (patches and bugs) better. It would be great if we could search for >> items >> based on possibly the following conditions. > > The best place to discuss these things are on tracker-discuss. I > think everything is pretty much done, we are waiting for someone to > help write doc (Brett knows more about this). We are also waiting > until after 2.5.1 is out before we switch over so there aren't > unforeseen issues with the release. Hopefully 2.5.1 will be out in a > month. Cool! :-) >> *Note: Rejections could be done sooner if it obviously a bad idea. > > Agreed. > > Note: I review nearly all traffic on patches through the mailing list. > If I see someone say this patch should be rejected and I agree, I > will do it immediately. I rarely do that because I don't see those > comments. Mostly I just close duplicates that happen from time to > time. > >> In the case of bugs, the tests would probably come first in order to >> verify >> and reproduce the specific bug. > > Yup. Also write a test case if one doesn't exist. That will greatly > speed debugging. Adding helpful information about what platform the > problem occurs on can speed things up too. Writing a NEWS entry would > also speed things up. > > This may sound kinda stupid, but if enough people help out, it might > reduce the time it takes to fix a bug from 30 minutes to 15 minutes. > If I only have 15 minutes to work on a problem, it's the difference > between it getting fixed or remaining unresolved. I never consider clarifying statements even a little bit stupid. Only a bit boring if I hear them more than a few times in a short time period. But that's definitely not a problem compared with not having them stated at all. For us who aren't (as) familiar with all the issues surround a bug or the specific code in question it can take considerably longer. >> What something like this would do is effectively distribute knowledge >> as you suggest. For example someone good at writing docs could do a >> search for all patches that do not have doc's or need docs reviewed >> and contribute in that way. The same for someone interested in writing >> and reviewing tests. > > Yes, every little bit helps! But the tracker needs to be able to actually track the status of individual items for this to work. Currently there's this huge list and you have to either wade though it to find out the status of each item, or depend on someone bring it to your attention. (This has been discussed before.) >> I wonder if a tracker could also have patch requests. Similar to bugs, >> except it's more of a todo list. Oh, never mind there is a feature >> request group already. Could that possibly be extended to other >> areas? > > I don't know what you mean, can you clarify? I suppose I'm thinking of things that are even more general than a feature request. Like possibly a research request. Or a request for something to be done first so that something else can then be done. Ron From dalke at dalkescientific.com Wed Mar 7 08:02:47 2007 From: dalke at dalkescientific.com (Andrew Dalke) Date: Wed, 7 Mar 2007 00:02:47 -0700 Subject: [Python-Dev] locals(), closures, and IronPython... In-Reply-To: <3d2ce8cb0703061523j438783eeu5eb4df719cb3840@mail.gmail.com> References: <7AD436E4270DD54A94238001769C22276A624C7482@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <45EDEB6A.1000409@canterbury.ac.nz> <3d2ce8cb0703061523j438783eeu5eb4df719cb3840@mail.gmail.com> Message-ID: On 3/6/07, Mike Klaas wrote: > There's nothing quite like running help(func) and getting *args, > **kwargs as the documented parameter list. I think >>> import resource >>> help(resource.getpagesize) Help on built-in function getpagesize in module resource: getpagesize(...) is pretty close. :) Andrew dalke at dalkescientific.com From martin at v.loewis.de Wed Mar 7 08:39:28 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 07 Mar 2007 08:39:28 +0100 Subject: [Python-Dev] Finding the python library binaries (and docs) In-Reply-To: <87odn57vx3.fsf@valverde.peloton> References: <87ps7m89gn.fsf@valverde.peloton> <45EDDCCE.6080403@v.loewis.de> <873b4i8297.fsf@valverde.peloton> <45EDE75B.1010205@v.loewis.de> <87odn57vx3.fsf@valverde.peloton> Message-ID: <45EE6C30.8040608@v.loewis.de> David Abrahams schrieb: >> That absolutely cannot be the case. Python is open source, you have >> *everything* you need to answer this question. > > That assumes this is one of those questions to which "use the source" > is applicable. I think answering it requires some understanding of > intention that's not explicit in the source. 'Course, it may be that > the answer is "nobody knows." Well, I probably didn't get your question then. Your original article seemed to focus on "the documentation is incomplete". Completing the documentation is always possible just by reading the source, documenting the status quo. It seemed that you were also asking what the status quo is. >> Ah, ok. If you want to know where that is installed (officially), >> check out what "make install" does. Also, ask yourself whether you know >> a Python module that should know how to find it. distutils > > No help there. It has at best a haphazard method of looking for > libraries, and never looks in a /config subdirectory on linux AFAICT. That's not true. Look, for example, at sysconfig.get_makefile_filename. Not sure what "haphazard method" you are referring to. It would be more helpful if you would say "I found this and that. Is it true that this is haphazard?" to which my likely answer is "no, this is intentional". I'm not aware of haphazard algorithms in Python. > Also AFAICT the results are not tied to a particular Python > executable. As you don't mention what specific algorithm you are referring to here, it's difficult to discuss whether your findings are correct. sysconfig.EXEC_PREFIX is definitely installation-specific > Looking at some code that works most of the time because the authors > tried to deduce intention by looking at the Python source and existing > installations isn't much better than trying to deduce intention by > looking at the Python source and existing installations myself. > >> and freeze > > Not part of Python? Tools/freeze Regards, Martin From martin at v.loewis.de Wed Mar 7 09:07:02 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 07 Mar 2007 09:07:02 +0100 Subject: [Python-Dev] how can I review? [was: Encouraging developers] In-Reply-To: References: <45EDE244.7080404@v.loewis.de> Message-ID: <45EE72A6.1020104@v.loewis.de> Jim Jewett schrieb: > It has been quite a while since I worried about my own patches going > stale; I just want to know how my review time can be more useful. > > Once a committer has already decided to look at a patch, comments may > make the next step easier. > > But is there anyway to flag an issue as either ready for that look, or > already judged not-yet-ready? Not in the current tracker, no. There is the "accepted/open" state, but we use it to indicate "just needs committing"; if a reviewer doesn't have the authority to actually approve the patch, it would give the submitter the false impression that the patch will go in (which it might, depending on whether the committer agrees with the review). > I had assumed that doing this patch-at-a-time would quickly get > annoying. To some, perhaps. However, they should unsubscribe from python-dev if they get annoyed. As glyph said: this is the list where patches should be discussed. I can see why automated messages shouldn't go to python-dev; if python-development-related hand-written messages annoy people, they clearly subscribed to the wrong list (these days, some of them better subscribe to python-ideas). > Is batching several patches together part of what you want? Not sure I understand. You mean several related bugs/patches? I tend to look at patches for a single module often at one time, in particular to find out whether they overlap or conflict, and what the best order to apply them would be. Often, submitters will have to redo their patch if a conflicting one is applied, so here batching is important. If you mean "do I work on python patches in batches?", then: sometimes. I tried "a bug a day" for some time, but couldn't sustain it. At the moment, I have some free time at hand, so I look at many patches. If you have a list of patches to be rejected, this could be easily batched (rejecting is more easy than applying). For committing, I still need to understand the issue in detail, and the proposed patch. Of course, I would hope that the review already lays it out so that it is more easy to understand pros and cons, and that it will give information that the submitter omitted (mostly because it was clear to him), so for trivial patches, batch-committing them is feasible. > (If so, I would personally still rather that the tracker or at least > a wiki page did that grouping.) Sure. I just created http://wiki.python.org/moin/PatchTriage Feel free to add patches there (if you want to, along with a date on which you added it) >> (I think a second >> check is still needed, as the case of splitext shows: you were in >> favor of rejecting it because of the incompatibility, ... > > (I think that was actually John J. Lee; I did just add a note that it > should no longer be considered bugfix candidate for 2.4) Oops, sorry, confusing different JJs here (this may have happened in the past also, I guess). Regards, Martin From martin at v.loewis.de Wed Mar 7 09:08:06 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 07 Mar 2007 09:08:06 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <060C1D8F-6492-4153-ADBB-C4A16C8CE710@python.org> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <200703060818.29114.phil@riverbankcomputing.co.uk> <45ED3506.9040707@v.loewis.de> <45EDE4F9.2020902@v.loewis.de> <17902.3189.985931.546009@montanaro.dyndns.org> <060C1D8F-6492-4153-ADBB-C4A16C8CE710@python.org> Message-ID: <45EE72E6.3070501@v.loewis.de> Barry Warsaw schrieb: > Jira had a way of automatically assigning certain categories to certain > people. I think the term was project leader or some such. Of course, > this didn't mean that someone else couldn't fix the bug or that the bug > couldn't be reassigned to someone else, but at least it initially gives > all issues to the person nominally responsible for it. The SF tracker has this also. I'm auto-assigned for Tkinter bugs, for example - not that I review them in a timely manner. Regards, Martin From greg.ewing at canterbury.ac.nz Wed Mar 7 09:22:15 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 07 Mar 2007 21:22:15 +1300 Subject: [Python-Dev] Lack of sequential decompression in the zipfile module In-Reply-To: <27fef5640703062121k3785fc84w4396bba9322b590c@mail.gmail.com> References: <5728eb900702161153t27f98db2ub06ec892b6039519@mail.gmail.com> <27fef5640703062121k3785fc84w4396bba9322b590c@mail.gmail.com> Message-ID: <45EE7637.6090901@canterbury.ac.nz> Nilton Volpato wrote: > If you open many member files concurrently how does file cache will > work? Or how many seeks you will have to do if you read from one > member file and from other alternatingly? If the OS file system cache is doing its job properly, I don't think the seeking should be a problem. > Or, if you need to open more than > one member file for writing in the same zip file, then this is not > possible at all. I don't think it would be unreasonable to just not support writing to more than one member at a time. -- Greg From martin at v.loewis.de Wed Mar 7 09:26:39 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 07 Mar 2007 09:26:39 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> References: <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <5.1.1.6.0.20070306162907.02b7e2a0@sparrow.telecommunity.com> <45EDE899.9020709@v.loewis.de> <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> Message-ID: <45EE773F.20003@v.loewis.de> glyph at divmod.com schrieb: > More to the point, we know the cost, what's the benefit? Is there any > sort of bug that it is likely to prevent in *new* code? Yes. People are more likely to classify the file as "no extension", which more likely meets the user's expectation. Also, it won't happen that splitext gives an empty file name, which otherwise may cause crashes. > It clearly isn't going to help any old code. That's not at all clear. Instead, old code will suddenly work correctly (in the user's point of view), where it before gave nonsensical results (which users have likely ignored). For example, from the standard library: py> import pydoc py> pydoc.importfile("/Users/loewis/.pythonrc") This is non-sensical: the module isn't an empty string. Instead, it should be '.pythonrc' - just as it would be 'pythonrc' if the file name was 'pythonrc'. So old code in the standard library will be improved. > It seems there are people who see it > both ways, and I haven't seen anything compelling to indicate that > either behavior is particularly less surprising in the edge case. Maybe you aren't grounded so much in Unix history. It really feels wrong that a dotfile is considered as having an extension. > In cases like this, historical context should be considered, even for a > major overhaul like 3.0. Of course, if the newly proposed semantics > provided a solution to a real problem or common error, compatibility > might be determined to be a less important issue. All of these apply here. > The use-cases being discussed here would be better served by having new > APIs that do particular things and don't change existing semantics, > though. For example, a "guess_mime_type(path)" function which could > examine a path and figure out its mime type based on its extension > (among other things). I disagree. The current behavior is incorrect (and always has been); the new behavior is clearly correct. Regards, Martin From martin at v.loewis.de Wed Mar 7 09:29:33 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 07 Mar 2007 09:29:33 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <17901.63078.595584.18386@terry-jones-computer.local> References: <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <5.1.1.6.0.20070306162907.02b7e2a0@sparrow.telecommunity.com> <17901.63078.595584.18386@terry-jones-computer.local> Message-ID: <45EE77ED.2050001@v.loewis.de> Terry Jones schrieb: > I do think the behavior can be improved, and that it should be fixed, but > at a place where other incompatible changes will also be being made, Indeed, 2.6 is such a place. Any feature release can contain incompatible behavior, and any feature release did contain incompatible behavior. Just look at the "porting to" sections of past whatsnew files. Regards, Martin From martin at v.loewis.de Wed Mar 7 09:37:19 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 07 Mar 2007 09:37:19 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45EE603E.1000807@ronadam.com> References: <20070305184614.GA19051@localhost.localdomain> <200703052116.48035.phil@riverbankcomputing.co.uk> <45ED00FC.6050406@v.loewis.de> <200703060834.06715.phil@riverbankcomputing.co.uk> <45EDD141.20605@ronadam.com> <45EE603E.1000807@ronadam.com> Message-ID: <45EE79BF.9020100@v.loewis.de> Ron Adam schrieb: > But the tracker needs to be able to actually track the status of individual > items for this to work. Currently there's this huge list and you have to > either wade though it to find out the status of each item, or depend on > someone bring it to your attention. Well, the tracker does have a status for each individual report: open or closed. open means "needs more work", closed means "no more work needed". So don't wade through a report - just assume that if it is open, it needs work. Regards, Martin From martin at v.loewis.de Wed Mar 7 09:41:24 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 07 Mar 2007 09:41:24 +0100 Subject: [Python-Dev] Lack of sequential decompression in the zipfile module In-Reply-To: <27fef5640703062121k3785fc84w4396bba9322b590c@mail.gmail.com> References: <5728eb900702161153t27f98db2ub06ec892b6039519@mail.gmail.com> <27fef5640703062121k3785fc84w4396bba9322b590c@mail.gmail.com> Message-ID: <45EE7AB4.5060908@v.loewis.de> Nilton Volpato schrieb: > My Google Summer of Code project was just about this, and I > implemented a lot of nice features. These features include: file-like > access to zip member files (which solves your problem, and also > provides a real file-like interface including .read(), .readline(), > etc); support for BZIP2 compression; support for removing a member > file; support for encrypting/decrypting member files. Unfortunately (?), the 2.6 zipfile module will have much of that, also, please take a look. Regards, Martin From martin at v.loewis.de Wed Mar 7 11:10:22 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 07 Mar 2007 11:10:22 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45EE82AC.2020103@develer.com> References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <45ED0371.5070108@v.loewis.de> <200703060848.04117.phil@riverbankcomputing.co.uk> <45ED39F0.2000907@v.loewis.de> <45EE82AC.2020103@develer.com> Message-ID: <45EE8F8E.3070409@v.loewis.de> Giovanni Bajo schrieb: > On 06/03/2007 10.52, Martin v. L?wis wrote: > >> I can't see that the barrier at contributing is high. > > I think this says it all. It now appears obvious to me that people > inside the "mafia" don't even realize there is one. Thus, it looks like > we are all wasting time in this thread, since they think there is > nothing to be changed. I can feel your pain that your (only) patch was unreviewed for three years. If it makes you happy to blame somebody feel free to, but I feel it is unfair to blame the single person that eventually *did* review and commit your patch. You may ask yourself why this specific patch was unreviewed for so long. My own explanation is that it is a highly complicated algorithm (as any kind of cryptographical algorithm), so nobody felt qualified to review it. I'm pretty sure it was *looked at* often (contrary to your remark in [1]), but none of the people reading it were qualified to comment. So if you want to work on encryption still, please go ahead. Regards, Martin [1] http://mail.python.org/pipermail/python-list/2004-November/293510.html From scott+python-dev at scottdial.com Tue Mar 6 06:59:47 2007 From: scott+python-dev at scottdial.com (Scott Dial) Date: Tue, 06 Mar 2007 00:59:47 -0500 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45ECFF37.6090901@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> Message-ID: <45ED0353.2090201@scottdial.com> As an outsider who has submitted a handful of patches and has always wanted to become more involved.. I would like to comment as I feel like I am the target audience in question. I apologize ahead of time if I am speaking out of place. Martin v. L?wis wrote: > Phil Thompson schrieb: >> 1. Don't suggest to people that, in order to get their patch reviewed, they >> should review other patches. The level of knowledge required to put together >> a patch is much less than that required to know if a patch is the right one. > > People don't *have* to review patches. They just can do that if they > want expedite review of their code. > While I understand that this tit-for-tat mechanism is meant to ensure participation, I believe in reality it doesn't, as the 400-some outstanding patches you referenced elswhere indicate. I can personally attest to having a patch that is over a year old with no "core developer" having any interest at all with the subject matter. And to be frank, nor did I really, but I saw a problem and was capable of solving it. My lack of caring about the patch means I am not going to beat people over the head to pay attention. This system is broken for someone like me (coder) that just wants to help out (non-coders). >> 2. Publically identify the core developers and their areas of expertise and >> responsibility (ie. which parts of the source tree they "own"). > > I doubt this will help. Much of the code isn't owned by anybody > specifically. Those parts that are owned typically find their patches > reviewed and committed quickly (e.g. the tar file module, maintained by > Lars Gust?bel). If nothing else, as an outsider there is no way to know why your patch gets ignored while others get swiftly dealt with. Any sort of information like this would at least provide more transparency in what may appear to be elitest processes. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From scott+python-dev at scottdial.com Tue Mar 6 13:20:17 2007 From: scott+python-dev at scottdial.com (Scott Dial) Date: Tue, 06 Mar 2007 07:20:17 -0500 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45ED57CA.7090101@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <45ED0353.2090201@scottdial.com> <45ED07EE.4060402@v.loewis.de> <79990c6b0703060320t23d4487rcc97ec69c669a8ee@mail.gmail.com> <45ED57CA.7090101@v.loewis.de> Message-ID: <45ED5C81.6070702@scottdial.com> Martin v. L?wis wrote: > Paul Moore schrieb: >> Here's a random offer - let me know the patch number for your patch, >> and I'll review it. > > Surprisingly (and I hope Scott can clarify that), I can't find anything. > Assuming Scott's SF account is "geekmug", I don't see any open patches > (1574068 was closed within 20 days by amk, last October). > Sadly the sf tracker doesn't let you search for "With comments by". The patch I was making reference to was 1410680. Someone else actually had wrote a patch that contained bugs and I corrected them. And with that, I was the last person to comment or review the patch in question. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From martin at v.loewis.de Wed Mar 7 12:10:34 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 07 Mar 2007 12:10:34 +0100 Subject: [Python-Dev] cross-compile patches In-Reply-To: <457607D9.70107@xs4all.nl> References: <4575A19F.2020609@xs4all.nl> <4575C4AF.7000803@v.loewis.de> <457607D9.70107@xs4all.nl> Message-ID: <45EE9DAA.6050905@v.loewis.de> Han-Wen Nienhuys schrieb: > 1608267 - added comment, asking for explanation. looks bogus. > 1608579 - problem analysis + solution looks good > 1507247 - I can reproduce the problem it tries to fix. Fix looks ok. > 1520879 - trivial fix; should go in. disregard the comment about relative --prefix > 1339673 - superseeded by #1597850 Thanks. Comming back to it only now, I added them into http://wiki.python.org/moin/PatchTriage Regards, Martin From jason.orendorff at gmail.com Wed Mar 7 12:52:25 2007 From: jason.orendorff at gmail.com (Jason Orendorff) Date: Wed, 7 Mar 2007 06:52:25 -0500 Subject: [Python-Dev] [ 1669539 ] Change (fix!) os.path.isabs() semantics on Win32 Message-ID: On 3/7/07, "Martin v. L?wis" wrote: > Terry Jones schrieb: > > I do think the behavior can be improved, and that it should be fixed, but > > at a place where other incompatible changes will also be being made, > > Indeed, 2.6 is such a place. Any feature release can contain > incompatible behavior, and any feature release did contain incompatible > behavior. Just look at the "porting to" sections of past whatsnew files. While we're at it, patch 1669539 makes a similar incompatible change to ntpath.isabs(). On Windows there are: - true relative paths, like Lib\ntpath.py - true absolute paths, like C:\Python25 and \\server\share - oddities, like C:ntpath.py and \Python25 isabs() is inconsistent about oddities: >>> ntpath.isabs(r'C:ntpath.py') False >>> ntpath.isabs(r'\Python25') True I don't think there's any logic behind this behavior. The current documentation for isabs() is: isabs(path) Return True if path is an absolute pathname (begins with a slash). The patch makes isabs(oddity) return False. I don't think existing code is a huge concern here. Google Code Search suggests that no one thinks about the oddities. Most existing code using isabs() has acceptable-but-slightly-odd behavior for oddities, and that kind of code would have different acceptable-but-slightly-odd behavior under the proposed change. And oddities are rare. The patch is incomplete (no docs) but ripe for a note of encouragement (or summary rejection) from a committer. -j From eric+python-dev at trueblade.com Wed Mar 7 14:03:29 2007 From: eric+python-dev at trueblade.com (Eric V. Smith) Date: Wed, 07 Mar 2007 08:03:29 -0500 Subject: [Python-Dev] Access to bits for a PyLongObject In-Reply-To: <45ED8A1A.60908@v.loewis.de> References: <45ECCA92.8090709@trueblade.com> <45ECF545.2080806@v.loewis.de> <45ED7091.3010705@trueblade.com> <45ED8A1A.60908@v.loewis.de> Message-ID: <45EEB821.9000001@trueblade.com> Martin v. L?wis wrote: > Ah, I had missed the point that it's just binary formatting that > you are concerned with (and then I missed that binary is "base 2", > rather than "sequence of bits") Apologies for not being clear. It's easy to forget that others don't share the context of something you've been immersed in. >> Having written all of this, I'm now thinking that Nick's suggestion of >> _PyLong_AsByteArray might be the way to go. I would use that for all of >> my formatting for longs. > > How would you do negative numbers, then? AsByteArray gives you two's > complement. _PyLong_Sign >> But I'd like PEP 3101 to be as efficient as possible, >> because once it's available I'll replace all of the '%' string >> formatting in my code with it. > > That is fine. However, don't trade efficiency for maintainability. > Keep encapsulation of types, this is what OO is for. Modularize > along with type boundaries. If that loses efficiency, come up with > interfaces that still modularize in that way but are efficient. > Don't "hack" to achieve performance. (Any other way I can formulate > the same objective :-?) Point taken. I currently have it using PyLong internals, just to get our tests to pass and so Pat can work on his part. As we still want to be a standalone module for a while, I'm going to modify the code to use AsByteArray and Sign to do the binary formatting only. When/if we integrate this into 3.0 (and 2.6, I hope), I'll look at adding __format__ to long, and possibly the other built in types. To do so we'll need to factor some code out to a library, because it doesn't make sense for all the built-in types to understand how to parse and operate on the format specifiers (the [[fill]align][sign][width][.precision][type] stuff). Thanks for your comments! Eric. From facundo at taniquetil.com.ar Wed Mar 7 14:10:16 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Wed, 7 Mar 2007 13:10:16 +0000 (UTC) Subject: [Python-Dev] Encouraging developers References: <20070305184614.GA19051@localhost.localdomain> Message-ID: A.M. Kuchling wrote: > FWIW, I have a related perception that we aren't getting new core > developers. These two problems are probably related: people don't get > patches processed and don't become core developers, and we don't have > enough core developers to process patches in a timely way. And so > we're stuck. > > Any ideas for fixing this problem? I think that there's a barrier entry: there's no place to ask for help on silly problems when you're trying to help (!). Let me explain my bad english wording, with an example. Yesterday night I started modifying socket.py and test_socket.py. "Of course", I said, "let's see if the tests pass ok *before* start changing anything". Went to ~/devel/reps/python/trunk/Lib, and made: $ python2.5 test/test_socket.py ... Wrong! Damn! Tried a lot of stuff... $ cd test $ python2.5 test_socket.py ... Wrong! $ python2.5 regrtest.py test_socket ... Wrong! $ python2.5 regrtest.py test_socket.py ... Wrong! $ python2.5 regrtest.py socket ... Wrong! And thousand more combinations. The best I could do is actually execute the tests, but python was getting the installed socket module, and not the repository socket module (even that I was in the same directory of the latter). I didn't know what to try. I was stuck. This never happened to me when working on Decimal. What went wrong in my head in the middle? I finally understood the problem, and build python from the repository, and made the tests from *this* python (actually, this was an easy step because I'm on Ubuntu, but *I* would be dead if working in Windows, for example). Ok. *Me*, that I'm not ashame of asking what I don't know, if I didn't resolve it I'd finally asked in python-dev. But how many people would have throw the towel withoug getting so far? How many people want to submit a patch, or even a bug, or finds a patch to review, but don't know how to do something and thinks that python-dev is not the place to ask (too high minds and experienced people and everything)? What I propose is a dedicated place (mailing list, for example), that is something like a place where you can go and ask the silliest questions about helping in the developing process. - How can I know if a patch is still open? - I found a problem, and know how to fix it, but what else need to do? - Found a problem in the docs, for this I must submit a patch or tell something about it? How? - I found an error in the docs, and fixed it, but I'm spanish speaker and my english sucks, can I submit a patch with bad wording or I must ask somebody to write it ok? Me, for example, has an actual question to this list: "How can I know, if I change something in the doc's .tex files, that I'm not broking the TeX document structure?". Just my two argentinian cents. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From fredrik at pythonware.com Wed Mar 7 14:34:49 2007 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 7 Mar 2007 14:34:49 +0100 Subject: [Python-Dev] splitext('.cshrc') References: <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com><20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de><20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de><20070306150459.GA8906@phd.pp.ru><5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com><5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <45EDD6B2.7060006@v.loewis.de> Message-ID: Martin v. L?wis wrote: > I never considered it an extension. Ask 10 people around you to see > what a leading dot on Unix in a file name means, and I would be > suprised if more than one answered "it separates the file name from > the extension". Most of them likely include "hidden file" in their > answer, and the rest (of those who know anything about it) will > say "dotfiles, not displayed by ls, and not included in globbing". 10 Unix programmers, perhaps. Windows users may disagree; after all, the Windows Explorer considers both foo.bar and .bar to be BAR files, and considers .py to be an unnamed Python file. From p.f.moore at gmail.com Wed Mar 7 14:42:02 2007 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Mar 2007 13:42:02 +0000 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45ED5C81.6070702@scottdial.com> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <45ED0353.2090201@scottdial.com> <45ED07EE.4060402@v.loewis.de> <79990c6b0703060320t23d4487rcc97ec69c669a8ee@mail.gmail.com> <45ED57CA.7090101@v.loewis.de> <45ED5C81.6070702@scottdial.com> Message-ID: <79990c6b0703070542w1479d8e4rdf30b1cc9a5f2a78@mail.gmail.com> On 06/03/07, Scott Dial wrote: > Martin v. L?wis wrote: > > Paul Moore schrieb: > >> Here's a random offer - let me know the patch number for your patch, > >> and I'll review it. > > > > Surprisingly (and I hope Scott can clarify that), I can't find anything. > > Assuming Scott's SF account is "geekmug", I don't see any open patches > > (1574068 was closed within 20 days by amk, last October). > > > > Sadly the sf tracker doesn't let you search for "With comments by". The > patch I was making reference to was 1410680. Someone else actually had > wrote a patch that contained bugs and I corrected them. And with that, I > was the last person to comment or review the patch in question. OK, as promised, I've reviewed it. I've recommended some actions from the original poster, or if they aren't forthcoming, then rejection. If you (Scott) want to pick this up, I'd recommend: 1. Open a new patch, with your recommended changes. 2. Address the comments made against Tony's patch, in yours. 3. Add a recommendation to Tony's patch that it be closed in favour of yours. Wait and/or canvas further opinion. That's about all I can do - to get the code *into* Python (if it's appropriate - remember that I've recommended rejection!) then you need someone with commit privileges to apply it. On the other hand, what I've done is similar to what you did - comment on someone else's patch. It seems relevant to me that the original poster (Tony Meyer) hasn't felt strongly enough to respond on his own behalf to comments on his patch. No disrespect to Tony, but I'd argue that the implication is that the patch should be rejected because even the submitter doesn't care enough to respond to comments! Paul. From fuzzyman at voidspace.org.uk Wed Mar 7 14:36:32 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 07 Mar 2007 13:36:32 +0000 Subject: [Python-Dev] splitext('.cshrc') (and encouraging developers) In-Reply-To: <45EE77ED.2050001@v.loewis.de> References: <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <5.1.1.6.0.20070306162907.02b7e2a0@sparrow.telecommunity.com> <17901.63078.595584.18386@terry-jones-computer.local> <45EE77ED.2050001@v.loewis.de> Message-ID: <45EEBFE0.1010106@voidspace.org.uk> Martin v. L?wis wrote: > Terry Jones schrieb: > >> I do think the behavior can be improved, and that it should be fixed, but >> at a place where other incompatible changes will also be being made, >> > > Indeed, 2.6 is such a place. Any feature release can contain > incompatible behavior, and any feature release did contain incompatible > behavior. Just look at the "porting to" sections of past whatsnew files. > > Now that Martin has made the changes and rejected the original patch I remember that it was written by me. This raises a point which is related to the 'encouraging developers' thread. I created this patch as part of a Python bug day over a year ago. The bug day was trumpeted as being a good place for 'newbies' who wanted to contribute to Python could start, and implied that some kind of mentoring would be in place. I enthusiastically turned up on IRC and asked how to join in. I was told (more or less) to 'pick a bug and fix it'. That one seemed to be the only one on the list I could tackle alone (and I still managed to screw it up it seems!). I submitted the patch to sourceforge, posted to IRC that I had done this - and then kind of hung around a while waiting to see what would happen. There was no response and after a while of waiting and nothing much else happening either I returned to working on my own projects... The bug days do seem like an opportunity to involve new developers - but some kind of active engagement would seem appropriate... All the best, Michael Foord > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > From g.brandl at gmx.net Wed Mar 7 16:32:27 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 07 Mar 2007 16:32:27 +0100 Subject: [Python-Dev] Bug Days Message-ID: Michael Foord schrieb: > This raises a point which is related to the 'encouraging developers' thread. > > I created this patch as part of a Python bug day over a year ago. The > bug day was trumpeted as being a good place for 'newbies' who wanted to > contribute to Python could start, and implied that some kind of > mentoring would be in place. > > I enthusiastically turned up on IRC and asked how to join in. I was told > (more or less) to 'pick a bug and fix it'. That one seemed to be the > only one on the list I could tackle alone (and I still managed to screw > it up it seems!). I submitted the patch to sourceforge, posted to IRC > that I had done this - and then kind of hung around a while waiting to > see what would happen. > > There was no response and after a while of waiting and nothing much else > happening either I returned to working on my own projects... > > The bug days do seem like an opportunity to involve new developers - but > some kind of active engagement would seem appropriate... If it was the last bug day you speak about, this was a bit unfortunate since it was only me and Tim who were present for a longer time, and not busy with some server maintenance tasks. I guess I just picked the wrong day ;) Georg From jcarlson at uci.edu Wed Mar 7 17:08:02 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 07 Mar 2007 08:08:02 -0800 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45EDB1F3.6060801@v.loewis.de> References: <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <45EDB1F3.6060801@v.loewis.de> Message-ID: <20070307080541.74AD.JCARLSON@uci.edu> "Martin v. L?wis" wrote: > Phillip J. Eby schrieb: > > I consider it correct, or at the least, don't think it should be > > changed, as it would make the behavior more difficult to reason about > > and introduce yet another thing to worry about when writing > > cross-version code. > > Now it's becoming difficult: several people in favor, some opposed... What about changing the semantics of splitext and creating a new function (available on all platforms) that does what the Windows version currently does? For people who want the one semantic on all platforms, they can pick their semantic explicitly. Alternatively, offer a flag argument in splitext to differentiate between the two cases. - Josiah From p.f.moore at gmail.com Wed Mar 7 17:06:08 2007 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Mar 2007 16:06:08 +0000 Subject: [Python-Dev] Bug Days In-Reply-To: References: Message-ID: <79990c6b0703070806i76ac986cy6c277eb288371f85@mail.gmail.com> On 07/03/07, Georg Brandl wrote: > If it was the last bug day you speak about, this was a bit unfortunate since it > was only me and Tim who were present for a longer time, and not busy with some > server maintenance tasks. I guess I just picked the wrong day ;) Agreed - the earlier bug day which I attended seemed to go pretty well, with a reasonable number of people to help with my (dumb) questions, and a fairly quick followup of someone (AMK, if I recall) going through the various patches/fixes and committing stuff. It seems to me that managing a good bug day requires a fair bit of effort - but when it works, it's a good way to get people involved. Paul. From g.brandl at gmx.net Wed Mar 7 17:10:01 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 07 Mar 2007 17:10:01 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> Message-ID: Facundo Batista schrieb: > How many people want to submit a patch, or even a bug, or finds a patch > to review, but don't know how to do something and thinks that python-dev > is not the place to ask (too high minds and experienced people and > everything)? > > What I propose is a dedicated place (mailing list, for example), that is > something like a place where you can go and ask the silliest questions > about helping in the developing process. In my experience, direct communication, e.g. via IRC channels, is much less "intimidating" for newcomers than a publicly read and archived mailing list. That said, I'm in #python-dev on Freenode most of the time and I'd be happy to help out new developers (as soon as the questions are still so simple for me to answer ;). Of course, the channel would have to be made an "official" Python development tool and advertised on e.g. the website. Also, it couldn't hurt if some of the other devs would frequent it too, then :) Georg From g.brandl at gmx.net Wed Mar 7 17:21:02 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 07 Mar 2007 17:21:02 +0100 Subject: [Python-Dev] Bug Days In-Reply-To: <79990c6b0703070806i76ac986cy6c277eb288371f85@mail.gmail.com> References: <79990c6b0703070806i76ac986cy6c277eb288371f85@mail.gmail.com> Message-ID: Paul Moore schrieb: > On 07/03/07, Georg Brandl wrote: >> If it was the last bug day you speak about, this was a bit unfortunate since it >> was only me and Tim who were present for a longer time, and not busy with some >> server maintenance tasks. I guess I just picked the wrong day ;) > > Agreed - the earlier bug day which I attended seemed to go pretty > well, with a reasonable number of people to help with my (dumb) > questions, and a fairly quick followup of someone (AMK, if I recall) > going through the various patches/fixes and committing stuff. > > It seems to me that managing a good bug day requires a fair bit of > effort - but when it works, it's a good way to get people involved. Speaking of which -- is anyone interested in holding a Bug Day in the near future? Georg From barry at python.org Wed Mar 7 17:36:07 2007 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Mar 2007 11:36:07 -0500 Subject: [Python-Dev] PEP 364, Transitioning to the Py3K standard library In-Reply-To: <17902.11922.341345.349583@montanaro.dyndns.org> References: <6EBA9F60-ECBF-426B-AC60-E1044AEDE938@python.org> <17902.11922.341345.349583@montanaro.dyndns.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mar 6, 2007, at 10:16 PM, skip at pobox.com wrote: > Does the proposed renaming include any restructuring (e.g. making > hierarchies out of all or part of the stdlib where none existed > before)? It > wasn't obvious to me. For example, might there be a restructuring > of the > entire stdlib? Skip, that's really for PEP 3108 to decide, not really my PEP 364. I believe that the mechanism I describe in PEP 364 could support a restructured stdlib, at least as far as renamed modules go. Basically, it's just a hack to support aliasing an import of one name (the old name) through the import of another name (the new name). - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRe7p+HEjvBPtnXfVAQKFDQP9HSL0o6mWj4iNMKJNjVy1qcbK1HCHbDVv JlnqPlegRiZR+KwDhPiGLD8epepuZ3d92F4ptORcTsiulC2H/nIAiYTRJWdqFj7I sCIXZkRc0lFskxEf4p9t2aOHOWq6D9VYrcHeZ8L5KYxck35M8KfGVmSWvbSoIXBs qx+nFWrBVwA= =rrCG -----END PGP SIGNATURE----- From terry at jon.es Wed Mar 7 17:38:48 2007 From: terry at jon.es (Terry Jones) Date: Wed, 7 Mar 2007 17:38:48 +0100 Subject: [Python-Dev] Bug Days In-Reply-To: Your message at 16:06:08 on Wednesday, 7 March 2007 References: <79990c6b0703070806i76ac986cy6c277eb288371f85@mail.gmail.com> Message-ID: <17902.60056.704449.104205@terry-jones-computer.local> Why not offer a Python patching tutorial at the next US/Euro PyCon? It seems like there's plenty that could be taught. I'd attend. I'd suggest that that specific tutorial be offered free, or be paid for by sponsors. Similarly, the first day of the post-PyCon sprints could have a group learning about the patching process. While I'd probably not attend a "sprint" (because I'd imagine I'd be doing more harm than good) I'd certainly be interested in showing up for a day explicitly aimed at those interested in learning / sprinting on patching python. Terry From martin at v.loewis.de Wed Mar 7 17:45:26 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 07 Mar 2007 17:45:26 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <79990c6b0703070542w1479d8e4rdf30b1cc9a5f2a78@mail.gmail.com> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <45ED0353.2090201@scottdial.com> <45ED07EE.4060402@v.loewis.de> <79990c6b0703060320t23d4487rcc97ec69c669a8ee@mail.gmail.com> <45ED57CA.7090101@v.loewis.de> <45ED5C81.6070702@scottdial.com> <79990c6b0703070542w1479d8e4rdf30b1cc9a5f2a78@mail.gmail.com> Message-ID: <45EEEC26.1050802@v.loewis.de> Paul Moore schrieb: > 1. Open a new patch, with your recommended changes. I'd like to second this. If you are creating a patch by responding in a comment, it likely gets ignored. > 2. Address the comments made against Tony's patch, in yours. > 3. Add a recommendation to Tony's patch that it be closed in favour of > yours. This also sounds good. Regards, Martin From rhamph at gmail.com Wed Mar 7 17:47:37 2007 From: rhamph at gmail.com (Adam Olsen) Date: Wed, 7 Mar 2007 09:47:37 -0700 Subject: [Python-Dev] Adding socket timeout to urllib2 In-Reply-To: References: Message-ID: On 3/6/07, Guido van Rossum wrote: > On 3/6/07, Adam Olsen wrote: > > I think calling it "timeout" in the API is fine. The documentation > > can then clarify that it's an idle timeout, except it only applies > > when blocked in a network operation. > > Since "idel timeout" is not a commonly understood term it would be > even better if it was explained without using it. I disagree, but meh, I'll stick to my http://pink.bikeshed.org/ -- Adam Olsen, aka Rhamphoryncus From janssen at parc.com Wed Mar 7 18:41:18 2007 From: janssen at parc.com (Bill Janssen) Date: Wed, 7 Mar 2007 09:41:18 PST Subject: [Python-Dev] Adding socket timeout to urllib2 In-Reply-To: <17902.11360.91118.646113@montanaro.dyndns.org> References: <17902.11360.91118.646113@montanaro.dyndns.org> Message-ID: <07Mar7.094120pst."57996"@synergy1.parc.xerox.com> > > Guido> Since "idel timeout" is not a commonly understood term it would > Guido> be even better if it was explained without using it. > > I think it's commonly understood, but it doesn't mean what the socket > timeout is used for. It's how long a connection can be idle (the client > doesn't make a request of the server) before the server will close the > connection. Let me chime it to support Skip here. "idle timeout" seems a really bad name for the new timeout feature under discussion. It's not about idling; it's about how long you'll wait for something you're expecting from the other end of the connection before assuming that something has gone wrong over there (infinite loop, etc.) and aborting the connection. OTOH, "idling" is when the TCP connection exists, but neither side wants anything from the other side. If you really need a name other than "timeout" (which seems fine to me), how about "waiting-with-mild-trepidation-timeout"? Bill From python at rcn.com Wed Mar 7 18:56:45 2007 From: python at rcn.com (Raymond Hettinger) Date: Wed, 7 Mar 2007 12:56:45 -0500 (EST) Subject: [Python-Dev] Fwd: Re: Encouraging developers Message-ID: <20070307125645.BAP02253@ms09.lnh.mail.rcn.net> [Scott Dial] >While I understand that this tit-for-tat mechanism is meant to ensure >participation, I believe in reality it doesn't, as the 400-some >outstanding patches you referenced elswhere indicate. . . . >If nothing else, as an outsider there is no way to know why your patch >gets ignored while others get swiftly dealt with. Any sort of >information like this would at least provide more transparency in what >may appear to be elitest processes. This thread is getting dull and is wasting everyone's time. If you want to contribute, then contribute. If you want to rant about elitism, bag on volunteer developers, or expound conspiracy theories about insiders and outsiders, then your scottdial.com blog would make a better forum. Raymond From amk at amk.ca Wed Mar 7 19:47:40 2007 From: amk at amk.ca (A.M. Kuchling) Date: Wed, 7 Mar 2007 13:47:40 -0500 Subject: [Python-Dev] Bug Days In-Reply-To: References: Message-ID: <20070307184740.GA20103@localhost.localdomain> On Wed, Mar 07, 2007 at 04:32:27PM +0100, Georg Brandl wrote: > If it was the last bug day you speak about, this was a bit unfortunate > since it > was only me and Tim who were present for a longer time, and not busy with > some > server maintenance tasks. I guess I just picked the wrong day ;) Bug days would be a good idea, but we'd need to firmly commit to having several committers around for the duration, and to helping people with bugs instead of working on our own things. --amk From rrr at ronadam.com Wed Mar 7 20:58:26 2007 From: rrr at ronadam.com (Ron Adam) Date: Wed, 07 Mar 2007 13:58:26 -0600 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45EE79BF.9020100@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <200703052116.48035.phil@riverbankcomputing.co.uk> <45ED00FC.6050406@v.loewis.de> <200703060834.06715.phil@riverbankcomputing.co.uk> <45EDD141.20605@ronadam.com> <45EE603E.1000807@ronadam.com> <45EE79BF.9020100@v.loewis.de> Message-ID: <45EF1962.4050404@ronadam.com> Martin v. L?wis wrote: > Ron Adam schrieb: >> But the tracker needs to be able to actually track the status of >> individual items for this to work. Currently there's this huge list >> and you have to either wade though it to find out the status of each >> item, or depend on someone bring it to your attention. > > Well, the tracker does have a status for each individual report: open or > closed. open means "needs more work", closed means "no more work > needed". So don't wade through a report - just assume that if it is > open, it needs work. > > Regards, > Martin I should have said... ... the status of individual items *in more detail* for this to work *well*. Sorry, Ron From tjreedy at udel.edu Wed Mar 7 21:16:44 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 7 Mar 2007 15:16:44 -0500 Subject: [Python-Dev] splitext('.cshrc') References: <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <20070306150459.GA8906@phd.pp.ru><45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru><45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <5.1.1.6.0.20070306162907.02b7e2a0@sparrow.telecommunity.com> <45EDE899.9020709@v.loewis.de><20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <45EE773F.20003@v.loewis.de> Message-ID: ""Martin v. L?wis"" wrote in message news:45EE773F.20003 at v.loewis.de... | glyph at divmod.com schrieb: | Maybe you aren't grounded so much in Unix history. It really feels | wrong that a dotfile is considered as having an extension. I have not been on *nix for nearly 20 years and I agree that the current behavior is a bug. If I were to prepare a file list like name .ext I would be annoyed is .cshrc was stuck in the ext column instead of the name column. tjr From tjreedy at udel.edu Wed Mar 7 21:26:45 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 7 Mar 2007 15:26:45 -0500 Subject: [Python-Dev] Encouraging developers References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <45ED0371.5070108@v.loewis.de> <200703060848.04117.phil@riverbankcomputing.co.uk><45ED39F0.2000907@v.loewis.de> <45EE82AC.2020103@develer.com> <45EE8F8E.3070409@v.loewis.de> Message-ID: ""Martin v. L?wis"" wrote in message news:45EE8F8E.3070409 at v.loewis.de... >You may ask yourself why this specific patch was unreviewed for >so long. My own explanation is that it is a highly complicated >algorithm (as any kind of cryptographical algorithm), so nobody >felt qualified to review it. I'm pretty sure it was *looked at* >often (contrary to your remark in [1]), but none of the people >reading it were qualified to comment. I have looked at nunerous patches without commenting because I could not or did not do a full review. You and Neal have convinced me that I have been too shy and that even little comments like 'the doc revision looks good' might help. tjr From collinw at gmail.com Wed Mar 7 21:48:02 2007 From: collinw at gmail.com (Collin Winter) Date: Wed, 7 Mar 2007 14:48:02 -0600 Subject: [Python-Dev] Tracker rights Message-ID: <43aa6ff70703071248g704f2d25yed5adbf0b87b3103@mail.gmail.com> Hi, Could I be granted rights to the SF tracker? I'm going through and reviewing some older patches, and I'd like to be able to close invalid/rejected patches (eg, 1492509) and upload changed patches. My SF username is collinwinter. Thanks, Collin Winter From jcarlson at uci.edu Wed Mar 7 22:21:36 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 07 Mar 2007 13:21:36 -0800 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45EDC618.2070402@develer.com> References: <20070305175116.749D.JCARLSON@uci.edu> <45EDC618.2070402@develer.com> Message-ID: <20070307081037.74B0.JCARLSON@uci.edu> Giovanni Bajo wrote: > On 3/6/2007 3:11 AM, Josiah Carlson wrote: > > Giovanni Bajo wrote: > >> I think this should be pushed to its extreme consequences for the standard > >> library. Patching the standard library requires *much less* knowledge than > >> patching the standard core. Basically, almost any Python developer in the wild > >> can quickly learn a module and start patching it in a few days/weeks -- still, > >> the stdlib is a total mess of outdated and broken modules. > > > > Sometimes code is correct. Sometimes code is complete. Just because a > > module hasn't been significantly altered or updated recently doesn't > > mean that it is outdated or broken. > > You can't decide this unless you are a maintainer of that module. If a > programmer felt the urge to pack up a patch and submit it to SF.NET, he > probably has an idea. It might be a good idea. It might not be the best, > but at least it's an idea. It might even be just a bugfix. > > If you see the patch, know the module well, and can express a judgement, > you just need to review it and discuss it. But I really can't see what's > *worse* than getting a patch unreviewed and unapplied. And who is advocating for patches to be unreviewed and unapplied? I'm not. If *anyone* is advocating such a position, then I claim that they are idiots. > > Asyncore, for example, is more or > > less feature complete for a minimal asynchronous socket framework. Could > > it gain more features (like good SSL support) or have better error > > handling? Of course. But it passes the test suite (via test_asynchat), > > so I would consider it *not* broken. > > Because you are the maintainer and you know well. I used asyncore once, > and I felt it was incomplete. So I went looking for something else. > That's fine. You probably know asyncore very well, so your judgement is > important and you'll be reviewing the patches and vetoing them if you > don't like. I'll tell you the biggest problem with asyncore: there are few *good* samples of asyncore usage in the wild, and there isn't even one really in the standard library (smtpd does OK, but it could be better). The asynchat module is supposed to add *just enough* functionality to get people started, but its lack of a sample collect_incoming_data() and found_terminator() that generally do the right thing, are sticking points for many people. Among the changes I'm going to be pushing for is the inclusion of sample implementations of those two methods in asynchat (as well as a fix for when a string terminator is longer than ac_in_buffer_size). If asynchat had them in the summer of 2001, I probably wouldn't have more or less reimplemented my own asynchronous sockets framework that summer and fall 3 different times. > But there are modules without maintainers. We agree that every module > should have a maintainer, and that's fine. But you seem to ignore the > fact that we ought to find a solution for modules and packages without > maintainers. What is your proposed solution, once if we agree that the > current state of affairs suck? You can't force people to step up for > maintainership of course, so you have to find something in the middle. > And the best way to encourage someone to maintain a package is... > accepting their patches. And that's what I think is bull. Whether or not we want or need maintainers for module or package X is independant of the fact that user Y has submitted a patch for it. If they say, "I would like to become maintainer of package X", ok, if they are better than not having a maintainer, great. But to ask them to be a maintainer of an unmaintained package because they submitted a patch? That's a bit like inviting everyone who has ever programmed to be an IEEE or ACM fellow. That's nice and inclusive, but not the way you gather together people who can and will develop quality code. > > Just because a patch doesn't break a module, doesn't mean the patch > > should be applied. Take, for example, any one of the patches currently > > offered against asyncore. One of them more or less changes the internal > > structure of the module for no other reason than to (in my opinion) be > > gratuitous. Certainly it has some good ideas, but it would be better to > > adapt portions rather than take it completely - which is what you are > > suggesting, and which is what would have happened due to the absence of > > an asyncore maintainer (until I took it up). > > Engineering is a matter of tastes and opinions, more often than not. > Once you are a maintainer, your taste wins. But if there are no > maintainers, I prefer to trust someone who wasted his time to produce a > patch, rather than just blatantly ignore his job. At least, he had an > urge and produced some code. He did put forward his opinion. Certainly opinions differ, which is what this is. However, even high-quality patches (like the one produced by Larry Hastings for string concatenation and views) are rejected because the functionality is deemed undesireable. If *any* of the recent changes that were rejected were engineered, it's that one. But not all patches are engineered, and certainly not all patches implement functionality that is desireable. > Moreover, you're debating my arguments speaking of refactoring patches > or new features. You're ignoring that bug-fixes go often unreviewed, and > sometimes even some basic new feature with very small patch which really > can't do *much* harm. I disagree. Not every feature that a user thinks should exist, should. Too many people are often too close to the problem to understand that their solution simply sucks. I've written more code than I care to admit that solved a problem in a particular way, which I built upon over months, that I realized later was a *horrible* design. Maybe this is a similar case. I personally don't find Python's development process as "broken" as some would claim, primarily because Python works for me (and I'm using the no longer maintained 2.3.5). Yes, I would really like a subclassable mmap. Yes, I would really like binascii.long2bin() and bin2long(). > For instance, I did a small patch to zipfile which adds decryption > support (to open zipfile crypted with password, with the standard zip > encryption algorithm). It's still unreviewed somewhere in SF. I wonder > how much harm could have be done if that patch had been simply applied > as-is years ago. A little game: without looking at this patch of mine, > how much are you willing to bet that committing that patch is going to > cause pain the Python community and other stdlib maintainers, to cause > headaches like deprecations because of broken interfaces, and whatnot? It's a toss-up. Considering my limited knowledge on zipfile handling, I would say no. In the realm of language design, standard library modification, etc., when it comes to Python, a conservative approach is the best approach. Changing things, just to change them, is a horrible idea. Changing things, because the change is positive with no overtly negative effects, is a great idea. Never mind that classic zipfile encryption is broken when there are more than 3 files in the archive, and there are 2 incompatible encryption rutines in modern zip variants, and one could make the claim that accessing files that claim to be secure, but really aren't, is a misfeature (which is also why the rotor module was removed). > In other words, I maintain that if my patch had been applied 3 years ago > without even looking at it, zipfile would be a better module now. It > might not be the best possible solution, it might have some little bug, > it might expose a new interface which is incomplete and might be > deprecated in the future if more work on encryption support will be > done. Heck, I don't even remember the patch myself. But *meanwhile*, I > counted several requests on comp.lang.python for such a feature in these > years, and my patch is simply getting dust. And you are entitled to your opinion. I think you are full of it, and thankfully, in this context, other people tend to agree that applying user patches without them being reviewed is a foolish idea. > Remember that also maintainers do mistakes, so it really can't be much > worse than just ignoring any patch. I believe that ignoring patches is better than accepting without review. Then again, I've received and reviewed some *horrible* patches, and am pessimistic about the quality of submissions until the submitter has proven themselves. Also, I'm a jerk. - Josiah From python at rcn.com Wed Mar 7 22:23:37 2007 From: python at rcn.com (Raymond Hettinger) Date: Wed, 7 Mar 2007 16:23:37 -0500 (EST) Subject: [Python-Dev] Tracker rights Message-ID: <20070307162337.BAP73875@ms09.lnh.mail.rcn.net> [Collin Winter]> >Could I be granted rights to the SF tracker? Done. Raymond From brett at python.org Wed Mar 7 22:36:13 2007 From: brett at python.org (Brett Cannon) Date: Wed, 7 Mar 2007 13:36:13 -0800 Subject: [Python-Dev] PEP 364, Transitioning to the Py3K standard library In-Reply-To: References: <6EBA9F60-ECBF-426B-AC60-E1044AEDE938@python.org> Message-ID: On 3/6/07, Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > >> Supported Renamings > >> =================== > >> > >> There are at least 4 use cases explicitly supported by this PEP: > >> > >> - - Simple top-level package name renamings, such as ``StringIO`` to > >> ``stringio``; > >> > >> - - Sub-package renamings where the package name may or may not be > >> renamed, such as ``email.MIMEText`` to ``email.mime.text``; > >> > >> - - Extension module renaming, such as ``cStringIO`` to > >> ``cstringio``; > >> > > > > This feels a little misleading. The renaming is not restricted to > > extension modules but also works with any top-level module. > > Possibly. It's just a use case that illustrates the solution isn't > tied to pure-Python modules. But I'm not wedded to making this > distinction in the PEP. > > >> Third party package renaming is also supported, via several public > >> interfaces accessible by any Python module. > >> > > > > I guess a .pth file could install the mappings for the third-party > > modules. > > How would that work? .pth files don't execute arbitrary Python code, It does through imports, though. > but it seems like that would be necessary to make this work. OTOH, > your suggestion does make me think that it might be possible to > piggyback the module name mappings onto .pth file processing. It > already understand the 'import' command, so why not a 'rename' command? > Not needed. Just put your mapping code in a module in your package and have a .pth file import that module. > >> ``.mv`` files can appear anywhere in the file system, and there is a > >> programmatic interface provided to parse them, and register the > >> remappings inside them. By default, when Python starts up, all the > >> ``.mv`` files in the ``oldlib`` package are read, and their > >> remappings > >> are automatically registered. This is where all the module > >> remappings > >> should be specified for top-level Python 2.x standard library > >> modules. > >> > > > > I personally prefer to just have this info in a dict with a specified > > format for the dict. I think that should at least be an Open Issue or > > address why a new file format was chosen over that solution (don't > > remember what you said at PyCon =). > > Think about what that big dictionary would look like. It would > essentially be like a text file with mappings one per line, but you'd > have a lot of extra goo on each line (quotes, colons, and commas). I > thought it was important to decentralize the mappings and then to put > them in the simplest format that would serve the purpose. > Right, but I am just not a big special-purpose format guy. =) Plus mappings for third-parties can be kept to a single file for the module they import by keeping the dict in that file. > > Probably should mention that sys.path_importer_cache is then set for > > the magic string. > > Yep. > > > There is no mention of what the sys.meta_path importer/loader is > > used for. > > Good point. > > >> Programmatic Interface > >> ====================== > >> > >> Several methods are added to the ``sys.stdlib_remapper`` object, > >> which > >> third party packages can use to register their own remappings. > > > > Is this really necessary? It might be helpful if this use of mapping > > old to new names gets picked up and used heavily, but otherwise this > > might be overkill. You can easily get the object from > > sys.path_importer_cache as long as you know the magic string used on > > sys.path. > > So, I was thinking that third party packages could use the same > mechanism to provide their own remappings to PEP 8 names. Certainly > if that's something that third party packages want to do, it would be > better to use some common Python API to do that rather than have them > bake their own (or worse, avoid PEP 8-ifying their module names > because of backward compatibility concerns). > > If we agree that's a use case we should support, then either we > support this programmatically or we don't expose the API and do > something automatic when the package is imported. I'm of two > different minds on this. Probably the way I'd rather see this work > is for a package to include a package.mv file in its package > directory, and that file would get automatically loaded when the > package was first imported. > I guess I would want to minimize the API in case we decide to directly integrate this into import itself so we don't have to have both a meta_path and path_importer_cache object. I think the solution works for PEP 3108 and the mail package but I think I would a more integrated solution if we went all out in terms of providing the support outside of the stdlib (at least eventually). > If people really don't like the separate data file, and really want > to just use a Python dictionary, then the way to keep the > decentralization would be to reserve a __*__ symbol in the package > namespace that contained this remapping. Something like __renames__ > = {...}. > I think the .pth file solution alleviates this need. > > If you are going to have the object accessible from outside > > sys.path_importer_cache I would also add an attribute that contains > > the magic string used on sys.path. > > Of course, that magic string is also available via oldlib._magic. I > tend to think it won't be necessary, but if people really want it, I > can see putting the magic string as an attribute on the > OldStdlibLoader object, e.g. sys.stdlib_remapper.importer_cache_key. > Yeah, that was what I was thinking of. When you manipulate sys.path you will need to be aware of it and what it is named. -Brett From greg.ewing at canterbury.ac.nz Wed Mar 7 23:28:59 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 08 Mar 2007 11:28:59 +1300 Subject: [Python-Dev] Adding socket timeout to urllib2 In-Reply-To: <07Mar7.094120pst.57996@synergy1.parc.xerox.com> References: <17902.11360.91118.646113@montanaro.dyndns.org> <07Mar7.094120pst.57996@synergy1.parc.xerox.com> Message-ID: <45EF3CAB.6070105@canterbury.ac.nz> Bill Janssen wrote: > If you really need a name other than "timeout" (which seems fine to > me), how about "waiting-with-mild-trepidation-timeout"? Something like "response timeout" might be more descriptive. -- Greg From bjourne at gmail.com Wed Mar 7 23:46:55 2007 From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=) Date: Wed, 7 Mar 2007 23:46:55 +0100 Subject: [Python-Dev] PEP 364, Transitioning to the Py3K standard library In-Reply-To: <6EBA9F60-ECBF-426B-AC60-E1044AEDE938@python.org> References: <6EBA9F60-ECBF-426B-AC60-E1044AEDE938@python.org> Message-ID: <740c3aec0703071446k709cd3bdvbdabd0805abb5fdb@mail.gmail.com> > When Python's import machinery is initialized, the oldlib package is > imported. Inside oldlib there is a class called ``OldStdlibLoader``. > This class implements the PEP 302 interface and is automatically > instantiated, with zero arguments. The constructor reads all the > ``.mv`` files from the oldlib package directory, automatically > registering all the remappings found in those ``.mv`` files. This is > how the Python 2.x standard library is remapped. Will not reading all those .mv files add a lot of overhead? Disk seeks is not cheap. -- mvh Bj?rn From titus at caltech.edu Wed Mar 7 18:47:59 2007 From: titus at caltech.edu (Titus Brown) Date: Wed, 7 Mar 2007 09:47:59 -0800 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45EE8F8E.3070409@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <45ED0371.5070108@v.loewis.de> <200703060848.04117.phil@riverbankcomputing.co.uk> <45ED39F0.2000907@v.loewis.de> <45EE82AC.2020103@develer.com> <45EE8F8E.3070409@v.loewis.de> Message-ID: <20070307174759.GF29575@caltech.edu> On Wed, Mar 07, 2007 at 11:10:22AM +0100, "Martin v. L?wis" wrote: -> Giovanni Bajo schrieb: -> > On 06/03/2007 10.52, Martin v. L?wis wrote: -> > -> >> I can't see that the barrier at contributing is high. -> > -> > I think this says it all. It now appears obvious to me that people -> > inside the "mafia" don't even realize there is one. Thus, it looks like -> > we are all wasting time in this thread, since they think there is -> > nothing to be changed. Hi, I just wanted to interject -- when I used the word "mafia", I meant it in this sense: """ Informal. A tightly knit group of trusted associates, as of a political leader: [He] is one of the personal mafia that [the chancellor] brought with him to Bonn. """ (Martin, I certainly didn't intend to offend anyone by implying that they were part of a criminal organization. ;) I have a longer explanation of my view in the blog entry, and anyway I don't want to belabor my own experiences too much, but I would like to point out three things: * there's definitely a group of "trusted associates" -- committers, perhaps? -- and it's not at all clear to outsiders like me how new features, old patches, and old bugs are dealt with. * python-dev is an all-volunteer community. In true open-source fashion, that means that it's incumbent upon people who HAVE problems/issues with a process (like me, and Giovanni) to propose solutions that either someone wants to implement, or that we can implement. I certainly don't expect any of the committers to put tons of effort into a new initiative. * Much of the discussion on this issue of encouraging developers comes down to communicating better with non-python-dev people. Unless someone is already doing it, I will try to write a summary of the last few days of discussion and post it for review. The idea would be to provide a simple one stop wiki page for people who want to contribute. cheers, --titus From andrew.mackeith at abaqus.com Mon Mar 5 21:25:30 2007 From: andrew.mackeith at abaqus.com (Andrew MacKeith) Date: Mon, 05 Mar 2007 15:25:30 -0500 Subject: [Python-Dev] PCBuild8 In-Reply-To: <45EC5EED.1050507@v.loewis.de> References: <87lkibd1od.fsf@valverde.peloton> <45EC5EED.1050507@v.loewis.de> Message-ID: <45EC7CBA.5050301@abaqus.com> Martin v. L?wis wrote: > David Abrahams schrieb: > >> I tried building with MS Visual Studio 2005 from PCBuild8/pcbuild.sln, >> and for me it fails miserably. The first major complaint comes when >> linking pythoncore, where the _init_types symbol can't be found. On >> the other hand, allowing MSVS to convert PCBuild/pcbuild.sln works >> okay. Am I missing something? >> > > Yes, it doesn't work in Python 2.5 as released. This specific problem > has been fixed in the trunk and the 2.5 branch several months ago, > so I recommend to use either of these (or use VS 2003 to build the > released version). > Is there a scheduled date for the release of Python-2.5.1 ? I presume that the PCBuild8 directory should be used when building in 64 bit on Win64. Andrew > HTH, > Martin > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/andrew%40mackeith.net > > From archive at voidspace.org.uk Sat Mar 3 01:03:23 2007 From: archive at voidspace.org.uk (Michael Foord) Date: Sat, 03 Mar 2007 00:03:23 +0000 Subject: [Python-Dev] with_traceback In-Reply-To: <45E68F93.2080607@canterbury.ac.nz> References: <20070301015824.17852.680861297.divmod.quotient.1162@ohm> <2CDD1A79-8D9B-463B-BC69-0FDEA3EC27BF@fuhm.net> <45E66DDA.8090406@voidspace.org.uk> <45E68F93.2080607@canterbury.ac.nz> Message-ID: <45E8BB4B.6040205@voidspace.org.uk> Greg Ewing wrote: > Michael Foord wrote: > >> With the >> proposed changes, modules that do this would *continue* to work, surely >> ? >> > > Probably, but it might mean they were no longer thread > safe. An exception caught and raised in one thread would > be vulnerable to having its traceback clobbered by > another thread raising the same instance. > Right - but that would still be *no worse* than the current situation where that information isn't available on the instance. The current patterns would continue to work unchanged, but the new information wouldn't be available because a single instance is being reused. > There's also the possibility of a traceback unexpectedly > kept alive causing GC problems -- cycles, files not > closed when you expect, etc. > That *could* be a problem, although explicitly closing files is always a good practise :-) Michael Foord > -- > Greg > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > From 2007a at usenet.alexanderweb.de Thu Mar 8 00:03:14 2007 From: 2007a at usenet.alexanderweb.de (Alexander Schremmer) Date: Thu, 08 Mar 2007 00:03:14 +0100 Subject: [Python-Dev] git (Was: Encouraging developers) References: <20070305184614.GA19051@localhost.localdomain> <9e804ac0703060126n18a8746lf84c0df80b72f995@mail.gmail.com> <20070306142720.GA31643@python.ca> <45ED8472.2060009@v.loewis.de> Message-ID: Hi python-dev, MvL wrote: >>> the on-disk repository is mighty big and it doesn't work very well >>> on non-Linux systems (at least, not last I looked.) Yes, mercurial or Bazaar do its job better on Windows etc. (and are written in Python :-) >> Not true. The on-disk repository is now one of the more efficient >> ones. After packing the repository, yes (which has to be done manually). > If this still makes git one of the more efficient > dvcs systems, I don't want to see the other ones :-( I do not know about speed issues in git, but I think that mercurial is said to be nearly as fast as git - and at least updating repos seems to work for me quickly. Thomas A. Hein has setup a one-shot mirror of the CPython SVN with branch support here: http://hg.intevation.org/tmp/python/python/ And there is a mirror here: http://hg.alexanderweb.de/python-temp/ You can clone it by running `hg clone URL`, and then `hg serve` to see the repo in your web browser. For me that needed less than 5 minutes on a well-connected machine. For beginners, there is a nice explanation about DVCS here: http://www.selenic.com/mercurial/wiki/index.cgi/UnderstandingMercurial Kind regards, Alexander From jtauber at jtauber.com Thu Mar 8 00:33:22 2007 From: jtauber at jtauber.com (James Tauber) Date: Wed, 7 Mar 2007 18:33:22 -0500 Subject: [Python-Dev] Google Summer of Code 2007 Message-ID: <3006B852-E84D-4C40-99CC-1C7FEE193CB6@jtauber.com> Google's Summer of Code is on again! I'm in the process of submitting the application for PSF to again be a mentoring organization. A mailing list has been set up for people who are interested in mentoring for the PSF. If you aren't able or willing to mentor but still want to participate in discussion of possible project ideas, you are more than welcome to join also. You can subscribe at: http://mail.python.org/mailman/listinfo/soc2007-mentors/ Because I would like core Python projects to be well represented, I particularly encourage python-devers to participate. James -- James Tauber http://jtauber.com/ journeyman of some http://jtauber.com/blog/ From jon at jon-foster.co.uk Thu Mar 8 00:40:36 2007 From: jon at jon-foster.co.uk (Jon Foster) Date: Wed, 07 Mar 2007 23:40:36 +0000 Subject: [Python-Dev] [ 1669539 ] Change (fix!) os.path.isabs() semantics on Win32 In-Reply-To: References: Message-ID: <45EF4D74.8020904@jon-foster.co.uk> Hi, Jason Orendorff wrote: > While we're at it, patch 1669539 makes a similar incompatible change > to ntpath.isabs(). > > The patch is incomplete (no docs) After reading your mail, I've just posted a new version of the patch which adds documentation. > but ripe for a note of encouragement (or summary rejection) from a committer. Comments welcome. Kind regards, Jon Foster From barry at python.org Thu Mar 8 00:43:15 2007 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Mar 2007 18:43:15 -0500 Subject: [Python-Dev] PEP 364, Transitioning to the Py3K standard library In-Reply-To: References: <6EBA9F60-ECBF-426B-AC60-E1044AEDE938@python.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mar 7, 2007, at 4:36 PM, Brett Cannon wrote: >> >> Third party package renaming is also supported, via several public >> >> interfaces accessible by any Python module. >> >> >> > >> > I guess a .pth file could install the mappings for the third-party >> > modules. >> >> How would that work? .pth files don't execute arbitrary Python code, > > It does through imports, though. I think it's important to import on demand only though. I should probably make that clear in the PEP . IOW, "import email" should not by side-effect import all sub-modules just because there's a remapping involved. This actually would be quite easy to do, but it's one of the reasons why the Python 2.5 email package hack goes through so much trouble. >> but it seems like that would be necessary to make this work. OTOH, >> your suggestion does make me think that it might be possible to >> piggyback the module name mappings onto .pth file processing. It >> already understand the 'import' command, so why not a 'rename' >> command? > > Not needed. Just put your mapping code in a module in your package > and have a .pth file import that module. >> Think about what that big dictionary would look like. It would >> essentially be like a text file with mappings one per line, but you'd >> have a lot of extra goo on each line (quotes, colons, and commas). I >> thought it was important to decentralize the mappings and then to put >> them in the simplest format that would serve the purpose. > > Right, but I am just not a big special-purpose format guy. =) Plus > mappings for third-parties can be kept to a single file for the module > they import by keeping the dict in that file. If you went with the dictionary approach, then you wouldn't need the .pth file. I guess a question for either approach is what you do if the top-level module is renamed. E.g. if we rename Carbon to carbon, where do we put the remapping for this? You can't keep both directories because of case-insensitive file systems (this has already been a "fun" pep to develop on a Mac ;). If we narrow the scope to only the official stdlib, then it's probably not an issue because we can stick it anywhere and arrange for it to get loaded. > I guess I would want to minimize the API in case we decide to directly > integrate this into import itself so we don't have to have both a > meta_path and path_importer_cache object. I think the solution works > for PEP 3108 and the mail package but I think I would a more > integrated solution if we went all out in terms of providing the > support outside of the stdlib (at least eventually). It depends on how explicit or magical you want the hooks to be. More magic is more convenient to hook into, but perhaps harder to decipher. >> If people really don't like the separate data file, and really want >> to just use a Python dictionary, then the way to keep the >> decentralization would be to reserve a __*__ symbol in the package >> namespace that contained this remapping. Something like __renames__ >> = {...}. > > I think the .pth file solution alleviates this need. At the cost of immediately importing all sub-modules when the package is imported. Do you agree that lazy loading is a requirement we should keep? >> > If you are going to have the object accessible from outside >> > sys.path_importer_cache I would also add an attribute that contains >> > the magic string used on sys.path. >> >> Of course, that magic string is also available via oldlib._magic. I >> tend to think it won't be necessary, but if people really want it, I >> can see putting the magic string as an attribute on the >> OldStdlibLoader object, e.g. sys.stdlib_remapper.importer_cache_key. > > Yeah, that was what I was thinking of. When you manipulate sys.path > you will need to be aware of it and what it is named. Good point. Yes, I like putting the magic key on the remapper object. That does give us a bit of a chicken and egg though because until you know the magic string, you can't know the path_importer_cache key, so I think putting the remapper in sys is useful. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRe9OFHEjvBPtnXfVAQIVhwP+Oloyxi/ldlbQ7RM2ifmgN03jc/SYN5Ht suxsXiK3xwiCU5B/J6uAKDSol/WtxKV/EI2yN42xJIUFV0B0C8DCLHDZHLDAI67V dfsJMAV+wgn8RjPVSuEw9vOFaVaoyIVvdDn+XMKKfKX6YOBnjLpnLaZJtEVx2QMg ZigImiWp3VI= =EHOG -----END PGP SIGNATURE----- From barry at python.org Thu Mar 8 00:45:10 2007 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Mar 2007 18:45:10 -0500 Subject: [Python-Dev] PEP 364, Transitioning to the Py3K standard library In-Reply-To: <740c3aec0703071446k709cd3bdvbdabd0805abb5fdb@mail.gmail.com> References: <6EBA9F60-ECBF-426B-AC60-E1044AEDE938@python.org> <740c3aec0703071446k709cd3bdvbdabd0805abb5fdb@mail.gmail.com> Message-ID: <04B71C90-61FF-45ED-ACF2-B5138579938F@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mar 7, 2007, at 5:46 PM, BJ?rn Lindqvist wrote: >> When Python's import machinery is initialized, the oldlib package is >> imported. Inside oldlib there is a class called ``OldStdlibLoader``. >> This class implements the PEP 302 interface and is automatically >> instantiated, with zero arguments. The constructor reads all the >> ``.mv`` files from the oldlib package directory, automatically >> registering all the remappings found in those ``.mv`` files. This is >> how the Python 2.x standard library is remapped. > > Will not reading all those .mv files add a lot of overhead? Disk seeks > is not cheap. I don't think so. There will likely be just a few .mv files in the stdlib and they'd all get loaded at Python startup time. A package's .mv file would only get loaded when the package gets imported. Once read I don't think we'd need to stat for those or read them again. It would definitely be worth measuring, but my guess is the additional overhead will be negligible. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRe9OhnEjvBPtnXfVAQLjYAP7BeVkCMGCVeVt/oqIoywtyP1+B4Ndgmdo uj0JQUSmu2ikRXRoSNtNSoSKdQsco9FI+LHYHwBRJmLKC+IIZ0glP/y+re6hkdOa vWfj1QLYBzWY6EzXRgf92xiR9dXJx92vsiSWAjibRwLlE7d+3eFYqrBa+fvEzQGv pkKjOlTYIKo= =5OL3 -----END PGP SIGNATURE----- From brett at python.org Thu Mar 8 01:01:01 2007 From: brett at python.org (Brett Cannon) Date: Wed, 7 Mar 2007 16:01:01 -0800 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070307174759.GF29575@caltech.edu> References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <45ED0371.5070108@v.loewis.de> <200703060848.04117.phil@riverbankcomputing.co.uk> <45ED39F0.2000907@v.loewis.de> <45EE82AC.2020103@develer.com> <45EE8F8E.3070409@v.loewis.de> <20070307174759.GF29575@caltech.edu> Message-ID: On 3/7/07, Titus Brown wrote: > On Wed, Mar 07, 2007 at 11:10:22AM +0100, "Martin v. L?wis" wrote: > -> Giovanni Bajo schrieb: > -> > On 06/03/2007 10.52, Martin v. L?wis wrote: > -> > > -> >> I can't see that the barrier at contributing is high. > -> > > -> > I think this says it all. It now appears obvious to me that people > -> > inside the "mafia" don't even realize there is one. Thus, it looks like > -> > we are all wasting time in this thread, since they think there is > -> > nothing to be changed. > > > > Hi, I just wanted to interject -- when I used the word "mafia", I meant > it in this sense: > > """ > Informal. A tightly knit group of trusted associates, as of a political > leader: [He] is one of the personal mafia that [the chancellor] > brought with him to Bonn. > """ > > (Martin, I certainly didn't intend to offend anyone by implying that > they were part of a criminal organization. ;) > > I have a longer explanation of my view in the blog entry, and anyway I > don't want to belabor my own experiences too much, but I would like to > point out three things: > > * there's definitely a group of "trusted associates" -- committers, > perhaps? -- and it's not at all clear to outsiders like me how new > features, old patches, and old bugs are dealt with. > > * python-dev is an all-volunteer community. In true open-source > fashion, that means that it's incumbent upon people who HAVE > problems/issues with a process (like me, and Giovanni) to propose > solutions that either someone wants to implement, or that we can > implement. I certainly don't expect any of the committers to put > tons of effort into a new initiative. > > * Much of the discussion on this issue of encouraging developers comes > down to communicating better with non-python-dev people. > > Unless someone is already doing it, I will try to write a summary of the > last few days of discussion and post it for review. The idea would be to > provide a simple one stop wiki page for people who want to contribute. > My hope (along with many other hopes =), is to get a good document that explains exactly what is expected for bugs and patches in general and then what is specifically expected for things like new modules for the stdlib or new syntax proposals. With it all written in one place we have something to point people towards (basically what http://www.python.org/dev/intro/ was meant to be, but replaces most of what is in http://www.python.org/dev/). And I hope that once the tracker is up and we have some experience we can make it help us by possibly annotating issues with that they hare or are lacking (e.g., needs tests, needs docs, etc.). That way people can read this doc, understand what is expected for code and such, and then search the tracker for stuff that needs those thing and help deal with it. Basically make it so that if someone goes, "I want to help write a test", they can find out what needs a test. But I get the docs for the new tracker written first. Once that is done I plan to start working on this doc (or docs if it gets too long) as my next big Python project. -Brett From martin at v.loewis.de Thu Mar 8 01:36:57 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 08 Mar 2007 01:36:57 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <20070307080541.74AD.JCARLSON@uci.edu> References: <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <45EDB1F3.6060801@v.loewis.de> <20070307080541.74AD.JCARLSON@uci.edu> Message-ID: <45EF5AA9.5090309@v.loewis.de> Josiah Carlson schrieb: >> Now it's becoming difficult: several people in favor, some opposed... > > What about changing the semantics of splitext and creating a new > function (available on all platforms) that does what the Windows version > currently does? > > For people who want the one semantic on all platforms, they can pick > their semantic explicitly. Most people object because of the potential breakage; this change wouldn't remove the breakage (people asking for the old, compatible, behavior still need to change their code). > Alternatively, offer a flag argument in splitext to differentiate > between the two cases. This isn't compatible, either, as older versions will raise an exception because of the extra argument. Regards, Martin From brett at python.org Thu Mar 8 01:39:46 2007 From: brett at python.org (Brett Cannon) Date: Wed, 7 Mar 2007 16:39:46 -0800 Subject: [Python-Dev] PEP 364, Transitioning to the Py3K standard library In-Reply-To: References: <6EBA9F60-ECBF-426B-AC60-E1044AEDE938@python.org> Message-ID: On 3/7/07, Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Mar 7, 2007, at 4:36 PM, Brett Cannon wrote: > > >> >> Third party package renaming is also supported, via several public > >> >> interfaces accessible by any Python module. > >> >> > >> > > >> > I guess a .pth file could install the mappings for the third-party > >> > modules. > >> > >> How would that work? .pth files don't execute arbitrary Python code, > > > > It does through imports, though. > > I think it's important to import on demand only though. And I agree. > I should > probably make that clear in the PEP . IOW, "import email" > should not by side-effect import all sub-modules just because there's > a remapping involved. This actually would be quite easy to do, but > it's one of the reasons why the Python 2.5 email package hack goes > through so much trouble. > I am not suggesting that every rename be imported. All I am saying is that a .pth file could ``import pkg.old_names`` and that old_names module adds the mappings, that's all. It doesn't do anything that you are not proposing be done automatically. -Brett From martin at v.loewis.de Thu Mar 8 01:50:39 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 08 Mar 2007 01:50:39 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070307174759.GF29575@caltech.edu> References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <45ED0371.5070108@v.loewis.de> <200703060848.04117.phil@riverbankcomputing.co.uk> <45ED39F0.2000907@v.loewis.de> <45EE82AC.2020103@develer.com> <45EE8F8E.3070409@v.loewis.de> <20070307174759.GF29575@caltech.edu> Message-ID: <45EF5DDF.5060200@v.loewis.de> Titus Brown schrieb: > Hi, I just wanted to interject -- when I used the word "mafia", I meant > it in this sense: > > """ > Informal. A tightly knit group of trusted associates, as of a political > leader: [He] is one of the personal mafia that [the chancellor] > brought with him to Bonn. > """ > > (Martin, I certainly didn't intend to offend anyone by implying that > they were part of a criminal organization. ;) Apology accepted. As to your specific comments: > there's definitely a group of "trusted associates" -- committers, > perhaps? Yes, but not all of the committers are part of the "mafia", i.e. some never review any patches. Also, the "mafia" isn't "tightly nit". I.e. they don't have a hidden agenda they follow, to implement a political quest or some such. > and it's not at all clear to outsiders like me how new > features, old patches, and old bugs are dealt with. The simple answer is "when we have time". There really is not more to it. Some patches get higher attention, e.g. because they fix serious bugs. Proposed new features of don't get any attention by the "mafia", because Python will just work fine without the new feature. > Much of the discussion on this issue of encouraging developers comes > down to communicating better with non-python-dev people. This is a two-sided thing, of course. The non-python-dev people should also communicate with the python-dev ones, instead of forming false (and unconfirmed) assumptions on how things really work. It's easy to assume conspiracy everywhere, much harder to understand that the people you are interacting with are regular humans who contribute to open source for the same reasons as the next guy. Regards, Martin From martin at v.loewis.de Thu Mar 8 02:11:05 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 08 Mar 2007 02:11:05 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> Message-ID: <45EF62A9.4010701@v.loewis.de> Facundo Batista schrieb: > I finally understood the problem, and build python from the repository, > and made the tests from *this* python (actually, this was an easy step > because I'm on Ubuntu, but *I* would be dead if working in Windows, for > example). > > Ok. *Me*, that I'm not ashame of asking what I don't know, if I didn't > resolve it I'd finally asked in python-dev. But how many people would > have throw the towel withoug getting so far? > > How many people want to submit a patch, or even a bug, or finds a patch > to review, but don't know how to do something and thinks that python-dev > is not the place to ask (too high minds and experienced people and > everything)? If people want to contribute to open source, they just need to learn the rules. One of the primary rules is that most of it is a meritocracy: it's not "high minds" that give reputation, but past contributions. Sure, some people will be shied (sp?) away by merely thinking about python-dev. For them, the barrier is high for *any* contribution to open source software. > What I propose is a dedicated place (mailing list, for example), that is > something like a place where you can go and ask the silliest questions > about helping in the developing process. In principle, python-dev should be such a place, but I can see how it isn't (due to nobody's fault). However, people should feel free to ask any question on python-help at python.org, and actually do so. > - How can I know if a patch is still open? Easy: if it's marked as Open. > - I found a problem, and know how to fix it, but what else need to do? Go to www.python.org, then "CORE DEVELOPMENT", then "Patch submission". > - Found a problem in the docs, for this I must submit a patch or tell > something about it? How? Read CORE DEVELOPMENT, Intro to development. > - I found an error in the docs, and fixed it, but I'm spanish speaker > and my english sucks, can I submit a patch with bad wording or I must > ask somebody to write it ok? What would your intuition tell you to do? > Me, for example, has an actual question to this list: "How can I know, > if I change something in the doc's .tex files, that I'm not broking > the TeX document structure?". You don't have to know. As a general contributor, just submit your patch, and perhaps the reviewer will catch the error. As a submitter, just submit the code, and George Yoshida will catch it. It's not strictly necessary that the documentation actually builds all the time. If you want to be sure it builds, run "make html" in Doc. Regards, Martin From martin at v.loewis.de Thu Mar 8 02:13:32 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 08 Mar 2007 02:13:32 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> Message-ID: <45EF633C.9090405@v.loewis.de> Georg Brandl schrieb: > > Of course, the channel would have to be made an "official" Python development > tool and advertised on e.g. the website. Also, it couldn't hurt if some of the > other devs would frequent it too, then :) > I definitely won't: I don't use IRC (or any other chat infrastructure), as a matter of principle. The only way to talk to me "in real time" is in person, or on the phone. (I do make an exception for the PSF board). Regards, Martin From python at rcn.com Thu Mar 8 02:16:28 2007 From: python at rcn.com (Raymond Hettinger) Date: Wed, 7 Mar 2007 20:16:28 -0500 (EST) Subject: [Python-Dev] Fwd: Re: [Python-3000] Removing functions from the operator module Message-ID: <20070307201628.BAQ33047@ms09.lnh.mail.rcn.net> [Collin Winter] >> I don't suppose you've changed your mind about removing operator.truth >> and operator.abs in the seven months since this discussion? [GvR] >No, though I think that operator.truth should be renamed to operator.bool. > >I like the idea that for each built-in op there's a callable in operator. It makes sense to me that built-in ops like + have a corresponding operator function like operator.__add__(), but I don't follow how this logic applies to abs() and bool(). ISTM, operator.bool and operator.abs would not add any value beyond what is already provided by __builtin__.bool and __builtin__.abs. Raymond From guido at python.org Thu Mar 8 02:20:50 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 7 Mar 2007 17:20:50 -0800 Subject: [Python-Dev] Fwd: Re: [Python-3000] Removing functions from the operator module In-Reply-To: <20070307201628.BAQ33047@ms09.lnh.mail.rcn.net> References: <20070307201628.BAQ33047@ms09.lnh.mail.rcn.net> Message-ID: They do, by emphasizing the relationship with special methods. On 3/7/07, Raymond Hettinger wrote: > [Collin Winter] > >> I don't suppose you've changed your mind about removing operator.truth > >> and operator.abs in the seven months since this discussion? > > [GvR] > >No, though I think that operator.truth should be renamed to operator.bool. > > > >I like the idea that for each built-in op there's a callable in operator. > > It makes sense to me that built-in ops like + have a corresponding operator function like operator.__add__(), but I don't follow how this logic applies to abs() and bool(). ISTM, operator.bool and operator.abs would not add any value beyond what is already provided by __builtin__.bool and __builtin__.abs. > > > Raymond > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From python at rcn.com Thu Mar 8 02:23:42 2007 From: python at rcn.com (Raymond Hettinger) Date: Wed, 7 Mar 2007 20:23:42 -0500 (EST) Subject: [Python-Dev] Fwd: Re: Encouraging developers Message-ID: <20070307202342.BAQ34320@ms09.lnh.mail.rcn.net> [Facundo] >> Me, for example, has an actual question to this list: "How can I know, >> if I change something in the doc's .tex files, that I'm not broking >> the TeX document structure?". [MvL] >You don't have to know. As a general contributor, just submit your >patch, and perhaps the reviewer will catch the error. As a submitter, >just submit the code, and George Yoshida will catch it. It's not >strictly necessary that the documentation actually builds all >the time. If you want to be sure it builds, run "make html" in >Doc. We should try to keep the documentation buildable at all times so that it will remain visible to everyone at http://docs.python.org/dev/ . The script in Tools/scripts/texcheck.py does a pretty good job of detecting the most common LaTeX markup errors. For small doc changes, it is usually not too much of a challenge because you're either modifying existing text or are copying in existing markup from the surrounding text. Raymond From nnorwitz at gmail.com Thu Mar 8 03:03:40 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 7 Mar 2007 18:03:40 -0800 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45EF62A9.4010701@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <45EF62A9.4010701@v.loewis.de> Message-ID: On 3/7/07, "Martin v. L?wis" wrote: > > > Me, for example, has an actual question to this list: "How can I know, > > if I change something in the doc's .tex files, that I'm not broking > > the TeX document structure?". > > You don't have to know. As a general contributor, just submit your > patch, and perhaps the reviewer will catch the error. As a submitter, > just submit the code, and George Yoshida will catch it. It's not > strictly necessary that the documentation actually builds all > the time. If you want to be sure it builds, run "make html" in > Doc. Even better, the same machine running refleaks every 12 hours also builds the docs. It will complain if the docs don't build, so be aggressive with the docs. Any failures are sent do python-checkins. You can also see the results from trunk here: http://docs.python.org/dev/ http://docs.python.org/dev/results/ Here are the 2.5 results (only builds doc, not refleak test): http://docs.python.org/dev/2.5/ http://docs.python.org/dev/2.5/results/ These pages are generated from Misc/build.sh. As I mentioned, build.sh runs every 12 hours. n From nnorwitz at gmail.com Thu Mar 8 03:07:11 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 7 Mar 2007 18:07:11 -0800 Subject: [Python-Dev] Fwd: Re: Encouraging developers In-Reply-To: <20070307202342.BAQ34320@ms09.lnh.mail.rcn.net> References: <20070307202342.BAQ34320@ms09.lnh.mail.rcn.net> Message-ID: On 3/7/07, Raymond Hettinger wrote: > > We should try to keep the documentation buildable at all times so that it will remain visible to everyone at http://docs.python.org/dev/ . Broken doc doesn't get pushed though. So the worst that will happen is that the dev doc will be slightly out of date if it doesn't build. Since we (or George Yoshida) tends to fix these pretty fast, the worst that happens is the docs are a day behind. To reiterate from my other msg to Martin, the building of doc (and the refleak test run) is all done in Misc/build.sh. This also does a make install in case something breaks there. n From fwierzbicki at gmail.com Thu Mar 8 03:13:28 2007 From: fwierzbicki at gmail.com (Frank Wierzbicki) Date: Wed, 7 Mar 2007 21:13:28 -0500 Subject: [Python-Dev] Google Summer of Code 2007 In-Reply-To: <3006B852-E84D-4C40-99CC-1C7FEE193CB6@jtauber.com> References: <3006B852-E84D-4C40-99CC-1C7FEE193CB6@jtauber.com> Message-ID: <4dab5f760703071813s416d68d2s5fbdc5c14e339a5f@mail.gmail.com> On 3/7/07, James Tauber wrote: > Google's Summer of Code is on again! > > I'm in the process of submitting the application for PSF to again be > a mentoring organization. > Because I would like core Python projects to be well represented, I > particularly encourage python-devers to participate. Just to be sure -- Jython is under the umbrella of the PSF for the purposes of the SoC, right? Thanks, -Frank From jtauber at jtauber.com Thu Mar 8 03:20:04 2007 From: jtauber at jtauber.com (James Tauber) Date: Wed, 7 Mar 2007 21:20:04 -0500 Subject: [Python-Dev] Google Summer of Code 2007 In-Reply-To: <4dab5f760703071813s416d68d2s5fbdc5c14e339a5f@mail.gmail.com> References: <3006B852-E84D-4C40-99CC-1C7FEE193CB6@jtauber.com> <4dab5f760703071813s416d68d2s5fbdc5c14e339a5f@mail.gmail.com> Message-ID: <53F07073-9853-40DC-B125-61A0EFBD12E5@jtauber.com> While I guess there's nothing to stop Jython applying in its own right, it makes far more sense to be under the umbrella of PSF. PyPy projects last year were under PSF, for example. I would *strongly* encourage the submission of some Jython projects under the PSF umbrella. James On 07/03/2007, at 9:13 PM, Frank Wierzbicki wrote: > On 3/7/07, James Tauber wrote: >> Google's Summer of Code is on again! >> >> I'm in the process of submitting the application for PSF to again be >> a mentoring organization. > >> Because I would like core Python projects to be well represented, I >> particularly encourage python-devers to participate. > Just to be sure -- Jython is under the umbrella of the PSF for the > purposes of the SoC, right? > > Thanks, > > -Frank From fwierzbicki at gmail.com Thu Mar 8 03:46:22 2007 From: fwierzbicki at gmail.com (Frank Wierzbicki) Date: Wed, 7 Mar 2007 21:46:22 -0500 Subject: [Python-Dev] Google Summer of Code 2007 In-Reply-To: <53F07073-9853-40DC-B125-61A0EFBD12E5@jtauber.com> References: <3006B852-E84D-4C40-99CC-1C7FEE193CB6@jtauber.com> <4dab5f760703071813s416d68d2s5fbdc5c14e339a5f@mail.gmail.com> <53F07073-9853-40DC-B125-61A0EFBD12E5@jtauber.com> Message-ID: <4dab5f760703071846t35fc2f41j2d5d7b1a6c6a2923@mail.gmail.com> On 3/7/07, James Tauber wrote: > I would *strongly* encourage the submission of some Jython projects > under the PSF umbrella. Great! I'll do my best to get some submitted. -Frank From barry at python.org Thu Mar 8 04:10:56 2007 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Mar 2007 22:10:56 -0500 Subject: [Python-Dev] PEP 364, Transitioning to the Py3K standard library In-Reply-To: References: <6EBA9F60-ECBF-426B-AC60-E1044AEDE938@python.org> Message-ID: <6F817E07-7106-4DFA-9D29-6734F6194D57@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mar 7, 2007, at 7:39 PM, Brett Cannon wrote: >> I think it's important to import on demand only though. > > And I agree. Cool. >> I should >> probably make that clear in the PEP . IOW, "import email" >> should not by side-effect import all sub-modules just because there's >> a remapping involved. This actually would be quite easy to do, but >> it's one of the reasons why the Python 2.5 email package hack goes >> through so much trouble. > I am not suggesting that every rename be imported. All I am saying is > that a .pth file could ``import pkg.old_names`` and that old_names > module adds the mappings, that's all. It doesn't do anything that you > are not proposing be done automatically. Cool, I see. What advantage do you see with importing the pkg.old_names from a .pth file rather than just including whatever is necessary right in pkg/__init__.py? - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRe9+wHEjvBPtnXfVAQI60AP8D7PFDNVR/BiLEoz8iQKYJUOsd8vtXTF5 lcmDi+L0SZ8N95KxlI2icQmaRk2wNhO+0Djp3XJA/aKiHTWa3ImIksuGFb7WIRSY HhSi2NXPRJOitHz3NhDrfMOFXbDq+Ci4E1YU/+XjtfnsEYqkcLvehvlpY3Urod/j Z/9g6BDGOuM= =pxLk -----END PGP SIGNATURE----- From brett at python.org Thu Mar 8 04:50:12 2007 From: brett at python.org (Brett Cannon) Date: Wed, 7 Mar 2007 19:50:12 -0800 Subject: [Python-Dev] PEP 364, Transitioning to the Py3K standard library In-Reply-To: <6F817E07-7106-4DFA-9D29-6734F6194D57@python.org> References: <6EBA9F60-ECBF-426B-AC60-E1044AEDE938@python.org> <6F817E07-7106-4DFA-9D29-6734F6194D57@python.org> Message-ID: On 3/7/07, Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Mar 7, 2007, at 7:39 PM, Brett Cannon wrote: > > >> I think it's important to import on demand only though. > > > > And I agree. > > Cool. > > >> I should > >> probably make that clear in the PEP . IOW, "import email" > >> should not by side-effect import all sub-modules just because there's > >> a remapping involved. This actually would be quite easy to do, but > >> it's one of the reasons why the Python 2.5 email package hack goes > >> through so much trouble. > > I am not suggesting that every rename be imported. All I am saying is > > that a .pth file could ``import pkg.old_names`` and that old_names > > module adds the mappings, that's all. It doesn't do anything that you > > are not proposing be done automatically. > > Cool, I see. What advantage do you see with importing the > pkg.old_names from a .pth file rather than just including whatever is > necessary right in pkg/__init__.py? > It is only needed if the name of the package itself changed since you need a bootstrap before you try to import the package itself. Otherwise __init__.py is the right place if the root package name has not changed. -Brett From andrew-pythondev at puzzling.org Thu Mar 8 05:52:29 2007 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Thu, 8 Mar 2007 15:52:29 +1100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45EE773F.20003@v.loewis.de> <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> References: <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <5.1.1.6.0.20070306162907.02b7e2a0@sparrow.telecommunity.com> <45EDE899.9020709@v.loewis.de> <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> Message-ID: <20070308045229.GQ14306@steerpike.home.puzzling.org> Hi Martin, "Martin v. L?wis" wrote: > glyph at divmod.com schrieb: [...] > > The use-cases being discussed here would be better served by having new > > APIs that do particular things and don't change existing semantics, > > though. For example, a "guess_mime_type(path)" function which could > > examine a path and figure out its mime type based on its extension > > (among other things). > > I disagree. The current behavior is incorrect (and always has been); > the new behavior is clearly correct. Glyph's proposing that rather than risk breaking existing code (and in the worst possible way: silently, giving wrong answers rather than exceptions), we examine what benefits changing splitext would bring, and see if there's a way to get those benefits without that risk. I don't think that's an idea to dismiss out of hand. You silently omitted this part of glyph's objection: glyph at divmod.com wrote: [...] > > Incompatible changes may be *acceptable* for feature releases, but that doesn't > mean they are desirable. The cost of incompatibility should be considered for > every change. This cost is particularly bad when the incompatibility is of the > "silent breakage" variety - the change being discussed here would not be the > sort of thing that one could, say, produce a warning about or gently deprecate; > anything relying on the old behavior would suddenly be incorrect, and any > software wishing to straddle the 2.5<->2.6 boundary would be better off just > implementing their own splitext() than relying on the stdlib. At the moment you seem to be saying that because you're possibly fixing some currently buggy code, it's ok to possibly break some currently working code. I'm not sure that's a good idea, and I expect it will create headaches for someone trying to be compatible with both 2.5 and 2.6. If the existing behaviour was clearly buggy by the existing documentation, rather than undefined, this would be less of a concern. The point about silent failures is an important one too. This will cause some code that was working correctly to give wrong answers, rather than an exception or any indication that this previously correct use is now incorrect and causing bugs. If "splitext" was removed and replaced with, say, a "split_ext" that behaves as you (and basically everyone, myself included) agrees it should behave, then code won't silently do the wrong thing. It will immediately and visibly fail, in an understandable way. Similarly, if split_ext was added side-by-side with splitext, and splitext changed to emit DeprecationWarnings for a release, there's minimal risk that existing code that works would break. See also the comment at the bottom of http://sourcefrog.net/weblog/software/aesthetics/interface-levels.html about "perverse interface design". I wonder if "First, do no harm" should be an explicit guideline when it comes evaluating incompatible changes, even if they fix other things? There seems to be the problem that all existing users of splitext that may encounter '.foo' will need to audit their code to see if they are relying on previously undefined behaviour that will change. I think it would be much more helpful for python to somehow help users discover this fact, than simply define the previously undefined behaviour to be different to what they might have been relying on. All that said, I don't think I've ever used splitext myself, so I don't care so much about what happens to it. But I do worry about the general approach to backwards compatibility. -Andrew. From nnorwitz at gmail.com Thu Mar 8 06:31:21 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 7 Mar 2007 21:31:21 -0800 Subject: [Python-Dev] new 2.5.1 release schedule Message-ID: Sorry for the delay. We really wanted to put out a 2.5.1 release earlier. Due to busy and conflicting schedules, we had to postpone a bit. We are on track for the following schedule: 2.5.1c1 - Tuesday, April 3 2.5.1 - Thursday April 12 (assuming a c2 is not necessary) I don't know of any release blocking problems. If you have any issues, please let me know. Cheers, n From jcarlson at uci.edu Thu Mar 8 06:49:39 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 07 Mar 2007 21:49:39 -0800 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <20070308045229.GQ14306@steerpike.home.puzzling.org> References: <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> Message-ID: <20070307214117.74CE.JCARLSON@uci.edu> Andrew Bennetts wrote: > Glyph's proposing that rather than risk breaking existing code (and in the worst > possible way: silently, giving wrong answers rather than exceptions), we examine > what benefits changing splitext would bring, and see if there's a way to get > those benefits without that risk. I don't think that's an idea to dismiss out > of hand. Because we should refuse the temptation to guess, what about: Rename the posix splitext to X (for some X), and offer a function with identical functionality to the posix variant under win32, also available as X for that platform. Rename the (current) win32 splitext to Y (for some Y) and offer a function with identical functionality to the posix platform as name Y. Offer a new splitext that uses X on posix and Y on win32, but causes a DeprecationWarning with pointers to the two renamed functions that are available on both platforms. For people who want the old platform-specific functionality in previous and subsequent Pythons, offer the ability to disable the DeprecationWarning via a module global; at least until the function is removed in some future Python, at which point path.splitext would cause an AttributeError . - Josiah From collinw at gmail.com Thu Mar 8 06:53:45 2007 From: collinw at gmail.com (Collin Winter) Date: Wed, 7 Mar 2007 23:53:45 -0600 Subject: [Python-Dev] Fwd: Re: [Python-3000] Removing functions from the operator module In-Reply-To: <20070307201628.BAQ33047@ms09.lnh.mail.rcn.net> References: <20070307201628.BAQ33047@ms09.lnh.mail.rcn.net> Message-ID: <43aa6ff70703072153n1fe873fapb17a6421f8d5de8e@mail.gmail.com> On 3/7/07, Raymond Hettinger wrote: > [Collin Winter] > >> I don't suppose you've changed your mind about removing operator.truth > >> and operator.abs in the seven months since this discussion? > > [GvR] > >No, though I think that operator.truth should be renamed to operator.bool. > > > >I like the idea that for each built-in op there's a callable in operator. > > It makes sense to me that built-in ops like + have a corresponding operator function like operator.__add__(), but I don't follow how this logic applies to abs() and bool(). ISTM, operator.bool and operator.abs would not add any value beyond what is already provided by __builtin__.bool and __builtin__.abs. > FWIW, I've always thought of the operator module as a place to look when I need to pass some bit of syntax (eg., attribute lookup, "in", addition) to a higher-order function. bool() and abs() aren't syntax, so I would never look in operator. Collin Winter From andrew-pythondev at puzzling.org Thu Mar 8 07:03:55 2007 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Thu, 8 Mar 2007 17:03:55 +1100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <20070307214117.74CE.JCARLSON@uci.edu> References: <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> <20070307214117.74CE.JCARLSON@uci.edu> Message-ID: <20070308060355.GS14306@steerpike.home.puzzling.org> Josiah Carlson wrote: [...] > > Offer a new splitext that uses X on posix and Y on win32, but causes a > DeprecationWarning with pointers to the two renamed functions that are > available on both platforms. > > For people who want the old platform-specific functionality in previous > and subsequent Pythons, offer the ability to disable the > DeprecationWarning via a module global; at least until the function is > removed in some future Python, at which point path.splitext would cause > an AttributeError . It's not about cross-platform behaviour. It's already the same: >>> import posixpath, ntpath >>> ntpath.splitext('.cshrc') ('', '.cshrc') >>> posixpath.splitext('.cshrc') ('', '.cshrc') And as I understand it, the current proposal would change this behaviour the same way on win32 and POSIX, although I may not have kept up with that part of the discussion enough. Is there a genuine desire for this to behave differently on different platforms? I think what I'd probably like to see is the existing undefined (but well-understood and predictable) behaviour to be declared defined, and a warning put in the documentation that it probably doesn't do what you expect for this case. And then for those that want the new behaviour, provide a new function (and prominently advertise it in the relevant part of the documentation). Yes it sucks having two similar functions, which will inevitably have similar names, but that's the price you pay for getting the interface wrong the first time around. -Andrew. From jcarlson at uci.edu Thu Mar 8 07:51:18 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 07 Mar 2007 22:51:18 -0800 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <20070308060355.GS14306@steerpike.home.puzzling.org> References: <20070307214117.74CE.JCARLSON@uci.edu> <20070308060355.GS14306@steerpike.home.puzzling.org> Message-ID: <20070307225018.74D4.JCARLSON@uci.edu> Andrew Bennetts wrote: > > Josiah Carlson wrote: > [...] > > > > Offer a new splitext that uses X on posix and Y on win32, but causes a > > DeprecationWarning with pointers to the two renamed functions that are > > available on both platforms. > > > > For people who want the old platform-specific functionality in previous > > and subsequent Pythons, offer the ability to disable the > > DeprecationWarning via a module global; at least until the function is > > removed in some future Python, at which point path.splitext would cause > > an AttributeError . > > It's not about cross-platform behaviour. It's already the same: > > >>> import posixpath, ntpath > >>> ntpath.splitext('.cshrc') > ('', '.cshrc') > >>> posixpath.splitext('.cshrc') > ('', '.cshrc') > > And as I understand it, the current proposal would change this behaviour the > same way on win32 and POSIX, although I may not have kept up with that part of > the discussion enough. Is there a genuine desire for this to behave differently > on different platforms? Ahh, I thought that it was different on different platforms. Nevermind then. Add a new function and be done with it. - Josiah From stephen at xemacs.org Thu Mar 8 09:50:17 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 08 Mar 2007 17:50:17 +0900 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070307081037.74B0.JCARLSON@uci.edu> References: <20070305175116.749D.JCARLSON@uci.edu> <45EDC618.2070402@develer.com> <20070307081037.74B0.JCARLSON@uci.edu> Message-ID: <878xe8dsqe.fsf@uwakimon.sk.tsukuba.ac.jp> Josiah Carlson writes: > > And the best way to encourage someone to maintain a package is... > > accepting their patches. > > And that's what I think is bull. Whether or not we want or need > maintainers for module or package X is independant of the fact that user > Y has submitted a patch for it. If they say, "I would like to become > maintainer of package X", ok, if they are better than not having a > maintainer, great. But to ask them to be a maintainer of an > unmaintained package because they submitted a patch? Actually, I regularly do this (three or four times a year, I'd guess, it would be more if there were more submitters) if the person seems sane. But I *don't* apply the patch! The condition is that he sign on to our process and shepherd his own patch through it (including the risk that a reviewer will ask for changes). The answer I typically get is "No, I'd rather wait than work. But thanks for asking!" :-) Caveat: the XEmacs packages where I do this are mostly end-user applications like MUAs or programmer editor modes, so it doesn't matter to anyone but the end-users if they break. Our process provides multiple workarounds for such a situation, and we do keep an eye on new maintainers to help out with emergency response in those cases, so that risk is considered acceptable. The users themselves generally accept one or two new maintainer problems with good humor, since they're proof positive that somebody has taken an interest in their package. The cost to this system is that you need people willing to mentor the new package maintainers. It works well for XEmacs because the cost is very low (as I just described). I think for Python's stdlib the cost would be substantially higher, but it might very well be worth it. > > A little game: without looking at this patch of mine, > > how much are you willing to bet that committing that patch is going to > > cause pain the Python community and other stdlib maintainers, to cause > > headaches like deprecations because of broken interfaces, and whatnot? I'm glad you asked. More than half of my commits (and way more than half the LOC) in February were due to broken previous patches that were approved merely because they addressed a bug and applied without fuzz to HEAD. :-( Committing patches unreviewed is a terrible idea. From martin at v.loewis.de Thu Mar 8 10:37:32 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 08 Mar 2007 10:37:32 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <20070308045229.GQ14306@steerpike.home.puzzling.org> References: <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <5.1.1.6.0.20070306140020.02c22668@sparrow.telecommunity.com> <5.1.1.6.0.20070306162907.02b7e2a0@sparrow.telecommunity.com> <45EDE899.9020709@v.loewis.de> <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> Message-ID: <45EFD95C.60706@v.loewis.de> Andrew Bennetts schrieb: > Glyph's proposing that rather than risk breaking existing code (and in the worst > possible way: silently, giving wrong answers rather than exceptions), we examine > what benefits changing splitext would bring, and see if there's a way to get > those benefits without that risk. I don't think that's an idea to dismiss out > of hand. But I did list what I consider the benefits of the change to be, in http://mail.python.org/pipermail/python-dev/2007-March/071675.html (the message you are responding to). I don't see any other way of achieving these benefits without changing splitext, so I didn't propose any alternatives. I haven't seen any other proposal, either. > At the moment you seem to be saying that because you're possibly fixing some > currently buggy code, it's ok to possibly break some currently working code. > I'm not sure that's a good idea, and I expect it will create headaches for > someone trying to be compatible with both 2.5 and 2.6. I personally don't expect severe or massive breakage. It's a small detail. > If the existing behaviour was clearly buggy by the existing documentation, > rather than undefined, this would be less of a concern. > > The point about silent failures is an important one too. I see all these objections. I just don't see them outweigh the advantages. > If "splitext" was removed and replaced with, say, a "split_ext" that behaves as > you (and basically everyone, myself included) agrees it should behave, then code > won't silently do the wrong thing. It will immediately and visibly fail, in an > understandable way. Similarly, if split_ext was added side-by-side with > splitext, and splitext changed to emit DeprecationWarnings for a release, > there's minimal risk that existing code that works would break. I considered these alternatives, and rejected them. Outright removing splitext would break a lot of code, period. I can't see how this could possibly be better than the currently implemented change. Raising a DeprecationWarning would also break a lot code, although in a less aggressive way. Who is going to see these warnings? The end users, running Python applications and using libraries they have not written. Fixing them will involve a very large cost in the community (spread over many people). Raising a DeprecationWarning should not be taken lightly. OTOH, silently changing the current behavior might break some applications. Again, it is the users who will notice, but for this change, it will be less users, as I expect that breakage will be really uncommon. > I wonder if "First, do no harm" should be an explicit guideline when it comes > evaluating incompatible changes, even if they fix other things? If possible, sure. However, I don't see how this is possible here. All your proposed alternatives do more harm than the one implemented. > All that said, I don't think I've ever used splitext myself, so I don't care so > much about what happens to it. But I do worry about the general approach to > backwards compatibility. It's not a general approach. It's a case-by-case decision. Please understand that *any* change is "incompatible", in the sense that I can come up with an application that breaks under the change. Really. *ANY CHANGE*. So the only way to get 100% compatibility is to not change anything at all. Regards, Martin From martin at v.loewis.de Thu Mar 8 11:44:55 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 08 Mar 2007 11:44:55 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45EEC9A4.1040900@develer.com> References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <45ED0371.5070108@v.loewis.de> <200703060848.04117.phil@riverbankcomputing.co.uk> <45ED39F0.2000907@v.loewis.de> <45EE82AC.2020103@develer.com> <45EE8F8E.3070409@v.loewis.de> <45EEC9A4.1040900@develer.com> Message-ID: <45EFE927.7090908@v.loewis.de> Giovanni Bajo schrieb: >>>> I can't see that the barrier at contributing is high. >>> >>> I think this says it all. It now appears obvious to me that people >>> inside the "mafia" don't even realize there is one. Thus, it looks >>> like we are all wasting time in this thread, since they think there >>> is nothing to be changed. > > But I am blaming you because you don't admit the fact that there *is* a > problem with the patch submission process. And we can't a solve a > problem if we don't admit there is one in the first place. I do think there is a problem with patch processing - I just don't think the barrier at contributing is high. The fact that there are so many patches contributed is proof that the barrier is not high: many people feel they can submit a patch. > You might be right, there's no way to really know of course. I think my > patch is a good example of how my proposal can be good for the Python > stdlib. My proposal (to recall) is to *automatically* apply any patch to > stdlib which follows normal guidelines if the package has no maintainer > and nobody objects in X days. While it would have worked in your case (although I still wonder who the automatic application of the patch should execute), please be ensured that this couldn't possibly work in general. Many patches are really really not acceptable in their initial form; for many of them, the biggest problem is that they lack documentation. There was a phase when patches got accepted with no documentation, for these modules, it was very difficult to come up with documentation in the following years. Regards, Martin From martin at v.loewis.de Thu Mar 8 11:53:29 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 08 Mar 2007 11:53:29 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <20070307214117.74CE.JCARLSON@uci.edu> References: <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> <20070307214117.74CE.JCARLSON@uci.edu> Message-ID: <45EFEB29.7080403@v.loewis.de> Josiah Carlson schrieb: > Because we should refuse the temptation to guess, what about: > > Rename the posix splitext to X (for some X), and offer a function with > identical functionality to the posix variant under win32, also available > as X for that platform. > > Rename the (current) win32 splitext to Y (for some Y) and offer a > function with identical functionality to the posix platform as name Y. > > Offer a new splitext that uses X on posix and Y on win32, but causes a > DeprecationWarning with pointers to the two renamed functions that are > available on both platforms. See my comment to Andrew. This would cause severe harm: there are many applications of splitext which work just fine and won't see any behavior change; there are those which actually got fixed; and then there are those that break. With the change, you are putting a burden on all three cases, and more so, on the users of the software. In addition, splitext should not be deprecated. It is a fine function, and I really don't see a conceptual need for two versions of it: you will only ever need the current one. > For people who want the old platform-specific functionality in previous > and subsequent Pythons, offer the ability to disable the > DeprecationWarning via a module global; at least until the function is > removed in some future Python, at which point path.splitext would cause > an AttributeError . That assumes there is a need for the old functionality. I really don't see it (pje claimed he needed it once, but I remain unconvinced, not having seen an actual fragment where the old behavior is helpful). Regards, Martin From p.f.moore at gmail.com Thu Mar 8 13:54:59 2007 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 8 Mar 2007 12:54:59 +0000 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45EF5DDF.5060200@v.loewis.de> References: <20070305184614.GA19051@localhost.localdomain> <200703052326.46667.phil@riverbankcomputing.co.uk> <45ED0371.5070108@v.loewis.de> <200703060848.04117.phil@riverbankcomputing.co.uk> <45ED39F0.2000907@v.loewis.de> <45EE82AC.2020103@develer.com> <45EE8F8E.3070409@v.loewis.de> <20070307174759.GF29575@caltech.edu> <45EF5DDF.5060200@v.loewis.de> Message-ID: <79990c6b0703080454h7dfdd1d8xce240888bb78a7bc@mail.gmail.com> On 08/03/07, "Martin v. L?wis" wrote: > Titus Brown schrieb: > > and it's not at all clear to outsiders like me how new > > features, old patches, and old bugs are dealt with. > > The simple answer is "when we have time". There really is not > more to it. Some patches get higher attention, e.g. because > they fix serious bugs. Proposed new features of don't get any > attention by the "mafia", because Python will just work fine > without the new feature. Just to elaborate a bit on Martin's comment. I (very occasionally) scan SF and review patches - I have no commit privilege, so that's all I can do. I find that "having enough time" is amazingly infrequent. Often, I start looking with all the best intentions, and get bogged down on one item, then find that real life has caught up and I've done what feels like nothing. As regards prioritisation, I don't know of any way of realistically doing this. All I do is scan the list (either from oldest to newest or vice versa depending on my mood) and look for "interesting" or "important" looking subjects. I suppose that emphasizes the need for using good subject lines, but not much else :-) As you can see, it's anything but scientific - and there's a lot of ways that "important" items could get missed. But it's not that way because I'm slacking, or snubbing particular areas or authors - it's just the best way I can find. If anyone can find a better way (and write it up as "How to do bug/patch reviews and triage" or something) that would be brilliant. I suspect there isn't one, though - at least with SF (Roundup may be better) and given the time and resources I have available. Paul. From facundo at taniquetil.com.ar Thu Mar 8 14:11:02 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Thu, 8 Mar 2007 13:11:02 +0000 (UTC) Subject: [Python-Dev] Encouraging developers References: <20070305184614.GA19051@localhost.localdomain> <45EF62A9.4010701@v.loewis.de> Message-ID: Martin v. L?wis wrote: >> - How can I know if a patch is still open? > > Easy: if it's marked as Open. > >> - I found a problem, and know how to fix it, but what else need to do? > > Go to www.python.org, then "CORE DEVELOPMENT", then "Patch submission". > >> - Found a problem in the docs, for this I must submit a patch or tell >> something about it? How? > > Read CORE DEVELOPMENT, Intro to development. > >> - I found an error in the docs, and fixed it, but I'm spanish speaker >> and my english sucks, can I submit a patch with bad wording or I must >> ask somebody to write it ok? > > What would your intuition tell you to do? Really didn't the answers to this, just were examples of questions that people may need to do, and feel shy to do it in python-dev... Ok, but I know now that python-dev *is* the place to ask. It's important, to redirect new possible developers and people willing to help. > patch, and perhaps the reviewer will catch the error. As a submitter, > just submit the code, and George Yoshida will catch it. It's not > strictly necessary that the documentation actually builds all > the time. If you want to be sure it builds, run "make html" in > Doc. Didn't know about the "make html", :) Thanks! -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From bjourne at gmail.com Thu Mar 8 14:20:21 2007 From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=) Date: Thu, 8 Mar 2007 14:20:21 +0100 Subject: [Python-Dev] Italic text in the manual Message-ID: <740c3aec0703080520r3237a403rba0046f575c0b5bd@mail.gmail.com> (This might be a silly question..) In the dev stdlib reference there are lots of pages in which all text is in italics. Such as all chapters after chapter 18. Is it supposed to be that way? It looks quite ugly. -- mvh Bj?rn From martin at v.loewis.de Thu Mar 8 14:42:49 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 08 Mar 2007 14:42:49 +0100 Subject: [Python-Dev] Italic text in the manual In-Reply-To: <740c3aec0703080520r3237a403rba0046f575c0b5bd@mail.gmail.com> References: <740c3aec0703080520r3237a403rba0046f575c0b5bd@mail.gmail.com> Message-ID: <45F012D9.5040309@v.loewis.de> BJ?rn Lindqvist schrieb: > (This might be a silly question..) > > In the dev stdlib reference there are lots of pages in which all text > is in italics. Such as all chapters after chapter 18. Certainly not. In today's copy (8.3.07, 13:30 GMT), this starts between 18.17 and 18.17.1. However, looking at the tex, I cannot find anything suspicious. texcheck complains about a missing ), which I added, but it only was a problem of the text, not of the markup. Some tex guru will have to figure this out; please submit a bugreport (if you can't figure it out yourself). Regards, Martin From Scott.Daniels at Acm.Org Thu Mar 8 15:28:50 2007 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Thu, 08 Mar 2007 06:28:50 -0800 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45EDB1F3.6060801@v.loewis.de> References: <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <45EDB1F3.6060801@v.loewis.de> Message-ID: Martin v. L?wis wrote: > Phillip J. Eby schrieb: >> I consider it correct, or at the least, don't think it should be >> changed, as it would make the behavior more difficult to reason about >> and introduce yet another thing to worry about when writing >> cross-version code. > > Now it's becoming difficult: several people in favor, some opposed... > > I'll wait a bit longer, but will still likely commit it, unless > opposition gets stronger: If the current behavior is incorrect > (in the sense that it contradicts wide-spread intuition), then > an application worrying about this detail should surely make the > 2.6 behavior also appear in 2.5 and earlier. > > I'm not sure what people actually use splitext for: I guess there > are two applications: > a) given a list of file names, give me all those belonging to a > hard-coded list of extensions (e.g. .py, .pyc, .c, .h). These > won't break, since they likely won't search for "all files > ending in .bash_profile" - there is only one per directory, > and if the want it, they use the entire filename. > b) given a list of file names, classify them for display (the > way the Windows explorer works, and similar file managers). > They use MIME databases and the like, and if they are unix-ish, > they probably reject the current splitext implementation already > as incorrect, and have work-arounds. As these files now show > up with "no extension", I rather expect that the work-around > won't trigger, and the default behavior will be the correct one. c) Given a filename, make an appropriately named associated file. pyo_name = os.path.splitext(name)[0] + '.pyo' This argues for os.path.splitext('.pythonrc') == ('.pythonrc','') -- -- Scott David Daniels Scott.Daniels at Acm.Org From snaury at gmail.com Thu Mar 8 18:36:18 2007 From: snaury at gmail.com (Alexey Borzenkov) Date: Thu, 8 Mar 2007 20:36:18 +0300 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <20070307080541.74AD.JCARLSON@uci.edu> References: <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <45EDB1F3.6060801@v.loewis.de> <20070307080541.74AD.JCARLSON@uci.edu> Message-ID: On 3/7/07, Josiah Carlson wrote: > "Martin v. L?wis" wrote: > > Now it's becoming difficult: several people in favor, some opposed... > What about changing the semantics of splitext and creating a new > function (available on all platforms) that does what the Windows version > currently does? I don't understand only one thing, why do people need new functions? You can anticipate the change today, and write functions that do exactly what you need no matter which way (current or proposed) python implements: def ensurenew(path): a,b = os.path.splitext(path) if a == '': # also possibly check if a contains dots only return b,a return a,b def ensureold(path): a,b = os.path.splitext(path) if b == '' and a.startswith('.'): # also possibly check if a starts with multiple dots return b,a return a,b Best regards, Alexey. From snaury at gmail.com Thu Mar 8 18:38:17 2007 From: snaury at gmail.com (Alexey Borzenkov) Date: Thu, 8 Mar 2007 20:38:17 +0300 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: References: <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <45EDB1F3.6060801@v.loewis.de> <20070307080541.74AD.JCARLSON@uci.edu> Message-ID: On 3/8/07, Alexey Borzenkov wrote: > On 3/7/07, Josiah Carlson wrote: > > "Martin v. L?wis" wrote: > > > Now it's becoming difficult: several people in favor, some opposed... > > What about changing the semantics of splitext and creating a new > > function (available on all platforms) that does what the Windows version > > currently does? > I don't understand only one thing, why do people need new functions? > You can anticipate the change today, and write functions that do > exactly what you need no matter which way (current or proposed) python > implements: [...snip...] On second thought I completely forgot about path parts. > Best regards, > Alexey. From martin at v.loewis.de Thu Mar 8 18:52:26 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 08 Mar 2007 18:52:26 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: References: <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <45EDB1F3.6060801@v.loewis.de> <20070307080541.74AD.JCARLSON@uci.edu> Message-ID: <45F04D5A.70006@v.loewis.de> Alexey Borzenkov schrieb: > I don't understand only one thing, why do people need new functions? > You can anticipate the change today, and write functions that do > exactly what you need no matter which way (current or proposed) python > implements: Indeed, that's also true. When people actually care, they can easily achieve compatibility across versions. Regards, Martin From martin at v.loewis.de Thu Mar 8 18:54:30 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 08 Mar 2007 18:54:30 +0100 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: References: <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <45EDB1F3.6060801@v.loewis.de> Message-ID: <45F04DD6.8090008@v.loewis.de> Scott David Daniels schrieb: > c) Given a filename, make an appropriately named associated file. > pyo_name = os.path.splitext(name)[0] + '.pyo' > This argues for os.path.splitext('.pythonrc') == ('.pythonrc','') Indeed, somebody found that people apparently do back_name = splitext(name[0]) + '.bak' which would severely break if somebody tried to create backups of dotfiles that way. Regards, Martin From phd at phd.pp.ru Thu Mar 8 19:02:05 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Thu, 8 Mar 2007 21:02:05 +0300 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45F04DD6.8090008@v.loewis.de> References: <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <45EDB1F3.6060801@v.loewis.de> <45F04DD6.8090008@v.loewis.de> Message-ID: <20070308180205.GA10322@phd.pp.ru> On Thu, Mar 08, 2007 at 06:54:30PM +0100, "Martin v. L?wis" wrote: > back_name = splitext(name[0]) + '.bak' back_name = splitext(name)[0] + '.bak' Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From collinw at gmail.com Thu Mar 8 20:14:07 2007 From: collinw at gmail.com (Collin Winter) Date: Thu, 8 Mar 2007 13:14:07 -0600 Subject: [Python-Dev] Moving test_unittest up in the test order Message-ID: <43aa6ff70703081114h1deec13dq47171a18be8719bf@mail.gmail.com> Now that unittest has a test suite, it would seem to make sense that it should be fairly high up in the testing order, since most of the regression suite depends on it. I'd like to have it included with test_grammar, test_opcodes, test_operations, etc, so that if a bug in unittest is causing other tests to fail, we don't have to wait until the end of the test run to find out. Thoughts? Collin Winter From pje at telecommunity.com Thu Mar 8 20:16:33 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 08 Mar 2007 14:16:33 -0500 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <45EFEB29.7080403@v.loewis.de> References: <20070307214117.74CE.JCARLSON@uci.edu> <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> <20070307214117.74CE.JCARLSON@uci.edu> Message-ID: <5.1.1.6.0.20070308141331.04c69da8@sparrow.telecommunity.com> At 11:53 AM 3/8/2007 +0100, Martin v. L?wis wrote: >That assumes there is a need for the old functionality. I really don't >see it (pje claimed he needed it once, but I remain unconvinced, not >having seen an actual fragment where the old behavior is helpful). The code in question was a type association handler that looked up loader functions based on file extension. This was specifically convenient for recognizing the difference between .htaccess files and other dotfiles that might appear in a web directory tree -- e.g. .htpasswd. The proposed change of splitext() would break that determination, because .htpasswd and .htaccess would both be considered files with empty extensions, and would be handled by the "empty extension" handler. From brett at python.org Thu Mar 8 20:16:36 2007 From: brett at python.org (Brett Cannon) Date: Thu, 8 Mar 2007 11:16:36 -0800 Subject: [Python-Dev] Moving test_unittest up in the test order In-Reply-To: <43aa6ff70703081114h1deec13dq47171a18be8719bf@mail.gmail.com> References: <43aa6ff70703081114h1deec13dq47171a18be8719bf@mail.gmail.com> Message-ID: On 3/8/07, Collin Winter wrote: > Now that unittest has a test suite, it would seem to make sense that > it should be fairly high up in the testing order, since most of the > regression suite depends on it. I'd like to have it included with > test_grammar, test_opcodes, test_operations, etc, so that if a bug in > unittest is causing other tests to fail, we don't have to wait until > the end of the test run to find out. > > Thoughts? Sounds reasonable to me, along with any tests for doctest. -Brett From collinw at gmail.com Thu Mar 8 21:02:42 2007 From: collinw at gmail.com (Collin Winter) Date: Thu, 8 Mar 2007 14:02:42 -0600 Subject: [Python-Dev] Moving test_unittest up in the test order In-Reply-To: References: <43aa6ff70703081114h1deec13dq47171a18be8719bf@mail.gmail.com> Message-ID: <43aa6ff70703081202j1ec5da0mf1a481d1e426f3aa@mail.gmail.com> On 3/8/07, Brett Cannon wrote: > On 3/8/07, Collin Winter wrote: > > Now that unittest has a test suite, it would seem to make sense that > > it should be fairly high up in the testing order, since most of the > > regression suite depends on it. I'd like to have it included with > > test_grammar, test_opcodes, test_operations, etc, so that if a bug in > > unittest is causing other tests to fail, we don't have to wait until > > the end of the test run to find out. > > > > Thoughts? > > Sounds reasonable to me, along with any tests for doctest. I've moved test_unittest, test_doctest and test_doctest2 up (rs 54226-8, in p3yk, trunk and release25-maint respectively). They'll now run right after test_types. Collin Winter From collinw at gmail.com Thu Mar 8 22:27:38 2007 From: collinw at gmail.com (Collin Winter) Date: Thu, 8 Mar 2007 15:27:38 -0600 Subject: [Python-Dev] Renaming Include/object.h In-Reply-To: References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org> <9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com> Message-ID: <43aa6ff70703081327s37fbdbdbkf2cdd782bf15153b@mail.gmail.com> On 1/3/07, Neal Norwitz wrote: > On 1/3/07, Thomas Wouters wrote: > > > > > > On 1/3/07, Guido van Rossum wrote: > > > On 1/3/07, Fred L. Drake, Jr. wrote: > > > > On Wednesday 03 January 2007 11:06, Martin v. L?wis wrote: > > > > > In #1626545, Anton Tropashko requests that object.h should be > > > > > renamed, because it causes conflicts with other software. > > > > > > > > > > I would like to comply with this requests for 2.6, assuming there > > > > > shouldn't be many problems with existing software as object.h > > > > > shouldn't be included directly, anyway. > > > > > > > > +1 > > > > > > Maybe this should be done in a more systematic fashion? E.g. by giving > > > all "internal" header files a "py_" prefix? > > > > I was thinking the same, and I'm sure Neal Norwitz is/was too (he suggested > > this a few times in the past, at least outside of python-dev.) > > Wow, I didn't realize I was that much of a broken record. :-) > I don't even remember talking to Thomas about it, only Guido. I > definitely would like to see all private header files clearly denoted > by their name or directory. > > I saw Jack's comment about Apple's naming scheme, but I'm ignoring > that for the moment. > I have bad memories from the Motif days of including everything with > one file. I prefer to see includes with the directory. This provides > a sort of namespace: > > #include "python/foo.h" > > Though, if the user only has to include a single Python.h like > currently, this wouldn't make as much sense. I don't recall the same > problems in Python that existed when using Motif. > > Here are some options (python/ can be omitted too): > > #include "python/public.h" > #include "python/internal/foo.h" > #include "python/private/foo.h" > #include "python/_private.h" > > I don't really like prefixing with py_ because from a user's > perspective I interpret py_ to be a public header that gives me a > namespace. I prefer a directory that indicates its intent because it > can't be misunderstood. IIRC Guido didn't want to introduce a new > directory. In that case my second choice is to prefix the filename > with an underscore as a leading underscore often is interpreted as > private. > > Going along with this change I would like to see all identifiers in > public header files prefixed with [_]Py. And public header files > shouldn't be able to include private header files (by convention). Or > we should have some other way of denoting which files must prefix all > their identifiers and which can use anything because they are only > used in Python code. > > For example, right now node (struct _node) is exported in node.h in > 2.4. I think it moved in 2.5, but it's still exported IIRC. Was any course of action ever decided on for this issue, or was the consensus that it would break too much code? If the latter, what about making the change for Python 3000? Collin Winter From facundo at taniquetil.com.ar Thu Mar 8 22:44:50 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Thu, 8 Mar 2007 21:44:50 +0000 (UTC) Subject: [Python-Dev] Adding timeout to socket.py and httplib.py Message-ID: I studied Skip patch, and I think he is in good direction: add a NetworkConnection object to socket.py, and then use it from the other modules. This NetworkConnection basically does what all the other modules do once and again, so no mistery about it (basically calls getaddrinfo over the received address, and try to open a socket to that address). I opened a new patch (#1676823) with the changes I propose regarding socket.py, because the goal from the two patches are different (my plan is go with the basic: first the change in socket.py and httplib, and no in all the other modules at this time). I do not know what to do with the previous patch (#723312), I guess it'll remain open until all the other modules get the timeout. Here're the differences between Skip patch and mine: - I only left changes regarding httplib and socket modules (both .py, docs, and NEWS). - I even removed a change in Python-ast.c (regarding __version__), but I don't know what's that for, so please enlighten me (thank you). - The NetworkConnection won't have a ``get_family`` method, if you need the family of the open socket, just ask the socket. - Added some test cases to test_socket.py regarding attributes, timeout and family; and a nice threaded test to actually try the timeout. - Added tests cases to test_httplib.py Feel free to review the patch, and commit it if you want (or tell me to do it after the review, it's just a command for me). Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From martin at v.loewis.de Thu Mar 8 22:55:26 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 08 Mar 2007 22:55:26 +0100 Subject: [Python-Dev] Renaming Include/object.h In-Reply-To: <43aa6ff70703081327s37fbdbdbkf2cdd782bf15153b@mail.gmail.com> References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org> <9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com> <43aa6ff70703081327s37fbdbdbkf2cdd782bf15153b@mail.gmail.com> Message-ID: <45F0864E.7060905@v.loewis.de> Collin Winter schrieb: > Was any course of action ever decided on for this issue, or was the > consensus that it would break too much code? If the latter, what about > making the change for Python 3000? Neither, nor. If a grand renaming is not feasible, I'd atleast do something about object.h for Python 2.6. Regards, Martin From glyph at divmod.com Fri Mar 9 01:31:35 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Fri, 09 Mar 2007 00:31:35 -0000 Subject: [Python-Dev] Policy Decisions, Judgment Calls, and Backwards Compatibility (was Re: splitext('.cshrc')) In-Reply-To: <45EFEB29.7080403@v.loewis.de> References: <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> <20070307214117.74CE.JCARLSON@uci.edu> <45EFEB29.7080403@v.loewis.de> Message-ID: <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> martin at v.loewis.de wrote: >That assumes there is a need for the old functionality. I really don't >see it (pje claimed he needed it once, but I remain unconvinced, not >having seen an actual fragment where the old behavior is helpful). This passage is symptomatic of the thing that really bothers me here. I have rarely used splitext before (since what I really want is determineMIMEType(), and that is easier to implement with one's own string-munging) and certainly won't use it again after following this discussion. The real issue I have here is one of process. Why is it that PJE (or any python user who wishes their code to keep working against new versions of Python) must frequent python-dev and convince you (or whatever Python developer might be committing a patch) of each use-case for old functionality? I would think that the goal here would be to keep all the old Python code running which it is reasonably possible to, regardless of whether the motivating use-cases are understood or not. It is the nature of infrastructure code to be used in bizarre and surprising ways. My understanding of the current backwards-compatibility policy for Python, the one that Twisted has been trying to emulate strictly, is that, for each potentially incompatible change, there will be: * at least one release with a pending deprecation warning and new, better API * at least one release with a deprecation warning * some number of releases later, the deprecated functionality is removed I was under the impression that this was documented in a PEP somewhere, but scanning the ones about backwards compatibility doesn't yield anything. I can't even figure out why I had this impression. *Is* there actually such a policy? If there isn't, there really should be. Deciding each one of these things on a case-by-case basis leaves a lot of wiggle room for a lot of old code to break in each release. For example, if you put me in charge of Python networking libraries and I simply used my judgment about what usages make sense and which don't, you might find that all the synchronous socket APIs were mysteriously gone within a few releases... ;-) Perhaps policy isn't the right way to solve the problem, but neither is asking every python application developer to meticulously follow and participate in every discussion on python-dev which *might* affect their code. As recently as last week, Guido commented that people build mental models of performance which have no relation to reality and we must rely on empirical data to see how things *really* perform. This is a common theme both here and anywhere good software development practices are discussed. These broken mental models are not limited to performance. In particular, people who develop software have inaccurate ideas about how it's really used. I am avoiding being specific to Python here because I've had a similarly broken idea of how people use Twisted, heard extremely strange ideas from kernel developers about the way programs in userland behave, and don't get me started on how C compiler writers think people write C code. If Python is not going to have an extremely conservative (and comprehensive, and strictly enforced) backwards-compatibility policy, we can't rely on those mental models as a way of determining what changes to allow. One way to deal with the question of "how do people really use python in the wild" is to popularize the community buildbots and make it easy to submit projects so that at least we have a picture of what Python developers are really doing. Buildbot has a "build this branch" feature which could be used to settle these discussions more rapidly, except for the fact that the community builders are currently in pretty sad shape: http://www.python.org/dev/buildbot/community/all/ By my count, currently only 9 out of 22 builders are passing. The severity of these failures varies (many of the builders are simply offline, not failing) but they should all be passing. If they were, rather than debating use-cases, we could at *least* have someone check this patch into a branch, and then build that branch across all these projects to see if any of them failed. Unfortunately open source code is quite often better tested and understood than the wider body of in-house and ad-hoc Python code out there, so it will be an easier target for changes like this than reality. I don't really have a problem with that though, because it creates a strong incentive for Python projects to both (A) be open source and (B) write comprehensive test suites, both of which are useful goals for many other reasons. In the past I've begged off of actually writing PEPs because I don't have the time, but if there is interest in codifying this I think I don't have the time *not* to write it. I'd prefer to document the pending/deprecate/remove policy first, but I can write up something more complicated about community buildbots and resolving disputes around potential breakages if there is actually no consensus about that. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070309/9e4d0d57/attachment.htm From glyph at divmod.com Fri Mar 9 02:20:56 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Fri, 09 Mar 2007 01:20:56 -0000 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <20070308180205.GA10322@phd.pp.ru> References: <20070306150459.GA8906@phd.pp.ru> <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <45EDB1F3.6060801@v.loewis.de> <45F04DD6.8090008@v.loewis.de> <20070308180205.GA10322@phd.pp.ru> Message-ID: <20070309012056.7769.1323665293.divmod.xquotient.259@joule.divmod.com> On 8 Mar, 06:02 pm, phd at phd.pp.ru wrote: >On Thu, Mar 08, 2007 at 06:54:30PM +0100, "Martin v. L?wis" wrote: >> back_name = splitext(name[0]) + '.bak' > >back_name = splitext(name)[0] + '.bak' This is really totally secondary to the point I actually care about, but seeing this antipattern bandied about as a reason to make the change is starting to bother me. There's no way to "fix" this idiom. Forget the corner case; what if you have a foobar.py and a foobar.txt in the same directory? This is not at all uncommon, and your "backup" function here will clobber those files in any case. Furthermore, although the module documentation is vague, the docstring for splitext specifically says "either part may be empty" and "extension is everything from the last dot to the end". Again, this is a case of needing a function designed to perform a specific task, and instead relying on half-broken idioms which involve other functions. make_backup_filename(x) might *use* splitext, but it is not what splitext is *for*. A correct implementation which did use splitext would look like this: def make_backup_filename(filename): base, extension = splitext(filename) return base + '.bak' + extension although personally I would probably prefer this: def make_backup_filename(filename): return filename + '.bak' If the behavior of the old code is going to be buggy in any case, it might as well be buggy and consistent. Consider a program that periodically makes and then cleans up backup files, and uses the "correct" splitext-using function above. Perhaps .cshrc.bak makes more sense than .bak.cshrc to some people, but that means that on a system where python is upgraded, .bak.cshrc will be left around forever. Even on a program whose functionality is improved by this change, the incompatibility between versions might create problems. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070309/f0d53bfe/attachment.htm From apt.shansen at gmail.com Fri Mar 9 03:08:20 2007 From: apt.shansen at gmail.com (Stephen Hansen) Date: Thu, 8 Mar 2007 18:08:20 -0800 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <20070309012056.7769.1323665293.divmod.xquotient.259@joule.divmod.com> References: <45ED6033.2020807@v.loewis.de> <20070306125356.GB22844@phd.pp.ru> <45ED81F1.502@v.loewis.de> <20070306150459.GA8906@phd.pp.ru> <5.1.1.6.0.20070306104954.028deec0@sparrow.telecommunity.com> <45EDB1F3.6060801@v.loewis.de> <45F04DD6.8090008@v.loewis.de> <20070308180205.GA10322@phd.pp.ru> <20070309012056.7769.1323665293.divmod.xquotient.259@joule.divmod.com> Message-ID: <7a9c25c20703081808w8873b60j35891def39f4b0b7@mail.gmail.com> I'm a long-term lurker and Python coder, and although I've never really contributed much to the list, I do make a point to keep up on it so I'm prepared at least when changes come through. This thread's gone on forever, so I thought I'd offer my opinion :) Mwha. Ahem. First of all, I think the current behavior is clearly broken; ".cshrc" is a file without an extension and marked as 'hidden' according to the conventions of the operating system. I totally think it should be fixed; but with others I'm worried about backwards compatability and more importantly the possibility of silent failures. Although none of my company's code will be hit (as I've always done fn.split('.')[-1] just... because it strikes me as more natural -- then again I'm in a situation where I don't have user-supplied filenames.), the thought that it's OK to make such changes even in a 'major' release is a bit disconcerting. Its not that I don't think there can be backwards-incompatible changes, but if at all possible they should be done in such a way that the change causes a hard failure or at least a clear warning in the offending code. I read that someone (... No idea who) suggested an optional keyword argument, and someone else objected to that on the grounds that it would let a second argument be passed in to alter the signature, and it would no longer throw an exception as people would be expecting. Well, I think it was a great idea-- whoever said it :) And gives the oppertunity to use the transitory period before 3.0 to loudly warn people about this fix. I don't expect a lot of people will be hit by it, but isn't that why this whole 2.6-to-3.0 thing is going on? Why wouldn't this work? I could submit a patch with a doc modification and tests even :P But it could begin the process of 'fixing' it, and warn people of the upcoming breakage, and although it slightly complicates the function... I think it only does it slightly :) (BTW, it raises a TypeError if the allow_dotfile isn't specified specifically, to address someone's objection that it would alter the signature) ----- import warnings def splitext(p, **kwargs): allow_dotfile = kwargs.pop('allow_dotfile', False) if kwargs: raise TypeError, "splitext() takes at most 2 arguments (%s given)" % (1 + len(kwargs)) i = p.rfind('.') if i<=max(p.rfind('/'), p.rfind('\\')): fn, ext = p, '' else: fn, ext = p[:i], p[i:] if allow_dotfile is False: if p.find('.') == 0: warnings.warn(FutureWarning('In Python 3.0, allow_dotfile default will become True.')) return fn, ext else: if p.find('.') == 0: return ext, '' else: return fn, ext ----- -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070308/55d1712f/attachment.htm From grig.gheorghiu at gmail.com Fri Mar 9 04:21:39 2007 From: grig.gheorghiu at gmail.com (Grig Gheorghiu) Date: Thu, 8 Mar 2007 19:21:39 -0800 Subject: [Python-Dev] Policy Decisions, Judgment Calls, and Backwards Compatibility (was Re: splitext('.cshrc')) In-Reply-To: <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> References: <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> <20070307214117.74CE.JCARLSON@uci.edu> <45EFEB29.7080403@v.loewis.de> <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> Message-ID: <3f09d5a00703081921y5b4e59a0w42723621c0dc67bb@mail.gmail.com> On 3/8/07, glyph at divmod.com wrote: > Buildbot has a "build this branch" feature which could be used to settle > these discussions more rapidly, except for the fact that the community > builders are currently in pretty sad shape: > > http://www.python.org/dev/buildbot/community/all/ > > By my count, currently only 9 out of 22 builders are passing. The severity > of these failures varies (many of the builders are simply offline, not > failing) but they should all be passing. If they were, rather than debating > use-cases, we could at *least* have someone check this patch into a branch, > and then build that branch across all these projects to see if any of them > failed. Titus and I are thinking about mentoring a Google Summer of Code project that would use the 'buildbot try' feature: set up a bunch of buildbot slaves and set them up so sending them a patch will trigger a checkout of the latest Python svn, followed by the application of the patch, followed by building and running unit tests for Python, followed by running test suites for various projects (similar to how it's being done in the community buildbot farm). This will hopefully give us a better grasp about how good a patch is, and will make the process of accepting patches more smooth. What do people think? Grig From steve at shrogers.com Fri Mar 9 04:11:20 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Thu, 08 Mar 2007 20:11:20 -0700 Subject: [Python-Dev] Policy Decisions, Judgment Calls, and Backwards Compatibility (was Re: splitext('.cshrc')) In-Reply-To: <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> References: <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> <20070307214117.74CE.JCARLSON@uci.edu> <45EFEB29.7080403@v.loewis.de> <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> Message-ID: <45F0D058.4080808@shrogers.com> glyph at divmod.com wrote: > > In the past I've begged off of actually writing PEPs because I don't > have the time, but if there is interest in codifying this I think I > don't have the time *not* to write it. I'd prefer to document the > pending/deprecate/remove policy first, but I can write up something more > complicated about community buildbots and resolving disputes around > potential breakages if there is actually no consensus about that. > Please do write this up. It's a good policy and should be codified. # Steve From tonynelson at georgeanelson.com Fri Mar 9 05:08:47 2007 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Thu, 8 Mar 2007 23:08:47 -0500 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: <5.1.1.6.0.20070308141331.04c69da8@sparrow.telecommunity.com> References: <20070307214117.74CE.JCARLSON@uci.edu> <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> <20070307214117.74CE.JCARLSON@uci.edu> <5.1.1.6.0.20070308141331.04c69da8@sparrow.telecommunity.com> Message-ID: At 2:16 PM -0500 3/8/07, Phillip J. Eby wrote: >At 11:53 AM 3/8/2007 +0100, Martin v. L?wis wrote: >>That assumes there is a need for the old functionality. I really don't >>see it (pje claimed he needed it once, but I remain unconvinced, not >>having seen an actual fragment where the old behavior is helpful). > >The code in question was a type association handler that looked up loader >functions based on file extension. This was specifically convenient for >recognizing the difference between .htaccess files and other dotfiles that >might appear in a web directory tree -- e.g. .htpasswd. The proposed >change of splitext() would break that determination, because .htpasswd and >.htaccess would both be considered files with empty extensions, and would >be handled by the "empty extension" handler. So, ".htaccess" and "foo.htaccess" should be treated the same way? Is that what Apache does? -- ____________________________________________________________________ TonyN.:' ' From anthony at interlink.com.au Fri Mar 9 06:36:20 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri, 9 Mar 2007 16:36:20 +1100 Subject: [Python-Dev] version-specific PYTHONPATH env var Message-ID: <200703091636.23343.anthony@interlink.com.au> What do people think about the idea of a version-specific PYTHONPATH environment variable? Something like PYTHON25PATH or the like. Reason I ask is that on our production systems, we have a couple of versions of Python being used by different systems. Yes, yes, in a perfect world they'd be all updated at the same time, sure. There's occasionally issues with the PYTHONPATH being pointed at something like .../lib/python2.4/site-packages or the like, and then have issues when python2.3 or some other different version is run. If we allowed people to optionally specify a more specific version this problem would go away. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From tjreedy at udel.edu Fri Mar 9 06:42:46 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 9 Mar 2007 00:42:46 -0500 Subject: [Python-Dev] Policy Decisions, Judgment Calls, and Backwards Compatibility (was Re: splitext('.cshrc')) References: <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com><20070308045229.GQ14306@steerpike.home.puzzling.org><20070307214117.74CE.JCARLSON@uci.edu><45EFEB29.7080403@v.loewis.de> <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> Message-ID: wrote in message news:20070309003135.7769.1063793143.divmod.xquotient.177 at joule.divmod.com... martin at v.loewis.de wrote: My understanding of the current backwards-compatibility policy for Python, the one that Twisted has been trying to emulate strictly, is that, for each potentially incompatible change, there will be: * at least one release with a pending deprecation warning and new, better API * at least one release with a deprecation warning * some number of releases later, the deprecated functionality is removed ================================================= There is not and hardly can be any such policy for bug fixes. Unfortunately, * Honest people can disagree about whether a change is a bug fix or feature addition or replacement. And there will be borderline cases regardless of one's definition, unless one makes either category empty. (Having only one category and hence only one change policy would simplify life but at some cost.) * Fixing bugs can break code just as it fixes other code. I believe the current policy is that depending on buggy behavior, as opposed to working around it, is proceed at your own risk. Bugs can be fixed without notice, at the next bug fix release. On the other hand, bug fixes can and sometime are delayed or modified in recognition of degrees of bugginess and differences of opinions and cost-benefit analyses. But deviations from a rigid procedure require case-by-case judgments, to me preferably based on information-gathering and discussion and a chance to influence judgment. For new features that don't break code (that doesn't depend on the features absence), the current (and somewhat new) policy is to wait until the next minor .x feature release, skipping any micro .x.y releases (now called bug-fix releases). For replacement features, the policy is what you give above. Example: f(*args) instead of apply(f, args). At one time, the expectation was that removal could be and possibly would be one release after deprecation. It is currently to wait until 3.0. What it will be after that I have not seen discussed. Semantic changes are trickier. We don't want a different API when we want a different meaning for the currently API. The integer division change is a mixture. Int//int replaces int/int and is already available. But int/int will change meaning to float(int)/float(int) rather than being removed. So I see five categories of code change -- clear bug, possibly fuzzy bug, semantic change, replacement, and addition -- each with different appropriate policies. And there are mixtures that belong together in a package. Terry Jan Reedy From guido at python.org Fri Mar 9 06:45:22 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 8 Mar 2007 21:45:22 -0800 Subject: [Python-Dev] version-specific PYTHONPATH env var In-Reply-To: <200703091636.23343.anthony@interlink.com.au> References: <200703091636.23343.anthony@interlink.com.au> Message-ID: I recommend not setting these variables at all, or to use wrapper scripts that set them instead. But there's probably some reason why you can't do that... On 3/8/07, Anthony Baxter wrote: > What do people think about the idea of a version-specific PYTHONPATH > environment variable? Something like PYTHON25PATH or the like. > Reason I ask is that on our production systems, we have a couple of > versions of Python being used by different systems. Yes, yes, in a > perfect world they'd be all updated at the same time, sure. There's > occasionally issues with the PYTHONPATH being pointed at something > like .../lib/python2.4/site-packages or the like, and then have > issues when python2.3 or some other different version is run. If we > allowed people to optionally specify a more specific version this > problem would go away. > > Anthony > -- > Anthony Baxter > It's never too late to have a happy childhood. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Fri Mar 9 08:57:08 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 09 Mar 2007 08:57:08 +0100 Subject: [Python-Dev] Policy Decisions, Judgment Calls, and Backwards Compatibility (was Re: splitext('.cshrc')) In-Reply-To: <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> References: <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> <20070307214117.74CE.JCARLSON@uci.edu> <45EFEB29.7080403@v.loewis.de> <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> Message-ID: <45F11354.2070700@v.loewis.de> glyph at divmod.com schrieb: > The real issue I have here is one of process. Why is it that PJE (or > any python user who wishes their code to keep working against new > versions of Python) must frequent python-dev and convince you (or > whatever Python developer might be committing a patch) of each use-case > for old functionality? Because this is the way free software works. I'm the one contributing (actually, merely integrating changes proposed by others, Larry Hastings in particular), so I get to make decisions. This patch had been open for three years. If pje had any problems with it, he could have rejected it. If you had problems with it, you could have commented that it should not be applied. Nobody did, so I decide. > I would think that the goal here would be to > keep all the old Python code running which it is reasonably possible to, > regardless of whether the motivating use-cases are understood or not. Exactly: which it is reasonably possible to. In this case, I found no reasonable way to keep it running while still fixing the bug, and not breaking too much other stuff in order to keep that particular code running. So the only alternative would have been to reject the change and live with the bug forever, something that many people here considered bad. > My understanding of the current backwards-compatibility policy for > Python, the one that Twisted has been trying to emulate strictly, is > that, for each potentially incompatible change, there will be: > > * at least one release with a pending deprecation warning and new, > better API > * at least one release with a deprecation warning > * some number of releases later, the deprecated functionality is removed If you want to contribute a patch that changes splitext to follow this policy, please go ahead and submit one. I personally consider a deprecation warning unacceptable in this specific case. > I was under the impression that this was documented in a PEP somewhere, > but scanning the ones about backwards compatibility doesn't yield > anything. I can't even figure out why I had this impression. *Is* > there actually such a policy? There is PEP 5, which is not followed, since it requires backwards-incompatible behavior to go through the PEP process. > If there isn't, there really should be. So submit a PEP. Or better, take over maintenance of PEP 5 (or better, cooperate with the author of the PEP on maintaining it). > Perhaps policy isn't the right way to solve the problem, but neither is > asking every python application developer to meticulously follow and > participate in every discussion on python-dev which *might* affect their > code. And there is no need to. If they find that their code breaks, they can submit bug reports, or, better, contribute patches. This is what alpha releases are for, and the trunk is buildable all the time, so you can test your code against the new release even before it happens. If you want to study the changes, you can read Misc/NEWS from time to time. > If Python is not going to have an extremely conservative (and > comprehensive, and strictly enforced) backwards-compatibility policy, we > can't rely on those mental models as a way of determining what changes > to allow. One way to deal with the question of "how do people really > use python in the wild" is to popularize the community buildbots and > make it easy to submit projects so that at least we have a picture of > what Python developers are really doing. I was going to say that. Now, if "the community" doesn't pick up the concept of community buildbots, "the community" has no right to complain (IMO, and I put that deliberately into quotes, knowing that "the community" doesn't exist as such, but is made up of individuals, some interested in the buildbots, stability, and so on, and others being less conservative). > In the past I've begged off of actually writing PEPs because I don't > have the time, but if there is interest in codifying this I think I > don't have the time *not* to write it. If you think it needs codification, I think you will need to write it yourself - nobody else will do it for you. Depending on what it will say, I might disagree. I do disagree with the requirement that every incompatible go through the PEP, and I understand "incompatible change" as "programs only relying on document behavior might break". In the case that triggered the discussion, the change implemented was not an incompatible change, because the new implementation still met the old specification (which, of course, was underspecified). Regards, Martin From martin at v.loewis.de Fri Mar 9 09:02:01 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 09 Mar 2007 09:02:01 +0100 Subject: [Python-Dev] Policy Decisions, Judgment Calls, and Backwards Compatibility (was Re: splitext('.cshrc')) In-Reply-To: <3f09d5a00703081921y5b4e59a0w42723621c0dc67bb@mail.gmail.com> References: <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> <20070307214117.74CE.JCARLSON@uci.edu> <45EFEB29.7080403@v.loewis.de> <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> <3f09d5a00703081921y5b4e59a0w42723621c0dc67bb@mail.gmail.com> Message-ID: <45F11479.1070106@v.loewis.de> Grig Gheorghiu schrieb: > Titus and I are thinking about mentoring a Google Summer of Code > project that would use the 'buildbot try' feature: set up a bunch of > buildbot slaves and set them up so sending them a patch will trigger a > checkout of the latest Python svn, followed by the application of the > patch, followed by building and running unit tests for Python, > followed by running test suites for various projects (similar to how > it's being done in the community buildbot farm). This will hopefully > give us a better grasp about how good a patch is, and will make the > process of accepting patches more smooth. > > What do people think? Sounds great! It would be even better if it could automatically pull patches off the roundup tracker, try to apply them, and post a message into roundup whether the patch applied and tested successfully. To prevent spamming, it may be necessary to explicitly release patches to buildbot (i.e. classify attachments as non-patch, released, or verified), but that could be done later. Regards, Martin From martin at v.loewis.de Fri Mar 9 09:11:19 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 09 Mar 2007 09:11:19 +0100 Subject: [Python-Dev] version-specific PYTHONPATH env var In-Reply-To: <200703091636.23343.anthony@interlink.com.au> References: <200703091636.23343.anthony@interlink.com.au> Message-ID: <45F116A7.7070300@v.loewis.de> Anthony Baxter schrieb: > What do people think about the idea of a version-specific PYTHONPATH > environment variable? I agree with Guido. I normally need an application-specific PYTHONPATH, and I do that by editing the start script adding a sys.path.append into it (these days, Python applications often come with a tiny start script that then imports the main functionality from a module). Examples where I do this include buildbot, moinmoin, and roundup. As a system's administrator, I discourage people from setting PATH, LD_LIBRARY_PATH, PYTHONPATH, and, if possible, CLASSPATH. Instead, I try to manage the system so that system-wide software is available to every user without additional setup. Regards, Martin From schmir at gmail.com Fri Mar 9 10:04:46 2007 From: schmir at gmail.com (Ralf Schmitt) Date: Fri, 9 Mar 2007 10:04:46 +0100 Subject: [Python-Dev] version-specific PYTHONPATH env var In-Reply-To: <200703091636.23343.anthony@interlink.com.au> References: <200703091636.23343.anthony@interlink.com.au> Message-ID: <932f8baf0703090104o617f9752vd1f7f91c5717fadf@mail.gmail.com> On 3/9/07, Anthony Baxter wrote: > > What do people think about the idea of a version-specific PYTHONPATH > environment variable? Something like PYTHON25PATH or the like. > Reason I ask is that on our production systems, we have a couple of > versions of Python being used by different systems. Yes, yes, in a > perfect world they'd be all updated at the same time, sure. There's > occasionally issues with the PYTHONPATH being pointed at something > like .../lib/python2.4/site-packages or the like, and then have > issues when python2.3 or some other different version is run. If we > allowed people to optionally specify a more specific version this > problem would go away. Few weeks ago I actually needed exactly this functionality. I worked around it with a .pth file installed for each version of python I'm using with the following contents: import sys; sys.path.insert(0, os.path.expanduser("~/pylib%s.%s" % sys.version_info[:2])) import site, os, sys; site.addsitedir(os.path.expanduser("~/pylib%s.%s" % sys.version_info[:2])) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070309/c0936f31/attachment.html From pmaupin at gmail.com Fri Mar 9 16:21:34 2007 From: pmaupin at gmail.com (Patrick Maupin) Date: Fri, 9 Mar 2007 09:21:34 -0600 Subject: [Python-Dev] splitext('.cshrc') In-Reply-To: References: <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> <20070307214117.74CE.JCARLSON@uci.edu> <5.1.1.6.0.20070308141331.04c69da8@sparrow.telecommunity.com> Message-ID: On 3/8/07, Tony Nelson wrote: > At 2:16 PM -0500 3/8/07, Phillip J. Eby wrote: > >The code in question was a type association handler that looked up loader > >functions based on file extension. This was specifically convenient for > >recognizing the difference between .htaccess files and other dotfiles that > >might appear in a web directory tree -- e.g. .htpasswd. The proposed > >change of splitext() would break that determination, because .htpasswd and > >.htaccess would both be considered files with empty extensions, and would > >be handled by the "empty extension" handler. > > So, ".htaccess" and "foo.htaccess" should be treated the same way? Is that > what Apache does? I've never personally used "foo.htaccess", but I have had files named, e.g. "test1.htaccess", or "backup.htaccess". And I don't know, but I assume a "foo.htaccess" would be an older or test version of a .htaccess file. So, my usecases say, "yes, they should be treated the same." Regards, Pat From lists at cheimes.de Fri Mar 9 17:03:16 2007 From: lists at cheimes.de (Christian Heimes) Date: Fri, 09 Mar 2007 17:03:16 +0100 Subject: [Python-Dev] datetime module enhancements Message-ID: Hello! This is my first posting to the Python developer list so please forgive me if I'm doing something wrong. In the past few weeks I've worked a lot with dates and times. I found several small annoyances in the datetime module. For example there was no obvious way to cast a timedelta to an int or float. I'm proposing some small additions to the datetime module: >>> td = timedelta(minutes=1, seconds=7, microseconds=500) >>> int(td) 67 >>> long(td) 67L >>> float(td) 67.5 >>> round(td) 68.0 datetime.datetime has a method (class factory) fromtimestamp() but its instances are missing a totimestamp() method that return the Unix timestamp for a date (seconds since 1970-01-01 00:00:00 UTC). Some people might miss a from and to julian day number (JDN) and modified julian day (MJD) converter in datetime.date and datetime.datetime. MJD is used as base in VMS and JDN, MJD and RJD are used by astronomers. It's also very useful when one has to deal with dates before 0 and converting dates between foreign calendars. >From http://www.slac.stanford.edu/~rkj/crazytime.txt --- November 17, 1858 is the base of the Modified Julian Day system. The original Julian Day (JD) is used by astronomers and expressed in days since noon January 1, 4713 B.C. This measure of time was introduced by Joseph Scaliger in the 16th century. It is named in honor of his father, Julius Caesar Scaliger (note that this Julian Day is different from the Julian calendar named for the Roman Emperor Julius Caesar!). --- http://en.wikipedia.org/wiki/Julian_Day I'm willing to implement the features myself but I need a mentor who helps me to correct my code and applies it to the Python svn repository. A patch for timedelta is already available at https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1665292&group_id=5470 Comments? Christian From pje at telecommunity.com Fri Mar 9 17:22:22 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 09 Mar 2007 11:22:22 -0500 Subject: [Python-Dev] Policy Decisions, Judgment Calls, and Backwards Compatibility (was Re: splitext('.cshrc')) In-Reply-To: <45F11354.2070700@v.loewis.de> References: <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> <20070307214117.74CE.JCARLSON@uci.edu> <45EFEB29.7080403@v.loewis.de> <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> Message-ID: <5.1.1.6.0.20070309111942.02ac4090@sparrow.telecommunity.com> At 08:57 AM 3/9/2007 +0100, Martin v. L??wis wrote: >In the case that triggered the discussion, the change implemented >was not an incompatible change, because the new implementation still >met the old specification (which, of course, was underspecified). No, it wasn't, actually. Read the doc strings, which state exactly what the code does. Windows: >>> import os >>> help(os.path.splitext) Help on function splitext in module ntpath: splitext(p) Split the extension from a pathname. Extension is everything from the last dot to the end. Return (root, ext), either part may be empty. Posix: >>> import os >>> help(os.path.splitext) Help on function splitext in module posixpath: splitext(p) Split the extension from a pathname. Extension is everything from the last dot to the end. Returns "(root, ext)", either part may be empty. From skip at pobox.com Fri Mar 9 17:31:17 2007 From: skip at pobox.com (skip at pobox.com) Date: Fri, 9 Mar 2007 10:31:17 -0600 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: Message-ID: <17905.35797.156230.974559@montanaro.dyndns.org> Christian> I'm proposing some small additions to the datetime module: >>> td = timedelta(minutes=1, seconds=7, microseconds=500) >>> int(td) 67 >>> long(td) 67L >>> float(td) 67.5 >>> round(td) 68.0 Casting to the number of seconds seems a bit arbitrary. Why not cast to the number of days or number of microseconds? If you allow interpretation of timedeltas as int, long or float I'd argue that round not be included. Instead, just round the output of float. Christian> datetime.datetime has a method (class factory) Christian> fromtimestamp() but its instances are missing a totimestamp() Christian> method that return the Unix timestamp for a date (seconds Christian> since 1970-01-01 00:00:00 UTC). The range of datetime objects far exceeds that of the current Unix timestamp. Given that the range of current (32-bit) Unix timestamps is approximately 1970 to 2038, What would the output of this be? dt = datetime.datetime(3000, 1, 1, 0, 0, 0) print dt.totimestamp() dt = datetime.datetime(1900, 1, 1, 0, 0, 0) print dt.totimestamp() Skip From guido at python.org Fri Mar 9 18:13:12 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Mar 2007 09:13:12 -0800 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <17905.35797.156230.974559@montanaro.dyndns.org> References: <17905.35797.156230.974559@montanaro.dyndns.org> Message-ID: On 3/9/07, skip at pobox.com wrote: > Christian> I'm proposing some small additions to the datetime module: > > >>> td = timedelta(minutes=1, seconds=7, microseconds=500) > >>> int(td) > 67 > >>> long(td) > 67L > >>> float(td) > 67.5 > >>> round(td) > 68.0 > > Casting to the number of seconds seems a bit arbitrary. Why not cast to the > number of days or number of microseconds? Because seconds are what's used everywhere else when Python represents time as a number (time.time(), time.sleep(), select, socket timeout, etc.) > If you allow interpretation of > timedeltas as int, long or float I'd argue that round not be included. > Instead, just round the output of float. > > Christian> datetime.datetime has a method (class factory) > Christian> fromtimestamp() but its instances are missing a totimestamp() > Christian> method that return the Unix timestamp for a date (seconds > Christian> since 1970-01-01 00:00:00 UTC). > > The range of datetime objects far exceeds that of the current Unix > timestamp. Given that the range of current (32-bit) Unix timestamps is > approximately 1970 to 2038, What would the output of this be? > > dt = datetime.datetime(3000, 1, 1, 0, 0, 0) > print dt.totimestamp() > dt = datetime.datetime(1900, 1, 1, 0, 0, 0) > print dt.totimestamp() If you extend the range to 64 bits there's no problem: the first should print 32503680000, the second -2208988800. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bjourne at gmail.com Fri Mar 9 18:24:34 2007 From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=) Date: Fri, 9 Mar 2007 18:24:34 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <17905.35797.156230.974559@montanaro.dyndns.org> Message-ID: <740c3aec0703090924g547dbaddt755d03937bb423e0@mail.gmail.com> On 3/9/07, Guido van Rossum wrote: > On 3/9/07, skip at pobox.com wrote: > > The range of datetime objects far exceeds that of the current Unix > > timestamp. Given that the range of current (32-bit) Unix timestamps is > > approximately 1970 to 2038, What would the output of this be? > > > > dt = datetime.datetime(3000, 1, 1, 0, 0, 0) > > print dt.totimestamp() > > dt = datetime.datetime(1900, 1, 1, 0, 0, 0) > > print dt.totimestamp() > > If you extend the range to 64 bits there's no problem: the first > should print 32503680000, the second -2208988800. I think it should be a ValueError, given that the programmer is very likely to further use the returned timestamp to for example insert stuff in a database. Unix timestamps are not unambiguously defined for any years other than 1970 to 2038 imho. -- mvh Bj?rn From guido at python.org Fri Mar 9 18:29:53 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Mar 2007 09:29:53 -0800 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <740c3aec0703090924g547dbaddt755d03937bb423e0@mail.gmail.com> References: <17905.35797.156230.974559@montanaro.dyndns.org> <740c3aec0703090924g547dbaddt755d03937bb423e0@mail.gmail.com> Message-ID: On 3/9/07, BJ?rn Lindqvist wrote: > On 3/9/07, Guido van Rossum wrote: > > On 3/9/07, skip at pobox.com wrote: > > > The range of datetime objects far exceeds that of the current Unix > > > timestamp. Given that the range of current (32-bit) Unix timestamps is > > > approximately 1970 to 2038, What would the output of this be? > > > > > > dt = datetime.datetime(3000, 1, 1, 0, 0, 0) > > > print dt.totimestamp() > > > dt = datetime.datetime(1900, 1, 1, 0, 0, 0) > > > print dt.totimestamp() > > > > If you extend the range to 64 bits there's no problem: the first > > should print 32503680000, the second -2208988800. > > I think it should be a ValueError, given that the programmer is very > likely to further use the returned timestamp to for example insert > stuff in a database. Unix timestamps are not unambiguously defined for > any years other than 1970 to 2038 imho. But they will be. And they already are on some platforms. The database should raise an OverflowError if a timestamp is out of range, but it would be unpythonic to limit those outcomes to 32 bits. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bjourne at gmail.com Fri Mar 9 18:51:18 2007 From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=) Date: Fri, 9 Mar 2007 18:51:18 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <17905.35797.156230.974559@montanaro.dyndns.org> <740c3aec0703090924g547dbaddt755d03937bb423e0@mail.gmail.com> Message-ID: <740c3aec0703090951hb8eb403i86c95b159ce925a1@mail.gmail.com> On 3/9/07, Guido van Rossum wrote: > On 3/9/07, BJ?rn Lindqvist wrote: > > On 3/9/07, Guido van Rossum wrote: > > > On 3/9/07, skip at pobox.com wrote: > > > > The range of datetime objects far exceeds that of the current Unix > > > > timestamp. Given that the range of current (32-bit) Unix timestamps is > > > > approximately 1970 to 2038, What would the output of this be? > > > > > > > > dt = datetime.datetime(3000, 1, 1, 0, 0, 0) > > > > print dt.totimestamp() > > > > dt = datetime.datetime(1900, 1, 1, 0, 0, 0) > > > > print dt.totimestamp() > > > > > > If you extend the range to 64 bits there's no problem: the first > > > should print 32503680000, the second -2208988800. > > > > I think it should be a ValueError, given that the programmer is very > > likely to further use the returned timestamp to for example insert > > stuff in a database. Unix timestamps are not unambiguously defined for > > any years other than 1970 to 2038 imho. > > But they will be. And they already are on some platforms. The database > should raise an OverflowError if a timestamp is out of range, but it > would be unpythonic to limit those outcomes to 32 bits. Then I guess datetime.fromtimestamp() also should be fixed?: >>> datetime.fromtimestamp(-1) Traceback (most recent call last): File "", line 1, in ValueError: timestamp out of range for platform localtime()/gmtime() function >>> datetime.fromtimestamp(-0.5) Traceback (most recent call last): File "", line 1, in ValueError: microsecond must be in 0..999999 >>> datetime.fromtimestamp(9999999999) Traceback (most recent call last): File "", line 1, in ValueError: timestamp out of range for platform time_t Would be nice if datetime.fromtimestamp(dt.totimestamp()) == dt worked for all datetimes. -- mvh Bj?rn From lists at cheimes.de Fri Mar 9 18:55:40 2007 From: lists at cheimes.de (Christian Heimes) Date: Fri, 09 Mar 2007 18:55:40 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <740c3aec0703090924g547dbaddt755d03937bb423e0@mail.gmail.com> References: <17905.35797.156230.974559@montanaro.dyndns.org> <740c3aec0703090924g547dbaddt755d03937bb423e0@mail.gmail.com> Message-ID: BJ?rn Lindqvist schrieb: > I think it should be a ValueError, given that the programmer is very > likely to further use the returned timestamp to for example insert > stuff in a database. Unix timestamps are not unambiguously defined for > any years other than 1970 to 2038 imho. IIRC the unix timestamp was originally definied as *signed* int with 32bit. http://en.wikipedia.org/wiki/Time_t --- Unix and POSIX-compliant systems implement the time_t type as a signed integer (typically 32 or 64 bits wide) which represents the number of seconds since the start of the Unix epoch: midnight UTC of January 1, 1970 (not counting leap seconds). Some systems support negative time values, while others do not. --- But you made a good point! What do you think about datetime.totimestamp(asLong=False)? The method would raise a ValueError or OverflowError if the value is > sys.maxint or < -(sys.maxint-1). Christian From lists at cheimes.de Fri Mar 9 19:07:27 2007 From: lists at cheimes.de (Christian Heimes) Date: Fri, 09 Mar 2007 19:07:27 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <740c3aec0703090951hb8eb403i86c95b159ce925a1@mail.gmail.com> References: <17905.35797.156230.974559@montanaro.dyndns.org> <740c3aec0703090924g547dbaddt755d03937bb423e0@mail.gmail.com> <740c3aec0703090951hb8eb403i86c95b159ce925a1@mail.gmail.com> Message-ID: BJ?rn Lindqvist schrieb: > Then I guess datetime.fromtimestamp() also should be fixed?: [...] > Would be nice if datetime.fromtimestamp(dt.totimestamp()) == dt worked > for all datetimes. Yeah great idea but I'm not sure if I'm up to the task. I'm not that familiar with C level date and time libraries. Christian From martin at v.loewis.de Fri Mar 9 19:18:23 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 09 Mar 2007 19:18:23 +0100 Subject: [Python-Dev] Policy Decisions, Judgment Calls, and Backwards Compatibility (was Re: splitext('.cshrc')) In-Reply-To: <5.1.1.6.0.20070309111942.02ac4090@sparrow.telecommunity.com> References: <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> <20070307214117.74CE.JCARLSON@uci.edu> <45EFEB29.7080403@v.loewis.de> <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> <5.1.1.6.0.20070309111942.02ac4090@sparrow.telecommunity.com> Message-ID: <45F1A4EF.3030109@v.loewis.de> Phillip J. Eby schrieb: > At 08:57 AM 3/9/2007 +0100, Martin v. L?wis wrote: >> In the case that triggered the discussion, the change implemented >> was not an incompatible change, because the new implementation still >> met the old specification (which, of course, was underspecified). > > No, it wasn't, actually. Read the doc strings, which state exactly what > the code does. The doc strings were precise, yes. The documentation (Doc/lib) was underspecified and allowed for both interpretations: splitext(path) Split the pathname path into a pair (root, ext) such that root + ext == path, and ext is empty or begins with a period and contains at most one period. Regards, Martin From ziga.seilnacht at gmail.com Fri Mar 9 20:52:20 2007 From: ziga.seilnacht at gmail.com (=?UTF-8?B?xb1pZ2EgU2VpbG5hY2h0?=) Date: Fri, 09 Mar 2007 20:52:20 +0100 Subject: [Python-Dev] Introduction Message-ID: <45F1BAF4.7060405@gmail.com> Hi all! I have just accepted an invitation to become a Python developer, so I feel obliged to introduce myself. My name is ?iga Seilnacht. I'm currently studying civil engineering at the University of Ljubljana, Slovenia. I started learning Python approximately two years ago, as my first computer language. I got involved with Python development a year ago, on the last bug day. My experience was much more pleasant than Michael Foord's; Georg checked in my first patch in under 30 minutes, and he had to fix my tests first :). As a Python developer, I will continue to focus on bug triage, bug fixing and reviewing patches; I don't feel I'm qualified for language design. I'm occasionally lurking on various Python related IRC channels under the nick zseil. Feel free to contact me! Regards, Ziga From collinw at gmail.com Fri Mar 9 20:53:23 2007 From: collinw at gmail.com (Collin Winter) Date: Fri, 9 Mar 2007 13:53:23 -0600 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: Message-ID: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> On the subject of datetime enhancements, I came across an SF patch (#1673403) the other day that proposed making it possible to compare date and datetime objects. Quoting from the patch summary: """ Comparing a date to a datetime currently throws an exception. This makes no sense. In what way is: datetime(2006, 1, 1, 0, 0, 0) < date(2007, 1, 1) not a perfectly reasonable and well-defined comparison? Throwing an exception here violates the "Principle of Least Surprise" to a considerable degree. Obviously some slight ambiguity arises if the date and the datetime differ only in the time part. There are two sensible responses in this situation that I can see: Treat dates as if they have a time-part of midnight. This is my preferred solution, and it is already what the datetime module does, for example, when subtracting two dates. Treat dates as if they refer to the entire day, i.e. if the date and datetime differ only in the time part then they are equal. This is consistent but becomes confusing in other situations such as when subtracting dates. """ Any thoughts on this? Collin Winter From martin at v.loewis.de Fri Mar 9 20:56:00 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 09 Mar 2007 20:56:00 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <17905.35797.156230.974559@montanaro.dyndns.org> <740c3aec0703090924g547dbaddt755d03937bb423e0@mail.gmail.com> Message-ID: <45F1BBD0.2000707@v.loewis.de> Christian Heimes schrieb: > BJ?rn Lindqvist schrieb: >> I think it should be a ValueError, given that the programmer is very >> likely to further use the returned timestamp to for example insert >> stuff in a database. Unix timestamps are not unambiguously defined for >> any years other than 1970 to 2038 imho. > > IIRC the unix timestamp was originally definied as *signed* int with 32bit. "Unix" time stamps where never defined as being 32bit. That is just an implementation detail. time_t was defined as an int, in second since 1970, and that is all what was defined. Whether or not an int is 32bits depends on the hardware (and the compiler); this was never part of Unix. Regards, Martin From guido at python.org Fri Mar 9 21:00:10 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Mar 2007 12:00:10 -0800 Subject: [Python-Dev] Introduction In-Reply-To: <45F1BAF4.7060405@gmail.com> References: <45F1BAF4.7060405@gmail.com> Message-ID: On 3/9/07, ?iga Seilnacht wrote: > Hi all! > > I have just accepted an invitation to become a Python > developer, so I feel obliged to introduce myself. Nice to meet you, Ziga! > My name is ?iga Seilnacht. I'm currently studying civil > engineering at the University of Ljubljana, Slovenia. > > I started learning Python approximately two years ago, as > my first computer language. I got involved with Python > development a year ago, on the last bug day. My experience > was much more pleasant than Michael Foord's; Georg checked > in my first patch in under 30 minutes, and he had to fix > my tests first :). > > As a Python developer, I will continue to focus on bug > triage, bug fixing and reviewing patches; I don't feel > I'm qualified for language design. Yet. :-) > I'm occasionally lurking on various Python related IRC > channels under the nick zseil. Feel free to contact me! > > Regards, > Ziga -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Fri Mar 9 21:00:16 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 09 Mar 2007 21:00:16 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <740c3aec0703090924g547dbaddt755d03937bb423e0@mail.gmail.com> References: <17905.35797.156230.974559@montanaro.dyndns.org> <740c3aec0703090924g547dbaddt755d03937bb423e0@mail.gmail.com> Message-ID: <45F1BCD0.2030503@v.loewis.de> BJ?rn Lindqvist schrieb: >> If you extend the range to 64 bits there's no problem: the first >> should print 32503680000, the second -2208988800. > > I think it should be a ValueError, given that the programmer is very > likely to further use the returned timestamp to for example insert > stuff in a database. Then this operation (the insertion into a database) should give a ValueError. Python conceptually has only a single integer type, and that has no range limitation. Of course, if "conversion to time_t" was an operation in datetime, than this should limit it in the range of time_t (which may or may not have 32 bits). > Unix timestamps are not unambiguously defined for > any years other than 1970 to 2038 imho. As others have said: this is simply not true. It depends on the hardware, Unix explicitly, deliberately, leaves that open to the specific operating system implementation. On a 36-bit hardware, the range will be different. Regards, Martin From brett at python.org Fri Mar 9 21:03:08 2007 From: brett at python.org (Brett Cannon) Date: Fri, 9 Mar 2007 12:03:08 -0800 Subject: [Python-Dev] Introduction In-Reply-To: <45F1BAF4.7060405@gmail.com> References: <45F1BAF4.7060405@gmail.com> Message-ID: On 3/9/07, ?iga Seilnacht wrote: > Hi all! > > I have just accepted an invitation to become a Python > developer, so I feel obliged to introduce myself. > > My name is ?iga Seilnacht. I'm currently studying civil > engineering at the University of Ljubljana, Slovenia. > Welcome aboard! > I started learning Python approximately two years ago, as > my first computer language. I got involved with Python > development a year ago, on the last bug day. My experience > was much more pleasant than Michael Foord's; Georg checked > in my first patch in under 30 minutes, and he had to fix > my tests first :). > > As a Python developer, I will continue to focus on bug > triage, bug fixing and reviewing patches; I don't feel > I'm qualified for language design. > Don't worry about it. You can still express your opinion as a user of Python. Language design is more about getting hit with some inspiration for something than being "qualified" for it. -Brett From python at rcn.com Fri Mar 9 21:04:41 2007 From: python at rcn.com (Raymond Hettinger) Date: Fri, 9 Mar 2007 15:04:41 -0500 (EST) Subject: [Python-Dev] Introduction Message-ID: <20070309150441.BAX27728@ms09.lnh.mail.rcn.net> [?iga Seilnacht] >I have just accepted an invitation to become a Python >developer, so I feel obliged to introduce myself. Welcome aboard. Raymond From martin at v.loewis.de Fri Mar 9 21:05:10 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 09 Mar 2007 21:05:10 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> Message-ID: <45F1BDF6.9020208@v.loewis.de> Collin Winter schrieb: > > Any thoughts on this? There are know problems comparing durations (e.g. is 30 days more or less than a month?). For time stamps, there is no issue. For calender dates, there are again problems, in particular with time zones. What I don't know (because I never used it) is whether the datetime module implements time stamps. If it does (i.e. no durations, no time zones, no issues with leap seconds), then supporting a total order seems reasonable. Regards, Martin From g.brandl at gmx.net Fri Mar 9 21:09:30 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 09 Mar 2007 21:09:30 +0100 Subject: [Python-Dev] Introduction In-Reply-To: <45F1BAF4.7060405@gmail.com> References: <45F1BAF4.7060405@gmail.com> Message-ID: ?iga Seilnacht schrieb: > Hi all! > > I have just accepted an invitation to become a Python > developer, so I feel obliged to introduce myself. Welcome from me too! I was always pleased to see your name in some SF issue because it generally meant to look for high-quality comment or patch next to it :) cheers, Georg From brett at python.org Fri Mar 9 21:33:30 2007 From: brett at python.org (Brett Cannon) Date: Fri, 9 Mar 2007 12:33:30 -0800 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> Message-ID: On 3/9/07, Collin Winter wrote: > On the subject of datetime enhancements, I came across an SF patch > (#1673403) the other day that proposed making it possible to compare > date and datetime objects. Quoting from the patch summary: > > """ > Comparing a date to a datetime currently throws an exception. This makes no > sense. In what way is: > > datetime(2006, 1, 1, 0, 0, 0) < date(2007, 1, 1) > > not a perfectly reasonable and well-defined comparison? Throwing an > exception here violates the "Principle of Least Surprise" to a considerable > degree. > > Obviously some slight ambiguity arises if the date and the datetime differ > only in the time part. There are two sensible responses in this situation > that I can see: > > Treat dates as if they have a time-part of midnight. This is my preferred > solution, and it is already what the datetime module does, for example, > when subtracting two dates. > > Treat dates as if they refer to the entire day, i.e. if the date and > datetime differ only in the time part then they are equal. This is > consistent but becomes confusing in other situations such as when > subtracting dates. > """ > > Any thoughts on this? > I personally like the current solution. The proposal to just assume midnight goes against EIBTI in my mind. -Brett From arigo at tunes.org Fri Mar 9 21:45:17 2007 From: arigo at tunes.org (Armin Rigo) Date: Fri, 9 Mar 2007 21:45:17 +0100 Subject: [Python-Dev] [Python-checkins] r53997 - in python/trunk: Lib/test/crashers/modify_dict_attr.py Lib/test/test_descr.py Objects/typeobject.c In-Reply-To: <20070227182950.28E361E4002@bag.python.org> References: <20070227182950.28E361E4002@bag.python.org> Message-ID: <20070309204517.GA22693@code0.codespeak.net> Hi Jeremy, On Tue, Feb 27, 2007 at 07:29:50PM +0100, jeremy.hylton wrote: > Removed: > python/trunk/Lib/test/crashers/modify_dict_attr.py > Modified: > python/trunk/Lib/test/test_descr.py > python/trunk/Objects/typeobject.c > Log: > Add checking for a number of metaclass error conditions. This check-in looks highly suspicious to me. It doesn't solve the problem it set out to solve, which is the modify_dict_attr crasher - see the slightly modified modify_dict_attr that I re-checked in (Lib/test/crashers/). Instead it adds more of the obscure undocumented rules about what kind of inheritence is allowed or not. We already have cases like: class X(str): pass class Y(str): pass class Z(X, Y): pass which are forbidden for no user-discernable reason. I don't think that forbidding: class X(Y, type): pass is going to help clarity there. The problem with modify_dict_attr is at a different level; if needed I can elaborate on the following comment from the SF tracker: """A proposed fix [for modify_dict_attr]: the __dict__ descriptor of user-defined classes should verify if there is another __dict__ descriptor along the solid path of the type (i.e. following "tp_base" pointers). If so, it should delegate to it.""" A bientot, Armin. From collinw at gmail.com Fri Mar 9 22:02:11 2007 From: collinw at gmail.com (Collin Winter) Date: Fri, 9 Mar 2007 15:02:11 -0600 Subject: [Python-Dev] [Python-checkins] buildbot warnings in g4 osx.4 trunk In-Reply-To: <20070309200606.3C25E1E4002@bag.python.org> References: <20070309200606.3C25E1E4002@bag.python.org> Message-ID: <43aa6ff70703091302x1822f492l3e869da0ec1c9270@mail.gmail.com> (Resending to python-dev instead of python-checkins) Any objections to applying something like the following to address the recent spate of buildbot failures in test_posixpath:test_islink? Index: Lib/test/test_posixpath.py --- Lib/test/test_posixpath.py (revision 54248) +++ Lib/test/test_posixpath.py (working copy) @@ -150,6 +150,11 @@ os.remove(test_support.TESTFN) def test_islink(self): + try: + os.remove(test_support.TESTFN + "2") + except os.error: + pass + self.assertIs(posixpath.islink(test_support.TESTFN + "1"), False) f = open(test_support.TESTFN + "1", "wb") try: Collin Winter On 3/9/07, buildbot at python.org wrote: > The Buildbot has detected a new failure of g4 osx.4 trunk. > Full details are available at: > http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/1814 > > Buildbot URL: http://www.python.org/dev/buildbot/all/ > > Build Reason: > Build Source Stamp: [branch trunk] HEAD > Blamelist: collin.winter,thomas.heller > > Build had warnings: warnings test > > Excerpt from the test logfile: > 1 test failed: > test_posixpath > > make: *** [buildbottest] Error 1 > > sincerely, > -The Buildbot > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > From pje at telecommunity.com Fri Mar 9 22:04:08 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 09 Mar 2007 16:04:08 -0500 Subject: [Python-Dev] Policy Decisions, Judgment Calls, and Backwards Compatibility (was Re: splitext('.cshrc')) In-Reply-To: <45F1A4EF.3030109@v.loewis.de> References: <5.1.1.6.0.20070309111942.02ac4090@sparrow.telecommunity.com> <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> <20070307214117.74CE.JCARLSON@uci.edu> <45EFEB29.7080403@v.loewis.de> <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> <5.1.1.6.0.20070309111942.02ac4090@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070309154617.04a23ec8@sparrow.telecommunity.com> At 07:18 PM 3/9/2007 +0100, Martin v. L?wis wrote: >Phillip J. Eby schrieb: > > At 08:57 AM 3/9/2007 +0100, Martin v. L?wis wrote: > >> In the case that triggered the discussion, the change implemented > >> was not an incompatible change, because the new implementation still > >> met the old specification (which, of course, was underspecified). > > > > No, it wasn't, actually. Read the doc strings, which state exactly what > > the code does. > >The doc strings were precise, yes. The documentation (Doc/lib) was >underspecified and allowed for both interpretations: > >splitext(path) >Split the pathname path into a pair (root, ext) such that root + ext == >path, and ext is empty or begins with a period and contains at most one >period. I'm just pointing out that anyone who reads *all* the documentation would reasonably conclude that the more specific of the two pieces is a reliable description of the behavior. Therefore, this is a change to documented behavior, and can't be considered a bug fix. I agree 100% with the people who think splitext() should arguably have had the new proposed behavior from the beginning. However, I also agree 100% with Glyph that it's ridiculous that we should even have to be having this argument about changing something that's been like this for about a *decade* and can reasonably be considered to not be a bug, given that its detailed behavior is clearly documented in the docstrings. Where were all these "10 Unix programmers" for the last ten years? Obviously they've been writing their own replacement for splitext(), if and when they needed one -- or else the normal use cases for splitext either aren't dotfile-sensitive, or are *conducive to the existing implementation*. In other words, one of four cases apply for existing code: 1. Old code uses its own fucntion - change has no effect 2. Old code uses splitext, but never has and never will touch dotfiles - change has no effect 3. Old code uses splitext, and has never touched dotfiles (otherwise it would have been changed to not use splitext!) but *might*, *possibly*, touch one at some point in the future - change *might* fix a latent bug in such code, but clearly it's unlikely 4. Old code uses splitext, does touch dotfiles, *likes it that way* - change breaks code. (A small, but definitely non-empty group.) So, ignoring the cases where the change makes no difference, this change will: 1. DEFINITELY break some code 2. MAYBE "fix" some code that hasn't actually broken before I don't see how a hypothetical fix for one small group justifies definitely breaking the code of some other small group. The only group the change benefits is people writing *new* code -- so give them a new function. From fumanchu at amor.org Fri Mar 9 22:06:14 2007 From: fumanchu at amor.org (Robert Brewer) Date: Fri, 9 Mar 2007 13:06:14 -0800 Subject: [Python-Dev] datetime module enhancements In-Reply-To: Message-ID: <435DF58A933BA74397B42CDEB8145A860A00F637@ex9.hostedexchange.local> Brett Cannon wrote: > On 3/9/07, Collin Winter wrote: > > On the subject of datetime enhancements, I came across an SF patch > > (#1673403) the other day that proposed making it possible to compare > > date and datetime objects... > > I personally like the current solution. The proposal to just assume > midnight goes against EIBTI in my mind. Yeah, but the current solution goes against, um, APBP*. Not in my mind, but in my code. Repeatedly. I can't count how many times I've written code like: if created > fdate: when I "should" have written: if isinstance(created, datetime.date): date_version_of_created = created else: date_version_of_created = created.date() if date_version_of_created > fdate: But it gets better, because: >>> isinstance(datetime.datetime(x, y, z), datetime.date) True So the above won't work, you must remember to reverse the if/else: if isinstance(created, datetime.datetime): date_version_of_created = created.date() else: date_version_of_created = created if date_version_of_created > fdate: That's at least one too many "must remembers" for dumb (and busy!) ol' me. EIBTI until it's a PITA. Is an implicit time of 0 really so surprising? It doesn't seem to be surprising for the datetime constructor: >>> datetime.datetime(2007, 3, 9) datetime.datetime(2007, 3, 9, 0, 0) Why should it be surprising for comparisons? Robert Brewer System Architect Amor Ministries fumanchu at amor.org * APBP = "Although, practicality beats purity" From collinw at gmail.com Fri Mar 9 22:14:42 2007 From: collinw at gmail.com (Collin Winter) Date: Fri, 9 Mar 2007 15:14:42 -0600 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> Message-ID: <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> On 3/9/07, Collin Winter wrote: > On the subject of datetime enhancements, I came across an SF patch > (#1673403) the other day that proposed making it possible to compare > date and datetime objects. Quoting from the patch summary: > > """ > Comparing a date to a datetime currently throws an exception. This makes no > sense. In what way is: > > datetime(2006, 1, 1, 0, 0, 0) < date(2007, 1, 1) > > not a perfectly reasonable and well-defined comparison? Throwing an > exception here violates the "Principle of Least Surprise" to a considerable > degree. > > Obviously some slight ambiguity arises if the date and the datetime differ > only in the time part. There are two sensible responses in this situation > that I can see: > > Treat dates as if they have a time-part of midnight. This is my preferred > solution, and it is already what the datetime module does, for example, > when subtracting two dates. > > Treat dates as if they refer to the entire day, i.e. if the date and > datetime differ only in the time part then they are equal. This is > consistent but becomes confusing in other situations such as when > subtracting dates. > """ > > Any thoughts on this? One solution that just occurred to me -- and that skirts the issue of choosing an interpretation -- is that, when comparing date and datetime objects, the datetime's .date() method is called and the result of that call is compared to the original date. That is, datetime_obj < date_obj is implicitly equivalent to datetime_obj.date() < date_obj Seems a ready-made use case for that method. Collin Winter From jon+python-dev at unequivocal.co.uk Fri Mar 9 22:36:20 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Fri, 9 Mar 2007 21:36:20 +0000 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <45F1BDF6.9020208@v.loewis.de> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <45F1BDF6.9020208@v.loewis.de> Message-ID: <20070309213620.GQ32401@snowy.squish.net> "\"Martin v. L?wis\"" wrote: > There are know problems comparing durations (e.g. is 30 days more > or less than a month?). For time stamps, there is no issue. For > calender dates, there are again problems, in particular with time > zones. Python durations (datetime.timedelta) do not support the concept of "months", so that's not an issue. The timedelta type only supports durations that are a fixed number of seconds. I don't see what the issue with time zones would be? Surely you just convert to the same time zone and then compare? > What I don't know (because I never used it) is whether the datetime > module implements time stamps. If it does (i.e. no durations, > no time zones, no issues with leap seconds), then supporting a > total order seems reasonable. It supports converting *from* timestamps, but not *to* timestamps, which is the subject of a feature request I submitted the other day (1673409) ;-) From jon+python-dev at unequivocal.co.uk Fri Mar 9 22:20:33 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Fri, 9 Mar 2007 21:20:33 +0000 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> Message-ID: <20070309212033.GP32401@snowy.squish.net> Brett Cannon wrote: > > Treat dates as if they have a time-part of midnight. This is my preferred > > solution, and it is already what the datetime module does, for example, > > when subtracting two dates. > > I personally like the current solution. The proposal to just assume > midnight goes against EIBTI in my mind. Not in mine. The comparison of a date with a datetime throwing an exception is astonishing in my mind. The problem is, if you don't define a 'date' as I suggest (i.e. midnight at the start of the day in question), then what *does* a 'date' represent? If you want the answer to be "an unknown time on that day" then you need to alter the datetime module so that subtracting one date from another throws an exception. If you want the answer to be "the entire of that day" then you need to alter the datetime module so that, e.g. subtracting 2007-03-08 from 2007-03-09 does not return "one day" as currently, but returns "zero days" instead (since of course there is no time between the end of one day and the start of the next day). Obviously both of those alternatives are ludicrous (not to mention they would break backwards-compatibility), so I would suggest the only logical solution is to make the change I proposed ;-) From steven.bethard at gmail.com Fri Mar 9 23:03:02 2007 From: steven.bethard at gmail.com (Steven Bethard) Date: Fri, 9 Mar 2007 15:03:02 -0700 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> Message-ID: On 3/9/07, Collin Winter wrote: > One solution that just occurred to me -- and that skirts the issue of > choosing an interpretation -- is that, when comparing date and > datetime objects, the datetime's .date() method is called and the > result of that call is compared to the original date. That is, > > datetime_obj < date_obj > > is implicitly equivalent to > > datetime_obj.date() < date_obj Using the .date() is fine when the year/month/day doesn't match. So the following are fine:: datetime.datetime(2005, 1, 1, 0, 0, 0) < datetime.date(2006, 1, 1) datetime.datetime(2007, 1, 1, 0, 0, 0) > datetime.date(2006, 1, 1) It's *not* okay to say that a date() is less than, greater than or equal to a datetime() if the year/month/day *does* match. The correct temporal relation is During, but Python doesn't have a During operator. During is not the same as less-than, greater-than or equal-to, so all of these should be False:: datetime.datetime(2006, 1, 1, 0, 0, 0) < datetime.date(2006, 1, 1) datetime.datetime(2006, 1, 1, 0, 0, 0) > datetime.date(2006, 1, 1) datetime.datetime(2006, 1, 1, 0, 0, 0) == datetime.date(2006, 1, 1) That is, the datetime() is not less than, greater than or equal to the corresponding date(). Some discussion of these kinds of issues is here: http://citeseer.ist.psu.edu/allen94actions.html The essence is that in order to properly compare intervals, you need the Meets, Overlaps, Starts, During and Finishes operators in addition to the Before (<) and Simulaneous (=) operators. So, let's not conflate Before, After or Simultaneous with the other relations -- if it's not strictly Before (<), After (>) or Simultaneous (=), we can just say so by returning False. (I believe these semantics would be just fine for both of the examples offered so far, but let me know if you think that's not true.) STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From collinw at gmail.com Fri Mar 9 23:08:47 2007 From: collinw at gmail.com (Collin Winter) Date: Fri, 9 Mar 2007 16:08:47 -0600 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> Message-ID: <43aa6ff70703091408w30476ea6t532c1317a29e4118@mail.gmail.com> On 3/9/07, Steven Bethard wrote: > On 3/9/07, Collin Winter wrote: > > One solution that just occurred to me -- and that skirts the issue of > > choosing an interpretation -- is that, when comparing date and > > datetime objects, the datetime's .date() method is called and the > > result of that call is compared to the original date. That is, > > > > datetime_obj < date_obj > > > > is implicitly equivalent to > > > > datetime_obj.date() < date_obj > > Using the .date() is fine when the year/month/day doesn't match. So > the following are fine:: > datetime.datetime(2005, 1, 1, 0, 0, 0) < datetime.date(2006, 1, 1) > datetime.datetime(2007, 1, 1, 0, 0, 0) > datetime.date(2006, 1, 1) > It's *not* okay to say that a date() is less than, greater than or > equal to a datetime() if the year/month/day *does* match. The correct > temporal relation is During, but Python doesn't have a During > operator. During is not the same as less-than, greater-than or > equal-to, so all of these should be False:: > datetime.datetime(2006, 1, 1, 0, 0, 0) < datetime.date(2006, 1, 1) > datetime.datetime(2006, 1, 1, 0, 0, 0) > datetime.date(2006, 1, 1) > datetime.datetime(2006, 1, 1, 0, 0, 0) == datetime.date(2006, 1, 1) > That is, the datetime() is not less than, greater than or equal to the > corresponding date(). > > Some discussion of these kinds of issues is here: > http://citeseer.ist.psu.edu/allen94actions.html > The essence is that in order to properly compare intervals, you need > the Meets, Overlaps, Starts, During and Finishes operators in addition > to the Before (<) and Simulaneous (=) operators. > > So, let's not conflate Before, After or Simultaneous with the other > relations -- if it's not strictly Before (<), After (>) or > Simultaneous (=), we can just say so by returning False. > > (I believe these semantics would be just fine for both of the examples > offered so far, but let me know if you think that's not true.) I can't say I'm well-versed in the intricacies of date/time issues, but what you say makes sense. This is exactly why I brought this patch up here : ) Thanks, Collin Winter From g.brandl at gmx.net Sat Mar 10 00:14:03 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 10 Mar 2007 00:14:03 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <435DF58A933BA74397B42CDEB8145A860A00F637@ex9.hostedexchange.local> References: <435DF58A933BA74397B42CDEB8145A860A00F637@ex9.hostedexchange.local> Message-ID: Robert Brewer schrieb: > Brett Cannon wrote: >> On 3/9/07, Collin Winter wrote: >> > On the subject of datetime enhancements, I came across an SF patch >> > (#1673403) the other day that proposed making it possible to compare >> > date and datetime objects... >> >> I personally like the current solution. The proposal to just assume >> midnight goes against EIBTI in my mind. > > Yeah, but the current solution goes against, um, APBP*. Not in my mind, > but in my code. Repeatedly. I can't count how many times I've written > code like: > > if created > fdate: > > when I "should" have written: > > if isinstance(created, datetime.date): > date_version_of_created = created > else: > date_version_of_created = created.date() > if date_version_of_created > fdate: > > But it gets better, because: > > >>> isinstance(datetime.datetime(x, y, z), datetime.date) > True > > So the above won't work, you must remember to reverse the if/else: > > if isinstance(created, datetime.datetime): > date_version_of_created = created.date() > else: > date_version_of_created = created > if date_version_of_created > fdate: Why not just give date objects a date() method that returns self? That way you can write if created.date() > fdate: Georg From pje at telecommunity.com Sat Mar 10 00:27:31 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 09 Mar 2007 18:27:31 -0500 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <20070309212033.GP32401@snowy.squish.net> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> Message-ID: <5.1.1.6.0.20070309182137.0375acd8@sparrow.telecommunity.com> At 09:20 PM 3/9/2007 +0000, Jon Ribbens wrote: >If you want the answer to be "the entire of that day" then you need >to alter the datetime module so that, e.g. subtracting 2007-03-08 >from 2007-03-09 does not return "one day" as currently, but returns >"zero days" instead (since of course there is no time between the end >of one day and the start of the next day). Unless you decide that subtraction is between days' ends. Or days' beginnings. Either suffices to produce a 1-day result in that case, and is still consistent with viewing days as slices of time. From lists at cheimes.de Sat Mar 10 00:38:26 2007 From: lists at cheimes.de (Christian Heimes) Date: Sat, 10 Mar 2007 00:38:26 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <43aa6ff70703091408w30476ea6t532c1317a29e4118@mail.gmail.com> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <43aa6ff70703091408w30476ea6t532c1317a29e4118@mail.gmail.com> Message-ID: Collin Winter schrieb: > I can't say I'm well-versed in the intricacies of date/time issues, > but what you say makes sense. This is exactly why I brought this patch > up here : ) Oh h...! Seems like I've opened a can of worms here. I only wanted to add some minor enhancements and now it looks like the improvements and the datetime module have to be carefully defined and PEPed. I'm trying to summarize the discussed issues and proposals: * Change the underlying time_t type (Unix timestamp) to use a unsigned 64bit integer even on 32 bit operating systems. * Carefully define the comparison between date and datetime objects. * int(timedelta) and float(timedelta) returns seconds * Add methods to datetime to cast from/to Julian Date Number (also MJD and RJD) * What about moving the C extension to _datetime so we can implement some non time consuming extensions in Python? * What do you think about including PyTz in the Python core? PyTz is really, REALLY useful when one has to deal with time zones. http://pytz.sourceforge.net/ * The dateutil package contains additional features which may prove as useful: http://labix.org/python-dateutil Christian From greg.ewing at canterbury.ac.nz Sat Mar 10 01:36:16 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 10 Mar 2007 13:36:16 +1300 Subject: [Python-Dev] Policy Decisions, Judgment Calls, and Backwards Compatibility (was Re: splitext('.cshrc')) In-Reply-To: <45F1A4EF.3030109@v.loewis.de> References: <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> <20070306233946.6305.746015806.divmod.xquotient.27@joule.divmod.com> <20070308045229.GQ14306@steerpike.home.puzzling.org> <20070307214117.74CE.JCARLSON@uci.edu> <45EFEB29.7080403@v.loewis.de> <20070309003135.7769.1063793143.divmod.xquotient.177@joule.divmod.com> <5.1.1.6.0.20070309111942.02ac4090@sparrow.telecommunity.com> <45F1A4EF.3030109@v.loewis.de> Message-ID: <45F1FD80.2090500@canterbury.ac.nz> Martin v. L?wis wrote: > splitext(path) > Split the pathname path into a pair (root, ext) such that root + ext == > path, and ext is empty or begins with a period and contains at most one > period. Actually, that spec could be satisfied by a function that *always* returned an empty string for ext... -- Greg From greg.ewing at canterbury.ac.nz Sat Mar 10 01:59:12 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 10 Mar 2007 13:59:12 +1300 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> Message-ID: <45F202E0.4040907@canterbury.ac.nz> Collin Winter wrote: > Treat dates as if they have a time-part of midnight. This is my preferred > solution, and it is already what the datetime module does, for example, > when subtracting two dates. Does it really? Seems to me you can pick any time of day to be the representative time and get the same result when subtracting two dates, as long as you pick the same one for both dates. So it's not really assuming anything here. -- Greg From python-dev at zesty.ca Sat Mar 10 02:26:17 2007 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Fri, 9 Mar 2007 19:26:17 -0600 (CST) Subject: [Python-Dev] datetime module enhancements In-Reply-To: <45F202E0.4040907@canterbury.ac.nz> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <45F202E0.4040907@canterbury.ac.nz> Message-ID: On Sat, 10 Mar 2007, Greg Ewing wrote: > Collin Winter wrote: > > Treat dates as if they have a time-part of midnight. This is my preferred > > solution, and it is already what the datetime module does, for example, > > when subtracting two dates. > > Does it really? Seems to me you can pick any time of day > to be the representative time and get the same result > when subtracting two dates, as long as you pick the same > one for both dates. So it's not really assuming anything > here. It certainly gives that appearance: >>> from datetime import datetime >>> datetime(2007, 1, 2) - datetime(2007, 1, 1, 0, 0, 0) datetime.timedelta(1) >>> str(_) '1 day, 0:00:00' The behaviour to the end user exposes the midnight assumption, regardless whether it's explained by saying the constructor makes the assumption or the subtraction makes the assumption. -- ?!ng From brett at python.org Sat Mar 10 03:31:26 2007 From: brett at python.org (Brett Cannon) Date: Fri, 9 Mar 2007 18:31:26 -0800 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <43aa6ff70703091408w30476ea6t532c1317a29e4118@mail.gmail.com> Message-ID: On 3/9/07, Christian Heimes wrote: > Collin Winter schrieb: > > I can't say I'm well-versed in the intricacies of date/time issues, > > but what you say makes sense. This is exactly why I brought this patch > > up here : ) > > Oh h...! Seems like I've opened a can of worms here. I only wanted to > add some minor enhancements and now it looks like the improvements and > the datetime module have to be carefully defined and PEPed. > > I'm trying to summarize the discussed issues and proposals: > > * Change the underlying time_t type (Unix timestamp) to use a unsigned > 64bit integer even on 32 bit operating systems. > > * Carefully define the comparison between date and datetime objects. > > * int(timedelta) and float(timedelta) returns seconds > > * Add methods to datetime to cast from/to Julian Date Number (also MJD > and RJD) > > * What about moving the C extension to _datetime so we can implement > some non time consuming extensions in Python? > When a use-case comes up we can make the move, but we don't need to do it pre-emptively. > * What do you think about including PyTz in the Python core? PyTz is > really, REALLY useful when one has to deal with time zones. > http://pytz.sourceforge.net/ > What is wrong with datetime's tzinfo objects? You need to show why datetime is not sufficient and why fixing it is not worth it and how it is better to bring in another chunk of code. And then we can discuss code standards, the author of PyTz stepping forward to contribute the code and maintain it, etc. > * The dateutil package contains additional features which may prove as > useful: http://labix.org/python-dateutil If someone wants to port features from dateutil to datetime, that's fine, but they just need to be worth it. -Brett From nd at perlig.de Sat Mar 10 03:40:11 2007 From: nd at perlig.de (=?iso-8859-1?q?Andr=E9_Malo?=) Date: Sat, 10 Mar 2007 03:40:11 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: Message-ID: <200703100340.12111@news.perlig.de> * Brett Cannon wrote: > On 3/9/07, Christian Heimes wrote: > > * What do you think about including PyTz in the Python core? PyTz is > > really, REALLY useful when one has to deal with time zones. > > http://pytz.sourceforge.net/ > > What is wrong with datetime's tzinfo objects? There aren't any. pytz fills that gap. But I think, pytz has no place in the stdlib anyway, because it has reasonably different release cycles (every time the timezone tables change). nd -- "Das Verhalten von Gates hatte mir bewiesen, dass ich auf ihn und seine beiden Gef?hrten nicht zu z?hlen brauchte" -- Karl May, "Winnetou III" Im Westen was neues: From brett at python.org Sat Mar 10 03:52:44 2007 From: brett at python.org (Brett Cannon) Date: Fri, 9 Mar 2007 18:52:44 -0800 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <200703100340.12111@news.perlig.de> References: <200703100340.12111@news.perlig.de> Message-ID: On 3/9/07, Andr? Malo wrote: > * Brett Cannon wrote: > > > On 3/9/07, Christian Heimes wrote: > > > * What do you think about including PyTz in the Python core? PyTz is > > > really, REALLY useful when one has to deal with time zones. > > > http://pytz.sourceforge.net/ > > > > What is wrong with datetime's tzinfo objects? > > There aren't any. pytz fills that gap. I remember some discussion about why tzinfo objects were not initially included in the stdlib. Don't remember why they were not (may just have been no one wanted to bother to write them). > > But I think, pytz has no place in the stdlib anyway, because it has > reasonably different release cycles (every time the timezone tables > change). That may have been why they were left out. -Brett From lists at cheimes.de Sat Mar 10 03:58:27 2007 From: lists at cheimes.de (Christian Heimes) Date: Sat, 10 Mar 2007 03:58:27 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <43aa6ff70703091408w30476ea6t532c1317a29e4118@mail.gmail.com> Message-ID: Brett Cannon schrieb: > What is wrong with datetime's tzinfo objects? You need to show why > datetime is not sufficient and why fixing it is not worth it and how > it is better to bring in another chunk of code. And then we can > discuss code standards, the author of PyTz stepping forward to > contribute the code and maintain it, etc. Nothing is wrong with the datetime's tzinfo *class*. PyTz is not an alternative implementation of the datetime.tzinfo *class*. It's a database containing several hundreds of datetime.tzinfo *objects* for various time zones. More precisely: The tz objects in PyTz are based on datetime.tzinfo but they are optimized for static offsets and dynamic offsets from UTC. The objects are also optimized for size (singletons) and pickling. The package was written by Stuart Bishop. He is a well known Python developer and known for high quality code. The code is well written and covered by lots of unit and doc tests. >From the README.txt pytz brings the Olson tz database into Python. This library allows accurate and cross platform timezone calculations using Python 2.3 or higher. It also solves the issue of ambiguous times at the end of daylight savings, which you can read more about in the Python Library Reference (datetime.tzinfo). Amost all (over 540) of the Olson timezones are supported. Christian From lists at cheimes.de Sat Mar 10 04:06:06 2007 From: lists at cheimes.de (Christian Heimes) Date: Sat, 10 Mar 2007 04:06:06 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <200703100340.12111@news.perlig.de> References: <200703100340.12111@news.perlig.de> Message-ID: Andr? Malo schrieb: > But I think, pytz has no place in the stdlib anyway, because it has > reasonably different release cycles (every time the timezone tables > change). The release cycles are a possible issue but what about PEP 297? http://www.python.org/dev/peps/pep-0297/ It sounds like a perfect solution for the issue. Christian From nnorwitz at gmail.com Sat Mar 10 06:47:05 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Fri, 9 Mar 2007 21:47:05 -0800 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> Message-ID: On 3/7/07, Facundo Batista wrote: > A.M. Kuchling wrote: > > > FWIW, I have a related perception that we aren't getting new core > > developers. These two problems are probably related: people don't get > > patches processed and don't become core developers, and we don't have > > enough core developers to process patches in a timely way. And so > > we're stuck. > > > > Any ideas for fixing this problem? > > I think that there's a barrier entry: there's no place to ask for help > on silly problems when you're trying to help (!). > > Let me explain my bad english wording, with an example. Yesterday night > I started modifying socket.py and test_socket.py. "Of course", I said, > "let's see if the tests pass ok *before* start changing anything". > > Went to ~/devel/reps/python/trunk/Lib, and made: > > $ python2.5 test/test_socket.py ... Wrong! > > Damn! Tried a lot of stuff... > > $ cd test > $ python2.5 test_socket.py ... Wrong! > $ python2.5 regrtest.py test_socket ... Wrong! > $ python2.5 regrtest.py test_socket.py ... Wrong! > $ python2.5 regrtest.py socket ... Wrong! > > And thousand more combinations. The best I could do is actually execute > the tests, but python was getting the installed socket module, and not > the repository socket module (even that I was in the same directory of > the latter). > > I didn't know what to try. I was stuck. This never happened to me when > working on Decimal. What went wrong in my head in the middle? > > I finally understood the problem, and build python from the repository, > and made the tests from *this* python (actually, this was an easy step > because I'm on Ubuntu, but *I* would be dead if working in Windows, for > example). > > Ok. *Me*, that I'm not ashame of asking what I don't know, if I didn't > resolve it I'd finally asked in python-dev. But how many people would > have throw the towel withoug getting so far? > > How many people want to submit a patch, or even a bug, or finds a patch > to review, but don't know how to do something and thinks that python-dev > is not the place to ask (too high minds and experienced people and > everything)? > > What I propose is a dedicated place (mailing list, for example), that is > something like a place where you can go and ask the silliest questions > about helping in the developing process. > > - How can I know if a patch is still open? > > - I found a problem, and know how to fix it, but what else need to do? > > - Found a problem in the docs, for this I must submit a patch or tell > something about it? How? > > - I found an error in the docs, and fixed it, but I'm spanish speaker > and my english sucks, can I submit a patch with bad wording or I must > ask somebody to write it ok? > > Me, for example, has an actual question to this list: "How can I know, > if I change something in the doc's .tex files, that I'm not broking > the TeX document structure?". I think this was answered in a separate thread, but I want to make sure everyone sees it here. In addition to the buildbots that run on every checkin (more or less), Misc/build.sh runs every 12 hours. This script does a full build, make install, runs the regression test suite *with refleak checking*, and builds the docs. Any errors are sent to python-checkins. The results from building the docs are here: http://docs.python.org/dev/ - for the trunk (ie, currently 2.6) http://docs.python.org/dev/2.5/ The results from running Misc/build.sh are found by adding results/ to either of the URLs above. If the doc doesn't build, the development versions of the doc are not overwritten. So to answer your question, you will know if you break a test or documentation by following python-checkins. Here is an example of when a refleak was detected: http://mail.python.org/pipermail/python-checkins/2007-February/058689.html Here's an example of when there was a doc failure: http://mail.python.org/pipermail/python-checkins/2006-February/049409.html I've tried to make it hard to screw up Python and not notice. The easiest way to do it now is to check in something without a test. If everything is checked in with a test, it will be very hard to screw something up and for it to not be noticed by some automated system. It would also be nice to run texcheck.py in build.sh to catch things like unbalanced parens. n From nnorwitz at gmail.com Sat Mar 10 07:10:48 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Fri, 9 Mar 2007 22:10:48 -0800 Subject: [Python-Dev] Encouraging developers In-Reply-To: <45ED0353.2090201@scottdial.com> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <45ED0353.2090201@scottdial.com> Message-ID: On 3/5/07, Scott Dial wrote: > > If nothing else, as an outsider there is no way to know why your patch > gets ignored while others get swiftly dealt with. Any sort of > information like this would at least provide more transparency in what > may appear to be elitest processes. Have you read the developer FAQ? http://www.python.org/dev/faq/ If we answered your questions in the FAQ, would that help? Can you come up with a list of questions that the FAQ doesn't address? If you haven't read the FAQ or didn't know it existed, where would you look for answers? Where can we make this info available? I don't believe there is anything elitist about it. I can see how one could get many opinions given that it's opaque from the outside. It's been described, but I'll re-iterate. It's completely up to the reviewers. There are only about a handful of people that I know of that review patches. I was one of those people up until about 2 months ago when I got just too much mail and have to archive most of the patches mail I get. I also look through the bugs pretty regularly to see if there are any very serious or very easy bugs to fix. When I reviewed patches, I would only look at ones I thought I could address in however much time I had and felt qualified to apply. If I don't know enough about a subject area, I typically don't look. I can't tell if a swizzle method would really be appropriate for a boombah since I didn't even know python had a boombah. Sometimes I comment on patches and don't get a response. We should probably be a lot more aggressive about closing bugs and patches without response. Unfortunately many fall into this category. Other bugs/patches go something like this: the documentation could be clearer. If I understand the current doc, there's no hope of me making it clearer. More guidance by others who might be able to provide concrete options (e.g. specific wording), can allow us to make a decision. n From g.brandl at gmx.net Sat Mar 10 08:48:01 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 10 Mar 2007 08:48:01 +0100 Subject: [Python-Dev] Import APIs Message-ID: Currently, all C code that needs to import a module uses PyImport_ImportModule which (1) calls __builtin__.__import__ (2) attempts relative imports Most of the time, though, at least (2) is wrong. If we want to keep (1), PyImport_ImportModuleLevel can't be used as a replacement. So there should be a new API, called PyImport_ImportAbsolute that gets a flag whether relative import should be allowed. Georg From ncoghlan at gmail.com Sat Mar 10 11:29:58 2007 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 10 Mar 2007 20:29:58 +1000 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <43aa6ff70703091408w30476ea6t532c1317a29e4118@mail.gmail.com> Message-ID: <45F288A6.1040904@gmail.com> Christian Heimes wrote: >>From the README.txt > pytz brings the Olson tz database into Python. This library allows > accurate and cross platform timezone calculations using Python 2.3 > or higher. It also solves the issue of ambiguous times at the end > of daylight savings, which you can read more about in the Python > Library Reference (datetime.tzinfo). Amost all (over 540) of the Olson > timezones are supported. This sounds like it may at least be worth a 'See also' in the datetime docs... Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From jon+python-dev at unequivocal.co.uk Sat Mar 10 12:36:16 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Sat, 10 Mar 2007 11:36:16 +0000 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <5.1.1.6.0.20070309182137.0375acd8@sparrow.telecommunity.com> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <5.1.1.6.0.20070309182137.0375acd8@sparrow.telecommunity.com> Message-ID: <20070310113616.GS32401@snowy.squish.net> "Phillip J. Eby" wrote: > At 09:20 PM 3/9/2007 +0000, Jon Ribbens wrote: > >If you want the answer to be "the entire of that day" then you need > >to alter the datetime module so that, e.g. subtracting 2007-03-08 > >from 2007-03-09 does not return "one day" as currently, but returns > >"zero days" instead (since of course there is no time between the end > >of one day and the start of the next day). > > Unless you decide that subtraction is between days' ends. Or days' > beginnings. Either suffices to produce a 1-day result in that case, and is > still consistent with viewing days as slices of time. So you're deciding that a 'date' is 'the entire of that day', except when you subtract two of them, when it suddenly means something else? ;-) From sjoerd at acm.org Sat Mar 10 13:04:45 2007 From: sjoerd at acm.org (Sjoerd Mullender) Date: Sat, 10 Mar 2007 13:04:45 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <45F1BBD0.2000707@v.loewis.de> References: <17905.35797.156230.974559@montanaro.dyndns.org> <740c3aec0703090924g547dbaddt755d03937bb423e0@mail.gmail.com> <45F1BBD0.2000707@v.loewis.de> Message-ID: <45F29EDD.1090206@acm.org> On 03/09/2007 08:56 PM, Martin v. L?wis wrote: -- Sjoerd Mullender > Christian Heimes schrieb: >> BJ?rn Lindqvist schrieb: >>> I think it should be a ValueError, given that the programmer is very >>> likely to further use the returned timestamp to for example insert >>> stuff in a database. Unix timestamps are not unambiguously defined for >>> any years other than 1970 to 2038 imho. >> IIRC the unix timestamp was originally definied as *signed* int with 32bit. > > "Unix" time stamps where never defined as being 32bit. That is just an > implementation detail. time_t was defined as an int, in second since > 1970, and that is all what was defined. Whether or not an int is 32bits > depends on the hardware (and the compiler); this was never part of Unix. As I remember, the time() system call on Version 7 Unix on the (16 bit) PDP 11 returned the time as a C long, which on that machine was 32 bits. I just looked at the Version 6 Unix listing, and there too, the time is returned as a 32 bit quantity. I also seem to remember that in the early days, there was no such thing as an unsigned long, so the value was necessarily signed. But I may be misremembering this bit. -- Sjoerd Mullender From jon+python-dev at unequivocal.co.uk Sat Mar 10 13:05:38 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Sat, 10 Mar 2007 12:05:38 +0000 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> Message-ID: <20070310120538.GT32401@snowy.squish.net> Steven Bethard wrote: > Using the .date() is fine when the year/month/day doesn't match. So > the following are fine:: > datetime.datetime(2005, 1, 1, 0, 0, 0) < datetime.date(2006, 1, 1) > datetime.datetime(2007, 1, 1, 0, 0, 0) > datetime.date(2006, 1, 1) > It's *not* okay to say that a date() is less than, greater than or > equal to a datetime() if the year/month/day *does* match. Why not? That only makes sense if you decide that a Python 'date' means 'the entire of that day'. It's not at all clear that that's what a Python 'date' *does* mean. And, as I mentioned before, if you do decide that then what Python currently does when you subtract dates is broken. I just found another case where 'date's pretend to have a time-part of midnight - if you add a date to a timedelta. Add 23 hours to a date and it's unchanged - add 24 and it moves forward a day. If a 'date' really is 'the entire of that day', then adding a timedelta to it which is not an integer multiple of 1 day should either raise an exception, or return some 'time duration' object that doesn't currently exist. > The correct temporal relation is During, but Python doesn't have a > During operator. During is not the same as less-than, greater-than > or equal-to, so all of these should be False:: I think you're assuming the Python datetime module has higher goals than it does. It doesn't have any concept of a duration in time, not unless you couple a 'datetime' with a 'timedelta' yourself. And 'timedelta's are restricted to periods of time that are a fixed number of seconds - i.e. there is no way to indicate "a month" or "a year" (although that lack is a whole different argument ;-) ) Your argument is quite correct if you're considering some fancy uber-complicated kitchen-sink-included all-encompassing "temporal logic package", but that's not what Python's datetime is, nor frankly is what it should be. From fuzzyman at voidspace.org.uk Sat Mar 10 13:19:22 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 10 Mar 2007 12:19:22 +0000 Subject: [Python-Dev] Non implementation dependent access to calling scope Message-ID: <45F2A24A.6000701@voidspace.org.uk> Hello all, I realise that this may be more relevant to Python ideas, in which case feel free to ignore this (and my apologies). I occasionally see code looking (something) like : calling_scope = sys._getframe(1).f_globals['__name__'] This looks and smells like a hack (not least because of the warning in the documentation about _getframe), plus stack frames are an implementation detail so this code is broken on IronPython. This makes me sad. It would be great to have a specified way to obtain [a read only view on (?)] the locals and globals from the calling scope. Perhaps built in functions ? If they were specified then the IronPython guys would have to implement it for us. B-) I realise that this can allow bad some bad programming patterns, but there are times when it can be very useful. All the best, Michael Foord From lists at cheimes.de Sat Mar 10 13:37:57 2007 From: lists at cheimes.de (Christian Heimes) Date: Sat, 10 Mar 2007 13:37:57 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <20070310113616.GS32401@snowy.squish.net> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <5.1.1.6.0.20070309182137.0375acd8@sparrow.telecommunity.com> <20070310113616.GS32401@snowy.squish.net> Message-ID: Jon Ribbens schrieb: > So you're deciding that a 'date' is 'the entire of that day', except > when you subtract two of them, when it suddenly means something else? ;-) It makes kinda sense although it looks like a contradiction at first. The common linguistic usage of dates in English and German: "On Saturday" or "On 2007-03-07" is a undefined time anytime on Saturday or the whole Saturday but Saturday and Sunday are a day apart. It feels right to me. Christian From jon+python-dev at unequivocal.co.uk Sat Mar 10 13:44:09 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Sat, 10 Mar 2007 12:44:09 +0000 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <5.1.1.6.0.20070309182137.0375acd8@sparrow.telecommunity.com> <20070310113616.GS32401@snowy.squish.net> Message-ID: <20070310124409.GW32401@snowy.squish.net> Christian Heimes wrote: > Jon Ribbens schrieb: > > So you're deciding that a 'date' is 'the entire of that day', except > > when you subtract two of them, when it suddenly means something else? ;-) > > It makes kinda sense although it looks like a contradiction at first. > > The common linguistic usage of dates in English and German: > "On Saturday" or "On 2007-03-07" is a undefined time anytime on Saturday > or the whole Saturday but Saturday and Sunday are a day apart. It feels > right to me. That doesn't help much. English is often very vague about times and dates, but computer programs must be have a defined answer. What do you feel "next Tuesday plus 12 hours" means? ;-) From martin at v.loewis.de Sat Mar 10 13:57:42 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 10 Mar 2007 13:57:42 +0100 Subject: [Python-Dev] Non implementation dependent access to calling scope In-Reply-To: <45F2A24A.6000701@voidspace.org.uk> References: <45F2A24A.6000701@voidspace.org.uk> Message-ID: <45F2AB46.4050609@v.loewis.de> Michael Foord schrieb: > Hello all, > > I realise that this may be more relevant to Python ideas, in which case > feel free to ignore this (and my apologies). > > I occasionally see code looking (something) like : > > > calling_scope = sys._getframe(1).f_globals['__name__'] > > This looks and smells like a hack (not least because of the warning in > the documentation about _getframe), Right. This should some helper functions in inspect (which currently aren't there). > plus stack frames are an > implementation detail so this code is broken on IronPython. I don't think there is much you can do about this. Local variables are also an implementation detail. > It would be great to have a specified way to obtain [a read only view on > (?)] the locals and globals from the calling scope. Perhaps built in > functions ? Please, no builtin functions. > If they were specified then the IronPython guys would have > to implement it for us. B-) I doubt it. If users of IronPython would want that badly enough, it would be there. sys._getframe has been in CPython for a while, yet it (apparently) is unavailable in IronPython. I would expect that implementing what you propose would similar in effort to implementing sys._getframe. > I realise that this can allow bad some bad programming patterns, but > there are times when it can be very useful. Users who want it can already do it, in CPython. The example you gave isn't even about locals, but the caller's *globals*. I would guess that there is already a way in IronPython to get at these, given that .NET also needs to support stack walks in various places. Regards, Martin From amk at amk.ca Sat Mar 10 13:58:46 2007 From: amk at amk.ca (A.M. Kuchling) Date: Sat, 10 Mar 2007 07:58:46 -0500 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <45ED0353.2090201@scottdial.com> Message-ID: <20070310125846.GA21867@andrew-kuchlings-computer.local> On Fri, Mar 09, 2007 at 10:10:48PM -0800, Neal Norwitz wrote: > We should probably be a lot more aggressive about closing bugs and > patches without response. Unfortunately many fall into this category. This question comes up every so often, and after much discussion I think python-dev always converges on leaving bugs open in case some future person finds the information useful. I hope that the Roundup tracker will let us tag such bugs with a 'waiting-for-reporter' tag, and we can then exclude such bugs when looking for ones to fix. --amk From lists at cheimes.de Sat Mar 10 14:00:08 2007 From: lists at cheimes.de (Christian Heimes) Date: Sat, 10 Mar 2007 14:00:08 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <20070310124409.GW32401@snowy.squish.net> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <5.1.1.6.0.20070309182137.0375acd8@sparrow.telecommunity.com> <20070310113616.GS32401@snowy.squish.net> <20070310124409.GW32401@snowy.squish.net> Message-ID: Jon Ribbens schrieb: > What do you feel "next Tuesday plus 12 hours" means? ;-) First thought: It's nonsense! Nobody would say that. ;) Second though: Tuesday noon (12h after the beginning of Tuesday) Christian From fuzzyman at voidspace.org.uk Sat Mar 10 14:22:49 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 10 Mar 2007 13:22:49 +0000 Subject: [Python-Dev] Non implementation dependent access to calling scope In-Reply-To: <45F2AB46.4050609@v.loewis.de> References: <45F2A24A.6000701@voidspace.org.uk> <45F2AB46.4050609@v.loewis.de> Message-ID: <45F2B129.9050204@voidspace.org.uk> Martin v. L?wis wrote: > Michael Foord schrieb: >> Hello all, >> >> I realise that this may be more relevant to Python ideas, in which >> case feel free to ignore this (and my apologies). >> >> I occasionally see code looking (something) like : >> >> >> calling_scope = sys._getframe(1).f_globals['__name__'] >> >> This looks and smells like a hack (not least because of the warning >> in the documentation about _getframe), > > Right. This should some helper functions in inspect (which currently > aren't there). That would be cool. > >> plus stack frames are an implementation detail so this code is broken >> on IronPython. > > I don't think there is much you can do about this. Local variables are > also an implementation detail. > >> It would be great to have a specified way to obtain [a read only view >> on (?)] the locals and globals from the calling scope. Perhaps built >> in functions ? > > Please, no builtin functions. :-) > >> If they were specified then the IronPython guys would have to >> implement it for us. B-) > > I doubt it. If users of IronPython would want that badly enough, it > would be there. sys._getframe has been in CPython for a while, yet > it (apparently) is unavailable in IronPython. I would expect that > implementing what you propose would similar in effort to implementing > sys._getframe. > The difference is that IronPython uses .NET stack frames. Because it does a lot of dynamic code generation there are often several .NET stack frames in between what would each Python stack frame. There may be a way to get at the globals for the calling scope in IronPython, I'll ask on the list. Thanks for your reply. Michael >> I realise that this can allow bad some bad programming patterns, but >> there are times when it can be very useful. > > Users who want it can already do it, in CPython. The example you gave > isn't even about locals, but the caller's *globals*. I would guess > that there is already a way in IronPython to get at these, given that > .NET also needs to support stack walks in various places. > > Regards, > Martin > From jon+python-dev at unequivocal.co.uk Sat Mar 10 14:29:09 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Sat, 10 Mar 2007 13:29:09 +0000 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <5.1.1.6.0.20070309182137.0375acd8@sparrow.telecommunity.com> <20070310113616.GS32401@snowy.squish.net> <20070310124409.GW32401@snowy.squish.net> Message-ID: <20070310132909.GY32401@snowy.squish.net> Christian Heimes wrote: > > What do you feel "next Tuesday plus 12 hours" means? ;-) > > First thought: It's nonsense! Nobody would say that. ;) > > Second though: Tuesday noon (12h after the beginning of Tuesday) I agree with you entirely. Your suggestions correspond to 'throw an exception' and 'return a datetime'. Note that in the second case you are agreeing with me that a date means midnight at the start of that date ;-) Python datetime disagrees with us both. It basically says 'add the timedelta to midnight at the start of the date, then return just the date part of the result'. I assume there is some reason for this surprising behaviour. But this is another example of where datetime is taking a pragmatic approach in an ambiguous situation - exactly as I am suggesting in the situation of comparing a date and a datetime. From collinw at gmail.com Sat Mar 10 15:50:50 2007 From: collinw at gmail.com (Collin Winter) Date: Sat, 10 Mar 2007 08:50:50 -0600 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <43aa6ff70703091408w30476ea6t532c1317a29e4118@mail.gmail.com> Message-ID: <43aa6ff70703100650i63c0b471pac3e36e951ea21b3@mail.gmail.com> On 3/9/07, Christian Heimes wrote: > Collin Winter schrieb: > > I can't say I'm well-versed in the intricacies of date/time issues, > > but what you say makes sense. This is exactly why I brought this patch > > up here : ) > > Oh h...! Seems like I've opened a can of worms here. I only wanted to > add some minor enhancements and now it looks like the improvements and > the datetime module have to be carefully defined and PEPed. > > I'm trying to summarize the discussed issues and proposals: > > * Change the underlying time_t type (Unix timestamp) to use a unsigned > 64bit integer even on 32 bit operating systems. > > * Carefully define the comparison between date and datetime objects. > > * int(timedelta) and float(timedelta) returns seconds > > * Add methods to datetime to cast from/to Julian Date Number (also MJD > and RJD) > > * What about moving the C extension to _datetime so we can implement > some non time consuming extensions in Python? > > * What do you think about including PyTz in the Python core? PyTz is > really, REALLY useful when one has to deal with time zones. > http://pytz.sourceforge.net/ > > * The dateutil package contains additional features which may prove as > useful: http://labix.org/python-dateutil Here's another one for the list, trawled off SF: patch #1118748 (http://python.org/sf/1118748). It proposes to add the ability to add/subtract time and timedelta objects. Collin Winter From collinw at gmail.com Sat Mar 10 16:13:28 2007 From: collinw at gmail.com (Collin Winter) Date: Sat, 10 Mar 2007 09:13:28 -0600 Subject: [Python-Dev] unittest enhancement for TestCase classes hierarchies Message-ID: <43aa6ff70703100713m786bd124h4677850f4730cb15@mail.gmail.com> In my continuing trawl through the SF patch tracker, I came across #1244929 (http://python.org/sf/1244929), which causes TestLoader.loadTestsFromModule() to skip classes whose name starts with an underscore. This addresses the warning in that method's docs: """ While using a hierarchy of TestCase-derived classes can be convenient in sharing fixtures and helper functions, defining test methods on base classes that are not intended to be instantiated directly does not play well with this method. Doing so, however, can be useful when the fixtures are different and defined in subclasses. """ "Does not play well", in this case, means that your base classes will be picked up against your will if they subclass TestCase. I like the patch and have worked up tests and doc changes for it. Any objections to including this in 2.6? Collin Winter From exarkun at divmod.com Sat Mar 10 16:49:45 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Sat, 10 Mar 2007 10:49:45 -0500 Subject: [Python-Dev] unittest enhancement for TestCase classes hierarchies In-Reply-To: <43aa6ff70703100713m786bd124h4677850f4730cb15@mail.gmail.com> Message-ID: <20070310154945.18920.1748181708.divmod.quotient.398@ohm> On Sat, 10 Mar 2007 09:13:28 -0600, Collin Winter wrote: >In my continuing trawl through the SF patch tracker, I came across >#1244929 (http://python.org/sf/1244929), which causes >TestLoader.loadTestsFromModule() to skip classes whose name starts >with an underscore. This addresses the warning in that method's docs: > >""" >While using a hierarchy of >TestCase-derived classes can be convenient in sharing >fixtures and helper functions, defining test methods on base classes >that are not intended to be instantiated directly does not play well >with this method. Doing so, however, can be useful when the >fixtures are different and defined in subclasses. >""" > >"Does not play well", in this case, means that your base classes will >be picked up against your will if they subclass TestCase. > >I like the patch and have worked up tests and doc changes for it. Any >objections to including this in 2.6? This use case is what mixins are for. You don't have to include TestCase in your ancestry until you get to a class which you actually want to run tests. The current rule of loading anything that subclasses TestCase is simple and straightforward. Complicating it to provide a feature which is already available through a widely used standard Python idiom doesn't seem worth while. Jean-Paul From fumanchu at amor.org Sat Mar 10 18:04:34 2007 From: fumanchu at amor.org (Robert Brewer) Date: Sat, 10 Mar 2007 09:04:34 -0800 Subject: [Python-Dev] datetime module enhancements References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> Message-ID: <435DF58A933BA74397B42CDEB8145A86224D5A@ex9.hostedexchange.local> On 3/9/07, Collin Winter wrote: > > On the subject of datetime enhancements, I came > > across an SF patch (#1673403) the other day that > > proposed making it possible to compare date and > > datetime objects. > > One solution that just occurred to me -- and that > skirts the issue of choosing an interpretation -- > is that, when comparing date and datetime objects, > the datetime's .date() method is called and the > result of that call is compared to the original > date. That is, > > datetime_obj < date_obj > > is implicitly equivalent to > > datetime_obj.date() < date_obj > > Seems a ready-made use case for that method. +1 ...and it's a decision that can be made independently of how to add or subtract dates. Robert Brewer System Architect Amor Ministries fumanchu at amor.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070310/ad8fcef4/attachment.htm From jon+python-dev at unequivocal.co.uk Sat Mar 10 18:07:33 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Sat, 10 Mar 2007 17:07:33 +0000 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <435DF58A933BA74397B42CDEB8145A86224D5A@ex9.hostedexchange.local> References: <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <435DF58A933BA74397B42CDEB8145A86224D5A@ex9.hostedexchange.local> Message-ID: <20070310170733.GD32401@snowy.squish.net> Robert Brewer wrote: > > One solution that just occurred to me -- and that > > skirts the issue of choosing an interpretation -- > > is that, when comparing date and datetime objects, > > the datetime's .date() method is called and the > > result of that call is compared to the original > > date. > > +1 > > ...and it's a decision that can be made independently > of how to add or subtract dates. I don't like it. It sounds suspiciously like "when comparing integers and floats, discard the non-integer portion of the float and then compare as integers". From guido at python.org Sat Mar 10 18:32:49 2007 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Mar 2007 09:32:49 -0800 Subject: [Python-Dev] Non implementation dependent access to calling scope In-Reply-To: <45F2A24A.6000701@voidspace.org.uk> References: <45F2A24A.6000701@voidspace.org.uk> Message-ID: Looks like you misunderstand what's going on. sys._getframe() *intentionally* smells like a hack, becase we don't *want* you to feel comfortable using it. Its mere existence may constrain other Python implementations from optimizing the code they generate; it is a compromise to enable those projects that don't care about this to occasionally engage in implementation-specific but useful behavior. On 3/10/07, Michael Foord wrote: > Hello all, > > I realise that this may be more relevant to Python ideas, in which case > feel free to ignore this (and my apologies). > > I occasionally see code looking (something) like : > > > calling_scope = sys._getframe(1).f_globals['__name__'] > > This looks and smells like a hack (not least because of the warning in > the documentation about _getframe), plus stack frames are an > implementation detail so this code is broken on IronPython. This makes > me sad. > > It would be great to have a specified way to obtain [a read only view on > (?)] the locals and globals from the calling scope. Perhaps built in > functions ? If they were specified then the IronPython guys would have > to implement it for us. B-) > > I realise that this can allow bad some bad programming patterns, but > there are times when it can be very useful. > > All the best, > > Michael Foord > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From steven.bethard at gmail.com Sat Mar 10 18:53:35 2007 From: steven.bethard at gmail.com (Steven Bethard) Date: Sat, 10 Mar 2007 10:53:35 -0700 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <20070310120538.GT32401@snowy.squish.net> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <20070310120538.GT32401@snowy.squish.net> Message-ID: On 3/10/07, Jon Ribbens wrote: > Steven Bethard wrote: > > Using the .date() is fine when the year/month/day doesn't match. So > > the following are fine:: > > datetime.datetime(2005, 1, 1, 0, 0, 0) < datetime.date(2006, 1, 1) > > datetime.datetime(2007, 1, 1, 0, 0, 0) > datetime.date(2006, 1, 1) > > It's *not* okay to say that a date() is less than, greater than or > > equal to a datetime() if the year/month/day *does* match. > > Why not? That only makes sense if you decide that a Python 'date' > means 'the entire of that day'. It's not at all clear that that's > what a Python 'date' *does* mean. And, as I mentioned before, if > you do decide that then what Python currently does when you > subtract dates is broken. No, as Phillip J. Eby pointed out, there's a variety of ways to define subtraction over intervals: (1) Distance between start1 and start2 (2) Distance between start1 and end2 (3) Distance between end1 and start2 (4) Distance between end1 and end2 Options (1) and (4) are consistent with the current implementation. Options (2) and (3) are not. I don't see why the docs can't be explicit about what subtraction means given that there are a number of possible interpretations. > I just found another case where 'date's pretend to have a time-part > of midnight - if you add a date to a timedelta. Add 23 hours to a > date and it's unchanged - add 24 and it moves forward a day. If a > 'date' really is 'the entire of that day', then adding a timedelta > to it which is not an integer multiple of 1 day should either raise > an exception, or return some 'time duration' object that doesn't > currently exist. You can draw the analogy here with integer addition:: >>> int(1) + int(.75) 1 >>> int(1) + int(1.25) 2 Though if I had written the datetime module, this would have raised an exception, rather than passing silently. But I suspect that almost no one is adding *hours* to *dates* so I don't particularly care how it behaves, nor do I think that some idiosyncratic behavior in a part of the module no one uses should dictate the behavior in the rest of the module. > Your argument is quite correct if you're considering some fancy > uber-complicated kitchen-sink-included all-encompassing "temporal > logic package", but that's not what Python's datetime is, nor > frankly is what it should be. I'm not expecting Python's datetime module to handle any complex temporal logic. But boy it would be nice if it didn't *break* basic temporal logic, and that's certainly possible with a very small bit of effort. However, assuming that a date() is a datetime() with a time of midnight will clearly break that logic. I still don't see why my more careful comparison would be bad for any of your code. Could you give an example where it would be bad for all the following to be False:: date(2006, 1, 1) < datetime(2006, 1, 1, 0, 0, 0) date(2006, 1, 1) > datetime(2006, 1, 1, 0, 0, 0) date(2006, 1, 1) == datetime(2006, 1, 1, 0, 0, 0) even though the following would be True:: date(2006, 1, 1) < datetime(2006, 1, 2, 0, 0, 0) date(2006, 1, 1) > datetime(2005, 12, 31, 0, 0, 0) ? Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From jon+python-dev at unequivocal.co.uk Sat Mar 10 19:10:57 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Sat, 10 Mar 2007 18:10:57 +0000 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <20070310120538.GT32401@snowy.squish.net> Message-ID: <20070310181057.GF32401@snowy.squish.net> Steven Bethard wrote: > I don't see why the docs can't be explicit about what subtraction > means given that there are a number of possible interpretations. I don't see why the docs can't be explicit about what comparison means given that there are a number of possible interpretations. > You can draw the analogy here with integer addition:: > > >>> int(1) + int(.75) > 1 > >>> int(1) + int(1.25) > 2 You're making my point for me. An integer (in the abstract sense) *is* the same as a float with a zero fractional part. You're agreeing that datetime treats dates as datetimes with a midnight time part. > However, assuming that a date() is a datetime() with a time of > midnight will clearly break that logic. Could you please explain how? It doesn't look to me like it breaks anything. I think you're assuming that "date" means "the whole of that day" - if so then why do you think that assumption is valid? Using your suggested analogy above, if you consider "date"s to be "ints" and "datetime"s to be "floats", then it seems to be that everything makes sense and is logical and coherent. > I still don't see why my more careful comparison would be bad for any > of your code. Could you give an example where it would be bad for all > the following to be False:: > date(2006, 1, 1) < datetime(2006, 1, 1, 0, 0, 0) > date(2006, 1, 1) > datetime(2006, 1, 1, 0, 0, 0) > date(2006, 1, 1) == datetime(2006, 1, 1, 0, 0, 0) You need me to explain why "a < b", "a > b" and "a == b" all being false simultaneously is bad!? What would "a != b" return? What would "cmp(a, b)" return? > even though the following would be True:: > date(2006, 1, 1) < datetime(2006, 1, 2, 0, 0, 0) > date(2006, 1, 1) > datetime(2005, 12, 31, 0, 0, 0) Actually, your suggestion would be much better than the current behaviour, for my uses, even though it's totally illogical. From nnorwitz at gmail.com Sat Mar 10 19:10:58 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sat, 10 Mar 2007 10:10:58 -0800 Subject: [Python-Dev] Encouraging developers In-Reply-To: <20070310125846.GA21867@andrew-kuchlings-computer.local> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <45ED0353.2090201@scottdial.com> <20070310125846.GA21867@andrew-kuchlings-computer.local> Message-ID: On 3/10/07, A.M. Kuchling wrote: > On Fri, Mar 09, 2007 at 10:10:48PM -0800, Neal Norwitz wrote: > > We should probably be a lot more aggressive about closing bugs and > > patches without response. Unfortunately many fall into this category. > > This question comes up every so often, and after much discussion I > think python-dev always converges on leaving bugs open in case some > future person finds the information useful. I was thinking about patches only in this case, not bugs. > I hope that the Roundup tracker will let us tag such bugs with a > 'waiting-for-reporter' tag, and we can then exclude such bugs when > looking for ones to fix. Georg added a comment/feature request for this. Actually, he requested a close on inactivity after some duration if we set a keyword or something like that. Since we are in control of the tracker, we can make it happen if someone cares enough. n From paul.hankin at pobox.com Sat Mar 10 19:47:51 2007 From: paul.hankin at pobox.com (Paul Hankin) Date: Sat, 10 Mar 2007 18:47:51 +0000 Subject: [Python-Dev] fmod.c Message-ID: Hi all, Is there any need for Python/fmod.c any more? I can't see how it can be included because there's no test for fmod in the ./configure script and grepping all files in the tree for fmod.c finds nothing except a commented out line in PC/os2vacpp/makefile.omk If it is needed, it needs fixing as it gives the wrong answer for x>=0 and y<0. Eg: fmod(0, -1) returns 1 rather than 0. (The comment in fmod.c is correct: it should return f such that x = i*y + f for some integer i, st |f| < |y| and f has the same sign as x). I'm happy to patch it, but I won't if it would be better just to remove the file. -- Paul Hankin From collinw at gmail.com Sat Mar 10 20:42:08 2007 From: collinw at gmail.com (Collin Winter) Date: Sat, 10 Mar 2007 13:42:08 -0600 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <435DF58A933BA74397B42CDEB8145A86224D5A@ex9.hostedexchange.local> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <435DF58A933BA74397B42CDEB8145A86224D5A@ex9.hostedexchange.local> Message-ID: <43aa6ff70703101142k1b41b93aq108200992d96a99c@mail.gmail.com> On 3/10/07, Robert Brewer wrote: > On 3/9/07, Collin Winter wrote: > > > On the subject of datetime enhancements, I came > > > across an SF patch (#1673403) the other day that > > > proposed making it possible to compare date and > > > datetime objects. > > > > One solution that just occurred to me -- and that > > skirts the issue of choosing an interpretation -- > > is that, when comparing date and datetime objects, > > the datetime's .date() method is called and the > > result of that call is compared to the original > > date. That is, > > > > datetime_obj < date_obj > > > > is implicitly equivalent to > > > > datetime_obj.date() < date_obj > > > > Seems a ready-made use case for that method. > > +1 > > ...and it's a decision that can be made independently > of how to add or subtract dates. That's what I thought, until Steve Bethard set me straight: http://mail.python.org/pipermail/python-dev/2007-March/071803.html I'm going to defer to him on this. Collin Winter From steven.bethard at gmail.com Sat Mar 10 21:01:40 2007 From: steven.bethard at gmail.com (Steven Bethard) Date: Sat, 10 Mar 2007 13:01:40 -0700 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <20070310181057.GF32401@snowy.squish.net> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <20070310120538.GT32401@snowy.squish.net> <20070310181057.GF32401@snowy.squish.net> Message-ID: On 3/10/07, Jon Ribbens wrote: > Steven Bethard wrote: > > I still don't see why my more careful comparison would be bad for any > > of your code. Could you give an example where it would be bad for all > > the following to be False:: > > date(2006, 1, 1) < datetime(2006, 1, 1, 0, 0, 0) > > date(2006, 1, 1) > datetime(2006, 1, 1, 0, 0, 0) > > date(2006, 1, 1) == datetime(2006, 1, 1, 0, 0, 0) > > You need me to explain why "a < b", "a > b" and "a == b" all being > false simultaneously is bad!? Yes. One of the main reasons for having having rich comparisons (__lt__, __eq__, etc.) is to be able to handle partial orderings like the ones we're talking about here. > What would "a != b" return? True. The intervals are not Simultaneous (equal). > What would "cmp(a, b)" return? According to the docs [1], "The return value is negative if x < y, zero if x == y and strictly positive if x > y." Which means that cmp() is undefined in this case (and should probably raise an exception). However, the current behavior is to return 1:: >>> class C(object): ... def __lt__(self, other): ... return False ... def __gt__(self, other): ... return False ... def __eq__(self, other): ... return False ... >>> cmp(C(), C()) 1 Since I don't know too many folks that use cmp() instead of using <, > or = directly, returning 1 is fine with me. Anyone who has a problem with that should take it up with the cmp() implementation. [1] http://docs.python.org/lib/built-in-funcs.html#l2h-17 > > even though the following would be True:: > > date(2006, 1, 1) < datetime(2006, 1, 2, 0, 0, 0) > > date(2006, 1, 1) > datetime(2005, 12, 31, 0, 0, 0) > > Actually, your suggestion would be much better than the current > behaviour, for my uses Good. Well at least we're agreed on that part. Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From collinw at gmail.com Sat Mar 10 21:44:47 2007 From: collinw at gmail.com (Collin Winter) Date: Sat, 10 Mar 2007 14:44:47 -0600 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <43aa6ff70703100650i63c0b471pac3e36e951ea21b3@mail.gmail.com> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <43aa6ff70703091408w30476ea6t532c1317a29e4118@mail.gmail.com> <43aa6ff70703100650i63c0b471pac3e36e951ea21b3@mail.gmail.com> Message-ID: <43aa6ff70703101244x1dcc53f4n8872e90375a37325@mail.gmail.com> On 3/10/07, Collin Winter wrote: > On 3/9/07, Christian Heimes wrote: > > Collin Winter schrieb: > > > I can't say I'm well-versed in the intricacies of date/time issues, > > > but what you say makes sense. This is exactly why I brought this patch > > > up here : ) > > > > Oh h...! Seems like I've opened a can of worms here. I only wanted to > > add some minor enhancements and now it looks like the improvements and > > the datetime module have to be carefully defined and PEPed. > > > > I'm trying to summarize the discussed issues and proposals: > > > > * Change the underlying time_t type (Unix timestamp) to use a unsigned > > 64bit integer even on 32 bit operating systems. > > > > * Carefully define the comparison between date and datetime objects. > > > > * int(timedelta) and float(timedelta) returns seconds > > > > * Add methods to datetime to cast from/to Julian Date Number (also MJD > > and RJD) > > > > * What about moving the C extension to _datetime so we can implement > > some non time consuming extensions in Python? > > > > * What do you think about including PyTz in the Python core? PyTz is > > really, REALLY useful when one has to deal with time zones. > > http://pytz.sourceforge.net/ > > > > * The dateutil package contains additional features which may prove as > > useful: http://labix.org/python-dateutil > > Here's another one for the list, trawled off SF: patch #1118748 > (http://python.org/sf/1118748). It proposes to add the ability to > add/subtract time and timedelta objects. And one last patch: #1100942 (http://python.org/sf/1100942), which would add a strptime() constructor for the date and time classes. Collin Winter From collinw at gmail.com Sat Mar 10 21:54:58 2007 From: collinw at gmail.com (Collin Winter) Date: Sat, 10 Mar 2007 14:54:58 -0600 Subject: [Python-Dev] unittest enhancement for TestCase classes hierarchies In-Reply-To: <20070310154945.18920.1748181708.divmod.quotient.398@ohm> References: <43aa6ff70703100713m786bd124h4677850f4730cb15@mail.gmail.com> <20070310154945.18920.1748181708.divmod.quotient.398@ohm> Message-ID: <43aa6ff70703101254g9f273a0t757aaeb4914110dc@mail.gmail.com> On 3/10/07, Jean-Paul Calderone wrote: > On Sat, 10 Mar 2007 09:13:28 -0600, Collin Winter wrote: > >In my continuing trawl through the SF patch tracker, I came across > >#1244929 (http://python.org/sf/1244929), which causes > >TestLoader.loadTestsFromModule() to skip classes whose name starts > >with an underscore. This addresses the warning in that method's docs: > > > >""" > >While using a hierarchy of > >TestCase-derived classes can be convenient in sharing > >fixtures and helper functions, defining test methods on base classes > >that are not intended to be instantiated directly does not play well > >with this method. Doing so, however, can be useful when the > >fixtures are different and defined in subclasses. > >""" > > > >"Does not play well", in this case, means that your base classes will > >be picked up against your will if they subclass TestCase. > > > >I like the patch and have worked up tests and doc changes for it. Any > >objections to including this in 2.6? > > This use case is what mixins are for. You don't have to include TestCase > in your ancestry until you get to a class which you actually want to run > tests. > > The current rule of loading anything that subclasses TestCase is simple > and straightforward. Complicating it to provide a feature which is already > available through a widely used standard Python idiom doesn't seem worth > while. Understood, but I don't like having to keep repeating, "yes, this subclass of X is also a TestCase; so is this one, and this one, and this one, etc". Python already ignores _-prefixed names in certain analogous, load-everything-from-a-module situations ("from x import *"), so I don't see this as being particularly unpythonic. Nor do I see it as all that complicated. Striking the current equivocating, paragraph-long warning from loadTestsFromModule()'s docs and replacing it with "Classes whose name starts with an underscore will be ignored" seems like a win to me. Collin Winter From greg.ewing at canterbury.ac.nz Sat Mar 10 23:53:18 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 11 Mar 2007 11:53:18 +1300 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <20070310113616.GS32401@snowy.squish.net> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <5.1.1.6.0.20070309182137.0375acd8@sparrow.telecommunity.com> <20070310113616.GS32401@snowy.squish.net> Message-ID: <45F336DE.9080700@canterbury.ac.nz> Jon Ribbens wrote: > So you're deciding that a 'date' is 'the entire of that day', except > when you subtract two of them, when it suddenly means something else? ;-) No, you're considering dates to be discrete entities, like integers. You wouldn't use the same reasoning to argue that the difference between two consecutive integers should be 0... would you? -- Greg From mithrandi-python-dev at mithrandi.za.net Sun Mar 11 01:18:36 2007 From: mithrandi-python-dev at mithrandi.za.net (Tristan Seligmann) Date: Sun, 11 Mar 2007 02:18:36 +0200 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <43aa6ff70703091408w30476ea6t532c1317a29e4118@mail.gmail.com> Message-ID: <20070311001836.GA2690@mithrandi.za.net> * Christian Heimes [2007-03-10 03:58:27 +0100]: > >From the README.txt > pytz brings the Olson tz database into Python. This library allows > accurate and cross platform timezone calculations using Python 2.3 > or higher. It also solves the issue of ambiguous times at the end > of daylight savings, which you can read more about in the Python > Library Reference (datetime.tzinfo). Amost all (over 540) of the Olson > timezones are supported. Unfortunately, it would appear that the Olson tz database contains some rather... uh... confusing data. For example: >>> pytz.timezone('Africa/Johannesburg') # SAST is UTC+2 not UTC+1.5 >>> pytz.timezone('Etc/GMT+2')._utcoffset datetime.timedelta(-1, 79200) # I thought I asked for GMT+2, not GMT-2 -- mithrandi, i Ainil en-Balandor, a faer Ambar -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20070311/674981ce/attachment.pgp From lists at cheimes.de Sun Mar 11 01:45:23 2007 From: lists at cheimes.de (Christian Heimes) Date: Sun, 11 Mar 2007 01:45:23 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <20070311001836.GA2690@mithrandi.za.net> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <43aa6ff70703091408w30476ea6t532c1317a29e4118@mail.gmail.com> <20070311001836.GA2690@mithrandi.za.net> Message-ID: Tristan Seligmann schrieb: > pytz.timezone('Africa/Johannesburg') > > # SAST is UTC+2 not UTC+1.5 > >>>> pytz.timezone('Etc/GMT+2')._utcoffset > datetime.timedelta(-1, 79200) > # I thought I asked for GMT+2, not GMT-2 > > > ------------------------------------------------------------------------ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org From lists at cheimes.de Sun Mar 11 02:27:59 2007 From: lists at cheimes.de (Christian Heimes) Date: Sun, 11 Mar 2007 02:27:59 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <20070311001836.GA2690@mithrandi.za.net> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <43aa6ff70703091408w30476ea6t532c1317a29e4118@mail.gmail.com> <20070311001836.GA2690@mithrandi.za.net> Message-ID: Tristan Seligmann wrote: > Unfortunately, it would appear that the Olson tz database contains some > rather... uh... confusing data. For example: > >>>> pytz.timezone('Africa/Johannesburg') > > # SAST is UTC+2 not UTC+1.5 The tz of Africa/Johannesburg initially started with an offset of 1h30 before 1903. It looks like a bug in pytz >>> dt = datetime(2007, 3, 11, 0, 0, 0) >>> aj = pytz.timezone("Africa/Johannesburg") >>> aj.utcoffset(dt) datetime.timedelta(0, 5400) should be 7200 seconds in 2007 >>>> pytz.timezone('Etc/GMT+2')._utcoffset > datetime.timedelta(-1, 79200) That's another bug. Side note: You shouldn't access the _utcoffset attribute. The offset may depend on the date. I notified Stuart about the two issues in his package. Christian From jon+python-dev at unequivocal.co.uk Sun Mar 11 03:05:39 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Sun, 11 Mar 2007 02:05:39 +0000 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <20070310120538.GT32401@snowy.squish.net> <20070310181057.GF32401@snowy.squish.net> Message-ID: <20070311020539.GG32401@snowy.squish.net> I see you snipped without response my request to back up your claim that "assuming that a date() is a datetime() with a time of midnight will clearly break that logic". Am I to assume you cannot back it up? From greg.ewing at canterbury.ac.nz Sun Mar 11 05:45:48 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 11 Mar 2007 17:45:48 +1300 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <20070310124409.GW32401@snowy.squish.net> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <5.1.1.6.0.20070309182137.0375acd8@sparrow.telecommunity.com> <20070310113616.GS32401@snowy.squish.net> <20070310124409.GW32401@snowy.squish.net> Message-ID: <45F3897C.6030404@canterbury.ac.nz> Jon Ribbens wrote: > What do you feel "next Tuesday plus 12 hours" means? ;-) I would say it's meaningless. My feeling is that subtracting two dates should give an integer number of days, and that is all you should be allowed to add to a date. -- Greg From arigo at tunes.org Sun Mar 11 13:20:50 2007 From: arigo at tunes.org (Armin Rigo) Date: Sun, 11 Mar 2007 13:20:50 +0100 Subject: [Python-Dev] Fwd: Re: [Python-3000] Removing functions from the operator module In-Reply-To: <43aa6ff70703072153n1fe873fapb17a6421f8d5de8e@mail.gmail.com> References: <20070307201628.BAQ33047@ms09.lnh.mail.rcn.net> <43aa6ff70703072153n1fe873fapb17a6421f8d5de8e@mail.gmail.com> Message-ID: <20070311122050.GA18273@code0.codespeak.net> Hi Collin, On Wed, Mar 07, 2007 at 11:53:45PM -0600, Collin Winter wrote: > bool() and abs() aren't syntax, so I would never look in operator. abs() is not syntax but bool() is part of every syntactic construction that takes a truth value argument (if, while, and, ...) A bientot, Armin From lgautier at gmail.com Sun Mar 11 15:46:01 2007 From: lgautier at gmail.com (Laurent Gautier) Date: Sun, 11 Mar 2007 22:46:01 +0800 Subject: [Python-Dev] pydoc II Message-ID: <27d1e6020703110746x4d85b2ffq8b0c6029fe5af29e@mail.gmail.com> Hi, There was a thread some weeks ago about work on pydoc, with the admitted aim of proposing something as an upgrade to the current pydoc. We had a number of suggestions, on this very list or privately, and we looked into having something that would not only retain the capabilities of the current pydoc but facilitate customization when it comes to formatting the help, or process the docstrings. Looking at things, and implementing along to see how it would look like, we are coming to something that looks very much like a new implementation... and before going further I would like to ensure that that does not mean an automatic "no way". The obvious question is "why change something that is working ?", and the answer is as simple as although pydoc is a very helpful, there are a number of functionalities that appeared wished so much that other modules appeared. What is proposed here is along the line of "the same but more extensible". The pydoc envisioned is both a library for extracting documentation, as well as few standalone applications using that library (a documentation browser, an interactive console help... just like pydoc currently has). A prototype is being worked on, and I have been looking at code and/or functionalities in pydoc, epydoc, pydoctor, and ipython (for the interactive console), and there a lot of nice things in all these efforts. The name of the game is now to have something that likely to offer the best of all (although it can be an easy way to get no one happy in the end). Hopefully someone kept reading up to here... Some of the key points so far: library: - ability to browse documentation inside module/package files without loading them - ability to browse documentation attached to "live" (that is loaded) objects - documentation API that is independent of whether the object is loaded or not - low-energy barrier for adding support for "other-than-plain-text" docstrings - features like searching objects by name, or by content in their doctring applications: - HTML server to consult the documentation for installed packages - interactive console to consults the documentation for installed and/or loaded objects - generation of HTML documentation How does this sound so far ? Laurent From gjcarneiro at gmail.com Sun Mar 11 15:56:57 2007 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Sun, 11 Mar 2007 14:56:57 +0000 Subject: [Python-Dev] pydoc II In-Reply-To: <27d1e6020703110746x4d85b2ffq8b0c6029fe5af29e@mail.gmail.com> References: <27d1e6020703110746x4d85b2ffq8b0c6029fe5af29e@mail.gmail.com> Message-ID: On 3/11/07, Laurent Gautier wrote:[...] > A prototype is being worked on, and I have been looking at code and/or > functionalities in pydoc, epydoc, pydoctor, and ipython (for the > interactive console), and there a lot of nice things in all these > efforts. The name of the game is now to have something that likely to > offer the best of all (although it can be an easy way to get no one > happy in the end). > > Hopefully someone kept reading up to here... > > Some of the key points so far: > > library: > - ability to browse documentation inside module/package files without > loading them Most importantly, we also need the reserve: ability to attach external documentation to modules/packages without increasing their size. For instance, one of the reasons no one ever bothered to add pydoc documentation to PyGTK (beyond a few docstrings that can be generated in runtime) is that the entire documentation for PyGTK would be far too large to place directly into the module readonly data segment. Documentation should be able to live in another file, separate from the module source code, and loaded on demand. Loading the pydoc file should be preferably done through read-only memory mapping a file. -- Gustavo J. A. M. Carneiro "The universe is always one step beyond logic." -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070311/705ce3d6/attachment.html From lgautier at gmail.com Sun Mar 11 16:10:08 2007 From: lgautier at gmail.com (Laurent Gautier) Date: Sun, 11 Mar 2007 23:10:08 +0800 Subject: [Python-Dev] pydoc II In-Reply-To: References: <27d1e6020703110746x4d85b2ffq8b0c6029fe5af29e@mail.gmail.com> Message-ID: <27d1e6020703110810r764de3bvef081b8d466e0b2d@mail.gmail.com> 2007/3/11, Gustavo Carneiro : > On 3/11/07, Laurent Gautier wrote: > [...] > > A prototype is being worked on, and I have been looking at code and/or > > functionalities in pydoc, epydoc, pydoctor, and ipython (for the > > interactive console), and there a lot of nice things in all these > > efforts. The name of the game is now to have something that likely to > > offer the best of all (although it can be an easy way to get no one > > happy in the end). > > > > Hopefully someone kept reading up to here... > > > > Some of the key points so far: > > > > library: > > - ability to browse documentation inside module/package files without > > loading them > > Most importantly, we also need the reserve: ability to attach external > documentation to modules/packages without increasing their size. I suspect that this will sparkle a long thread about python documentation system, but I will comment nevertheless while trying to avoid being caught in the middle. Well this sort of thing is technically feasible, and when I am looking solely at pydoc the deal for me is how can it be made easily and as little hackish as possible. > For instance, one of the reasons no one ever bothered to add pydoc > documentation to PyGTK (beyond a few docstrings that can be generated in > runtime) is that the entire documentation for PyGTK would be far too large > to place directly into the module readonly data segment. > > Documentation should be able to live in another file, separate from the > module source code, and loaded on demand. Loading the pydoc file should be > preferably done through read-only memory mapping a file. Well, that's an interesting idea... and the structure we start having could see something along those lines implemented. Being something non-standard for the time being, that would be something that the maintainer of the module having that particular request would take care of. (the deal with pydoc would be to make plugging that in easy) Instructions to use that "other-documentation" through pydoc could be included in the top-level docstring for the pygtk module for example... how does this sound ? L. > -- > Gustavo J. A. M. Carneiro > "The universe is always one step beyond logic." From tjreedy at udel.edu Sun Mar 11 16:54:32 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 11 Mar 2007 11:54:32 -0400 Subject: [Python-Dev] pydoc II References: <27d1e6020703110746x4d85b2ffq8b0c6029fe5af29e@mail.gmail.com> Message-ID: "Gustavo Carneiro" wrote in message news:a467ca4f0703110756w12deee46k333e0d5118fbbacc at mail.gmail.com... | Most importantly, we also need the reserve: ability to attach external | documentation to modules/packages without increasing their size. Perhaps use .pd for the extension tjr From tonynelson at georgeanelson.com Sun Mar 11 17:02:39 2007 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Sun, 11 Mar 2007 11:02:39 -0500 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <45F3897C.6030404@canterbury.ac.nz> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <5.1.1.6.0.20070309182137.0375acd8@sparrow.telecommunity.com> <20070310113616.GS32401@snowy.squish.net> <20070310124409.GW32401@snowy.squish.net> <45F3897C.6030404@canterbury.ac.nz> Message-ID: At 5:45 PM +1300 3/11/07, Greg Ewing wrote: >Jon Ribbens wrote: > >> What do you feel "next Tuesday plus 12 hours" means? ;-) > >I would say it's meaningless. My feeling is that subtracting >two dates should give an integer number of days, and that is >all you should be allowed to add to a date. Apple's old MacOS had a very flexible LongDateRecord and date utilities. Nearly anything one could do to a date had a useful meaning. Perhaps Python should be different, but I've found Apple's date calculations and date parsing to be very useful, in a Pythonic sort of way. >From old New Inside Macintosh, _Operating System Utilities_, Ch. 4 "Date, Time, and Measurement Utilities": Calculating Dates ----------------- In the date-time record and long date-time record, any value in the month, day, hour, minute, or second field that exceeds the maximum value allowed for that field, will cause a wraparound to a future date and time when you modify the date-time format. * In the month field, values greater than 12 cause a wraparound to a future year and month. * In the day field, values greater than the number of days in a given month cause a wraparound to a future month and day. * In the hour field, values greater than 23 cause a wraparound to a future day and hour. * In the minute field, values greater than 59 cause a wraparound to a future hour and minute. * In the seconds field, values greater than 59 cause a wraparound to a future minute and seconds. You can use these wraparound facts to calculate and retrieve information about a specific date. For example, you can use a date-time record and the DateToSeconds and SecondsToDate procedures to calculate the 300th day of 1994. Set the month field of the date-time record to 1 and the year field to 1994. To find the 300th day of 1994, set the day field of the date-time record to 300. Initialize the rest of the fields in the record to values that do not exceed the maximum value allowed for that field. (Refer to the description of the date-time record on page 4-23 for a complete list of possible values). To force a wrap-around, first convert the date and time (in this example, January 1, 1994) to the number of seconds elapsed since midnight, January 1, 1904 (by calling the DateToSeconds procedure). Once you have converted the date and time to a number of seconds, you convert the number of seconds back to a date and time (by calling the SecondsToDate procedure). The fields in the date-time record now contain the values that represent the 300th day of 1994. Listing 4-6 shows an application-defined procedure that calculates the 300th day of the Gregorian calendar year using a date-time record. Listing 4-6 Calculating the 300th day of the year PROCEDURE MyCalculate300Day; VAR myDateTimeRec: DateTimeRec; mySeconds: LongInt; BEGIN WITH myDateTimeRec DO BEGIN year := 1994; month := 1; day := 300; hour := 0; minute := 0; second := 0; dayOfWeek := 1; END; DateToSeconds (myDateTimeRec, mySeconds); SecondsToDate (mySeconds, myDateTimeRec); END; The DateToSeconds procedure converts the date and time to the number of seconds elapsed since midnight, January 1, 1904, and the SecondsToDate procedure converts the number of seconds back to a date and time. After the conversions, the values in the year, month, day, and dayOfWeek fields of the myDateTimeRec record represent the year, month, day of the month, and day of the week for the 300th day of 1994. If the values in the hour, minute, and second fields do not exceed the maximum value allowed for each field, the values remain the same after the conversions (in this example, the time is exactly 12:00 A.M.). Similarly, you can use a long date-time record and the LongDateToSeconds and LongSecondsToDate procedures to compute the day of the week corresponding to a given date. Listing 4-7 shows an application-defined procedure that computes and retrieves the name of the day for July 4, 1776. Note that because the year is prior to 1904, it is necessary to use a long date-time record. -- ____________________________________________________________________ TonyN.:' ' From phd at phd.pp.ru Sun Mar 11 17:13:15 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Sun, 11 Mar 2007 19:13:15 +0300 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <5.1.1.6.0.20070309182137.0375acd8@sparrow.telecommunity.com> <20070310113616.GS32401@snowy.squish.net> <20070310124409.GW32401@snowy.squish.net> <45F3897C.6030404@canterbury.ac.nz> Message-ID: <20070311161315.GA6401@phd.pp.ru> On Sun, Mar 11, 2007 at 11:02:39AM -0500, Tony Nelson wrote: > Apple's old MacOS had a very flexible LongDateRecord and date utilities. > Nearly anything one could do to a date had a useful meaning. Perhaps > Python should be different, but I've found Apple's date calculations and > date parsing to be very useful, in a Pythonic sort of way. Python world has excellent mxDateTime that does a very good job of date parsing and date/time arithmetic! Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From steven.bethard at gmail.com Sun Mar 11 22:08:52 2007 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 11 Mar 2007 14:08:52 -0700 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <20070311020539.GG32401@snowy.squish.net> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <20070310120538.GT32401@snowy.squish.net> <20070310181057.GF32401@snowy.squish.net> <20070311020539.GG32401@snowy.squish.net> Message-ID: On 3/10/07, Jon Ribbens wrote: > I see you snipped without response my request to back up your claim > that "assuming that a date() is a datetime() with a time of midnight > will clearly break that logic". > > Am I to assume you cannot back it up? I was trying to minimize the extent of this already too long thread, assuming you could go do the reading I referred you to if you were really interested in the answer. I still encourage you to read the reference, but for your sake, here's a brief example of temporal logic that would break:: We know that: date(2006, 1, 1) *Includes* datetime(2006, 1, 1, 0, 0, 1) And under your definition date(2006, 1, 1) *Is Simultaneous With* datetime(2006, 1, 1, 0, 0, 0) But that's a contradiction because datetime(2006, 1, 1, 0, 0, 0) *Does Not Include* datetime(2006, 1, 1, 0, 0, 1) Hope that helps, STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From lists at cheimes.de Sun Mar 11 22:56:10 2007 From: lists at cheimes.de (Christian Heimes) Date: Sun, 11 Mar 2007 22:56:10 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <20070311020539.GG32401@snowy.squish.net> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <20070310120538.GT32401@snowy.squish.net> <20070310181057.GF32401@snowy.squish.net> <20070311020539.GG32401@snowy.squish.net> Message-ID: Jon Ribbens schrieb: > I see you snipped without response my request to back up your claim > that "assuming that a date() is a datetime() with a time of midnight > will clearly break that logic". I've another idea. Date and datetime objects are compared by date. The date object for a given date is always smaller than the datetime for the same day - even for midnight. The date 2007-01-02 is smaller than 2007-01-02 00:00:00 but larger than 2007-01-01 24:00:00. >>> date(2007, 1, 2) < datetime(2007, 1, 2, 0, 0, 0) True >>> date(2007, 1, 2) > datetime(2007, 1, 1, 24, 0, 0) True >>> date(2007, 1, 2) == datetime(2007, 1, 2, 0, 0, 0) False >>> date(2007, 1, 2) in datetime(2007, 1, 2, 0, 0, 0) TypeError(...) >>> datetime(2007, 1, 2, 0, 0, 0) < date(2007, 1, 2) TypeError(...) >>> datetime(2007, 1, 2, 0, 0, 0) > date(2007, 1, 2) TypeError(...) >>> datetime(2007, 1, 2, 0, 0, 0) == date(2007, 1, 2) False >>> datetime(2007, 1, 2, 0, 0, 0) in date(2007, 1, 2) True From pmaupin at gmail.com Sun Mar 11 23:32:28 2007 From: pmaupin at gmail.com (Patrick Maupin) Date: Sun, 11 Mar 2007 17:32:28 -0500 Subject: [Python-Dev] Backports of standard library modules Message-ID: Please forgive me if this has already been discussed -- I only recently started paying any attention to this list. Many versions of python include "new" standard library modules that could (or often do already) work fine with previous versions. Some of the modules are brand-new, written explicitly for the standard library, and others are preexisting modules that, by community consensus, are added to the library. It seems that once a new library module is added, if it doesn't rely heavily on new core language features, it should also be added for previous versions. Obviously, it doesn't make sense to re-release an old version of Python just because a new library is available (or even to include the new library in the old version), but it might make sense to have a separate package of "extras" or "backported" modules that is re-released for prior versions whenever a new (compatible) module is added to the standard library. Although it is often quite easy to find and download a module for use with a prior version of Python, it would be really convenient if all of these modules were bundled up together and available as a single download, especially when it comes to Windows users and extension modules. If this seems useful to others, I could try to start a PEP on how the process would work (but since I'm fairly new, it would be great if someone could help out a bit by validating my verbiage against some of the process requirements). Regards, Pat From steven.bethard at gmail.com Sun Mar 11 23:37:56 2007 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 11 Mar 2007 15:37:56 -0700 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <20070310120538.GT32401@snowy.squish.net> <20070310181057.GF32401@snowy.squish.net> <20070311020539.GG32401@snowy.squish.net> Message-ID: On 3/11/07, Christian Heimes wrote: > Jon Ribbens schrieb: > > I see you snipped without response my request to back up your claim > > that "assuming that a date() is a datetime() with a time of midnight > > will clearly break that logic". > > I've another idea. Date and datetime objects are compared by date. The > date object for a given date is always smaller than the datetime for the > same day - even for midnight. I really don't understand what the problem is with having a date() behave like a proper temporal interval. The only person complaining about that interpretation acknowledged that for his purposes, it would be better than the status quo. And I have yet to see a use case where being consistent with temporal interval logic is a problem. STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From bjourne at gmail.com Mon Mar 12 00:09:16 2007 From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=) Date: Mon, 12 Mar 2007 00:09:16 +0100 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <20070310120538.GT32401@snowy.squish.net> <20070310181057.GF32401@snowy.squish.net> <20070311020539.GG32401@snowy.squish.net> Message-ID: <740c3aec0703111609t41502c88ude6acf13d6c784be@mail.gmail.com> On 3/11/07, Steven Bethard wrote: > On 3/11/07, Christian Heimes wrote: > > I've another idea. Date and datetime objects are compared by date. The > > date object for a given date is always smaller than the datetime for the > > same day - even for midnight. > > I really don't understand what the problem is with having a date() > behave like a proper temporal interval. The only person complaining > about that interpretation acknowledged that for his purposes, it would > be better than the status quo. And I have yet to see a use case where > being consistent with temporal interval logic is a problem. I do not really understand proper temporal intervals. But I am interested what "temporal interval logic" has to say about this problem: def get_most_recent_articles(articles, cutoff_date): recent_articles = [] for article in articles: if article.datetime_posted > cutoff_date: recent_articles.append(article) return recent_articles Would temporal interval logic make it so an article with datetime(2007, 1, 1, 0, 0, 0) be included, if cutoff_date was date(2007, 1, 1)? What about articles with datetimes (2007, 1, 1, 0, 0, 1) and (2007, 1, 2, 0, 0, 0) respectively? I believe that "temporal interval logic" has to include at least the last two examples in recent_articles, otherwise it would be highly annoying. -- mvh Bj?rn From brett at python.org Mon Mar 12 00:19:00 2007 From: brett at python.org (Brett Cannon) Date: Sun, 11 Mar 2007 16:19:00 -0700 Subject: [Python-Dev] Backports of standard library modules In-Reply-To: References: Message-ID: On 3/11/07, Patrick Maupin wrote: > Please forgive me if this has already been discussed -- I only > recently started paying any attention to this list. > > Many versions of python include "new" standard library modules that > could (or often do already) work fine with previous versions. Some of > the modules are brand-new, written explicitly for the standard > library, and others are preexisting modules that, by community > consensus, are added to the library. > > It seems that once a new library module is added, if it doesn't rely > heavily on new core language features, it should also be added for > previous versions. Obviously, it doesn't make sense to re-release an > old version of Python just because a new library is available (or even > to include the new library in the old version), but it might make > sense to have a separate package of "extras" or "backported" modules > that is re-released for prior versions whenever a new (compatible) > module is added to the standard library. > > Although it is often quite easy to find and download a module for use > with a prior version of Python, it would be really convenient if all > of these modules were bundled up together and available as a single > download, especially when it comes to Windows users and extension > modules. > A problem with doing this is whether this will put any extra burden on python-dev. It's fine if people want to pull out of the stdlib, package them, and do that work, but in no way should we have to maintain version compatibility or worry about backporting fixes once the package is released. I would personally rather have an informal package done to completely avoid these issues. That way no one feels obliged to keep doing this after every release. -Brett From steven.bethard at gmail.com Mon Mar 12 00:41:47 2007 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 11 Mar 2007 16:41:47 -0700 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <740c3aec0703111609t41502c88ude6acf13d6c784be@mail.gmail.com> References: <20070310120538.GT32401@snowy.squish.net> <20070310181057.GF32401@snowy.squish.net> <20070311020539.GG32401@snowy.squish.net> <740c3aec0703111609t41502c88ude6acf13d6c784be@mail.gmail.com> Message-ID: On 3/11/07, BJ?rn Lindqvist wrote: > I do not really understand proper temporal intervals. But I am > interested what "temporal interval logic" has to say about this > problem: > > def get_most_recent_articles(articles, cutoff_date): > recent_articles = [] > for article in articles: > if article.datetime_posted > cutoff_date: > recent_articles.append(article) > return recent_articles Thanks for being concrete here. =) > Would temporal interval logic make it so an article with > datetime(2007, 1, 1, 0, 0, 0) be included, if cutoff_date was > date(2007, 1, 1)? No, datetime(2007, 1, 1, 0, 0, 0) is not *after* date(2007, 1, 1), it is *included* within it. If you want an article with that datetime to be included, you should be comparing against datetime(2006, 12, 31, 23, 59, 59) or date(2006, 12, 31). > What about articles with datetimes (2007, 1, 1, 0, 0, 1) No. For the same reason as above. > and (2007, 1, 2, 0, 0, 0) respectively? Yes. > I believe that "temporal interval logic" has to include at least > the last two examples in recent_articles, otherwise it would > be highly annoying. Could you elaborate? Right now, to get the results you want with your code, you have to compare against datetime(2006, 12, 31, 23, 59, 59). How is comparing against date(2006, 12, 31) any worse? Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From fumanchu at amor.org Mon Mar 12 01:07:53 2007 From: fumanchu at amor.org (Robert Brewer) Date: Sun, 11 Mar 2007 17:07:53 -0700 Subject: [Python-Dev] datetime module enhancements References: <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com><435DF58A933BA74397B42CDEB8145A86224D5A@ex9.hostedexchange.local> <20070310170733.GD32401@snowy.squish.net> Message-ID: <435DF58A933BA74397B42CDEB8145A86224D5C@ex9.hostedexchange.local> Jon Ribbens wrote: > > Robert Brewer wrote: > > > One solution that just occurred to me -- and that > > > skirts the issue of choosing an interpretation -- > > > is that, when comparing date and datetime objects, > > > the datetime's .date() method is called and the > > > result of that call is compared to the original > > > date. > > +1 > > > > ...and it's a decision that can be made independently > > of how to add or subtract dates. > > I don't like it. It sounds suspiciously like "when > comparing integers and floats, discard the non-integer > portion of the float and then compare as integers". ...which can happen quite often, depending on your domain's requirements for significant digits. Integers and floats are numbers, not units, so a more proper analogy would have been, "it sounds like when comparing feet and inches, divide inches by 12 and discard any remainder." Which, again, can happen quite often. But even feet and inches aren't a good analogy, because people don't say, "six-foot-three is six feet". But they *do* say, "is it Thursday right now?" quite often, and expect a yes-or-no answer, not "cannot compute". A slightly better analogy would be armies and platoons: when you compare armies (on almost any scale except size), you can't say an army is more or less than a platoon in that same army. The platoon is (part of) the army and the army is (made of) the platoons. So order is important only at a given level of granularity: Army1 = [P1, P2, P3] Army2 = [Px, Py, Pz] sorted([Army1, Army2, P1, P2, P3, Px, Py, Pz]) [Army1, P1, P2, P3, Px, Py, Army2, Pz] ...that is, it doesn't matter where Army2 appears, as long as it's in the second half of the list. Note that the platoons can still be ordered relative to other platoons in the same army. Likewise, when comparing dates to date-times (but not when adding them), the mental model of dates and times acts more like nested sets than continuous functions: Date1 = Datetimes(1, 2, 3) Date2 = Datetimes(x, y, z) sorted([Date1, Date2, ...]) [Date1, T1, T2, T3, Tx, Ty, Date2, Tz] ...and the above can be achieved by: class nested_datetime(datetime): def __cmp__(self, other): if isinstance(other, datetime): return datetime.__cmp__(self, other) elif isinstance(other, date): return cmp(self.date(), other) raise TypeError Robert Brewer System Architect Amor Ministries fumanchu at amor.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070311/27832859/attachment-0001.html From kbk at shore.net Mon Mar 12 01:27:51 2007 From: kbk at shore.net (Kurt B. Kaiser) Date: Sun, 11 Mar 2007 19:27:51 -0500 (EST) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200703120027.l2C0RpKe005438@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 380 open (-36) / 3658 closed (+65) / 4038 total (+29) Bugs : 965 open ( -9) / 6555 closed (+35) / 7520 total (+26) RFE : 272 open ( +4) / 253 closed ( +2) / 525 total ( +6) New / Reopened Patches ______________________ Extension documentation for subclasses (2007-03-01) CLOSED http://python.org/sf/1671450 opened by Pete Shinners MultiCall bug crashing the config dialog (2007-03-02) CLOSED http://python.org/sf/1672481 opened by Tal Einat urllib2 requests history + HEAD support (2007-03-03) http://python.org/sf/1673007 opened by KDanilov aka koder README update: Bash default on ac OS X (2007-03-03) CLOSED http://python.org/sf/1673121 opened by Witten Mac OS X: libtool (2007-03-03) http://python.org/sf/1673122 opened by Witten change repr(...) from 'Ellipsis' to '...' (2007-03-04) CLOSED http://python.org/sf/1673355 opened by John Reese Identify modules to be built and not built (2007-03-04) CLOSED http://python.org/sf/1673619 opened by Skip Montanaro 'G' formatting doesn't catch same errors as 'g' (2007-03-04) http://python.org/sf/1673759 opened by Eric V. Smith Identify modules which couldn't be built (2007-03-04) CLOSED http://python.org/sf/1673619 reopened by loewis sq_ass_slice ignored if sq_slice not defined (2007-03-05) CLOSED http://python.org/sf/1674228 opened by ?iga Seilnacht Epoll wrapper (2007-03-06) http://python.org/sf/1675118 opened by TH Draft implementation for PEP 364 (2007-03-06) http://python.org/sf/1675334 opened by Barry A. Warsaw PEP 3114 -- next() -> __next__() (2007-03-07) http://python.org/sf/1675363 opened by Georg Brandl Make PyComplex_AsCComplex use __complex__ (2007-03-07) http://python.org/sf/1675423 opened by Mark Dickinson Zipfile tweaks and test coverage improvement (2007-03-06) http://python.org/sf/1675424 opened by Alan McIntyre Use getaddrinfo() in urllib2.py for IPv6 support (2007-03-07) http://python.org/sf/1675455 opened by David Cantrell Refactor test_pty.py to use unittest. (2007-03-06) CLOSED http://python.org/sf/1675471 opened by Jerry Seutter [gzip] Performance for small reads and fix seek problem (2007-03-07) http://python.org/sf/1675951 opened by Florian Festi Remove dead code in typeobject's type_new() (2007-03-07) CLOSED http://python.org/sf/1675981 opened by ?iga Seilnacht Remove trailing slash from --prefix (2007-03-07) http://python.org/sf/1676135 opened by Bj?rn Lindqvist New File I/O type for Python 3000, plus .h and unit tests (2007-02-28) CLOSED http://python.org/sf/1671314 reopened by collinwinter Add a PeriodicTimer to threading (2007-03-08) http://python.org/sf/1676820 opened by Bj?rn Lindqvist Adding timeout to socket.py and httplib.py (2007-03-08) http://python.org/sf/1676823 opened by Facundo Batista Refactor test_popen2.py to use unittest. (2007-03-08) http://python.org/sf/1676994 opened by Jerry Seutter Support CREATE_SUSPENDED flag in subprocess.py for Win32 (2007-03-09) http://python.org/sf/1677688 opened by Chris Heller site.py small ?bug fix | change? (2007-03-10) http://python.org/sf/1677862 opened by KDanilov aka koder Efficient reverse line iterator (2007-03-10) http://python.org/sf/1677872 opened by Mark Russell Removal of Tuple Parameter Unpacking [PEP3113] (2007-03-10) http://python.org/sf/1678060 opened by Tony Lownds improve telnetlib.Telnet so option negotiation becomes easie (2007-03-10) http://python.org/sf/1678077 opened by Bj?rn Lindqvist Refactor test_operations.py to use unittest. (2007-03-10) http://python.org/sf/1678088 opened by Jerry Seutter Adding a testcase for the bug in find_longest_match (2007-03-11) http://python.org/sf/1678339 opened by Denys Rtveliashvili A fix for the bug #1528074 [warning: quite slow] (2007-03-11) http://python.org/sf/1678345 opened by Denys Rtveliashvili Patches Closed ______________ Extension documentation for subclasses (2007-03-01) http://python.org/sf/1671450 closed by gbrandl MultiCall bug crashing the config dialog (2007-03-02) http://python.org/sf/1672481 closed by gbrandl README update: Bash default on Mac OS X (2007-03-03) http://python.org/sf/1673121 closed by gbrandl change repr(...) from 'Ellipsis' to '...' (2007-03-04) http://python.org/sf/1673355 closed by gvanrossum Use a set to keep interned strings (2006-06-16) http://python.org/sf/1507011 closed by loewis Add support for the If-Modified-Since header (2006-03-04) http://python.org/sf/1442867 closed by loewis N-d array interface for array object (2006-03-18) http://python.org/sf/1452906 closed by loewis Fix for win32 proxy bypass support (no_proxy) in urllib(2) (2005-03-01) http://python.org/sf/1154227 closed by loewis Names conflict with QT (on linux) (2005-03-08) http://python.org/sf/1158926 closed by loewis PYTHONBYTECODEBASE patch (PEP 304) (2003-01-29) http://python.org/sf/677103 closed by loewis Identify modules which couldn't be built (2007-03-04) http://python.org/sf/1673619 closed by montanaro Identify modules which couldn't be built (2007-03-04) http://python.org/sf/1673619 closed by loewis sq_ass_slice ignored if sq_slice not defined (2007-03-05) http://python.org/sf/1674228 closed by gbrandl Add syntax for dynamic attribute access (2007-02-11) http://python.org/sf/1657573 closed by bennorth ZipFile.open - read-only file-like obj for files in archive (2005-02-12) http://python.org/sf/1121142 closed by loewis 2 Tools for easy inter-thread communication->Queue,threading (2006-03-20) http://python.org/sf/1454452 closed by loewis Patch to implement PEP 351 (2005-10-23) http://python.org/sf/1335812 closed by bwarsaw PEP 343 implementation (2005-07-11) http://python.org/sf/1235943 closed by gbrandl Fix for #947380 - sys.path is wrong on windows sometimes (2004-05-04) http://python.org/sf/947386 closed by zseil enable "python -m doctest FILE" to run tests in FILE (2007-02-19) http://python.org/sf/1663234 closed by gbrandl small sanity checks for user-defined mros (2005-03-15) http://python.org/sf/1163731 closed by loewis add None values in SimpleXMLRPCServer (2006-12-18) http://python.org/sf/1618485 closed by collinwinter HTMLParser should support entities in attributes (2004-03-09) http://python.org/sf/912410 closed by loewis Little (improvement and standarization) to asyncore. (2004-03-11) http://python.org/sf/914096 closed by josiahcarlson gzip.GzipFile to accept stream as fileobj. (2004-03-11) http://python.org/sf/914340 closed by gbrandl operator.c slice functions need to parse ssize_t (2007-02-07) http://python.org/sf/1654417 closed by gbrandl ConfigParser getboolean() consistency (2007-01-28) http://python.org/sf/1646432 closed by gbrandl clarify comparison return values (2006-11-24) http://python.org/sf/1602128 closed by gbrandl Fix to the long("123\0", 10) problem (2007-01-18) http://python.org/sf/1638879 closed by gbrandl Implementation of PEP 362 (2006-08-22) http://python.org/sf/1544909 closed by bcannon Improved profiler (2005-06-01) http://python.org/sf/1212837 closed by arigo fix bug #670311: sys.exit and PYTHONINSPECT (2003-03-04) http://python.org/sf/697613 closed by gbrandl Fix strange warnings from interpreter if PYTHONSTARTUP used (2003-03-14) http://python.org/sf/703779 closed by gbrandl glob doesn't return unicode with no dir in unicode filename (2004-08-01) http://python.org/sf/1001604 closed by gbrandl dircache fix; raise like os.listdir, plus optimization (2004-07-25) http://python.org/sf/997726 closed by gbrandl Refactor test_pty.py to use unittest. (2007-03-07) http://python.org/sf/1675471 closed by gbrandl More precise realm parsing in AbstractBasicAuthHandler (2003-09-25) http://python.org/sf/812285 closed by gbrandl Add a test suite for unittest (2006-09-01) http://python.org/sf/1550272 closed by gbrandl Fix numerous bugs in unittest (2006-09-01) http://python.org/sf/1550273 closed by gbrandl unittest.py: Custom TestRunners and --verbose (2003-08-13) http://python.org/sf/787789 closed by gbrandl set comprehensions (2006-08-29) http://python.org/sf/1548388 closed by gbrandl ConfigParser to save with order (2006-01-07) http://python.org/sf/1399309 closed by facundobatista Remove dead code in typeobject's type_new() (2007-03-07) http://python.org/sf/1675981 closed by zseil Fixes SocketServer Bug 1531963 (2006-08-22) http://python.org/sf/1545011 closed by collinwinter Unification of list-comp and for syntax (2006-05-21) http://python.org/sf/1492509 closed by collinwinter Complex representation (2006-05-19) http://python.org/sf/1491866 closed by collinwinter Remove operator.truth() and operator.abs() (2006-07-13) http://python.org/sf/1522059 closed by collinwinter XML Test Runner for unittest module (2006-07-14) http://python.org/sf/1522704 closed by collinwinter New File I/O type for Python 3000, plus .h and unit tests (2007-02-28) http://python.org/sf/1671314 closed by collinwinter New File I/O type for Python 3000, plus .h and unit tests (2007-02-28) http://python.org/sf/1671314 closed by gvanrossum ast branch changed interactive module name (2006-06-25) http://python.org/sf/1512007 closed by collinwinter Patch for bug #1380970 (2006-01-25) http://python.org/sf/1414934 closed by collinwinter email.utils.parsedate documentation (2007-01-19) http://python.org/sf/1639973 closed by collinwinter Module fixedlenfields for standard lib (2005-11-29) http://python.org/sf/1369028 closed by collinwinter don't use '-' and '_' in mkstemp (2007-02-25) http://python.org/sf/1668482 closed by collinwinter Strobe Timer for guis (2005-09-26) http://python.org/sf/1304895 closed by collinwinter Fix for bug 494589 (2002-12-25) http://python.org/sf/658599 closed by nnorwitz Support of HTTP_REFERER in CGIHTTPServer.py (2006-05-03) http://python.org/sf/1481079 closed by collinwinter Improves an error message from setattr (2006-06-26) http://python.org/sf/1512942 closed by collinwinter RFE: Extend smtplib.py with support for LMTP (2004-05-19) http://python.org/sf/957003 closed by loewis TCPServer option to bind and activate (2006-11-20) http://python.org/sf/1599845 closed by collinwinter os.py: base class _Environ on dict instead of UserDict (2005-01-02) http://python.org/sf/1094387 closed by collinwinter Add missing types to __builtin__ (2004-08-16) http://python.org/sf/1009811 closed by collinwinter Patch for 1115886 - splitext incorrectly handles filenames l (2006-03-31) http://python.org/sf/1462106 closed by loewis New / Reopened Bugs ___________________ python code fails with "MemoryError" on HP-UX 11.11 PA_RISC (2007-03-01) CLOSED http://python.org/sf/1671411 opened by shashi test_mailbox is hanging while doing gmake test on HP-UX v3 (2007-03-01) http://python.org/sf/1671676 opened by shashi shutil.copy2 modifies decimal part of last modification time (2007-03-01) CLOSED http://python.org/sf/1671965 opened by Guy Dalberto cPickle can pickle, but cannot unpickle on 64bit machines (2007-03-02) http://python.org/sf/1672332 opened by Bartlomiej Ogryczak Building python 2.5 for AMD64 (windows) and VS2005 (2007-03-02) http://python.org/sf/1672336 opened by fred2k datetime.fromtimestamp fails with negative fractional times (2007-01-28) CLOSED http://python.org/sf/1646728 reopened by gvanrossum silent error in email.message.Message.get_payload (2007-03-02) http://python.org/sf/1672568 opened by Renaud Blanch Error reading files larger than 4GB (2007-03-03) http://python.org/sf/1672853 opened by Case Van Horsen date-datetime comparison doesn't work (2007-03-04) http://python.org/sf/1673403 opened by Jon Ribbens None-datetime comparison doesn't work (2007-03-04) CLOSED http://python.org/sf/1673405 opened by Jon Ribbens datetime module missing some important methods (2007-03-04) http://python.org/sf/1673409 opened by Jon Ribbens string and unicode formatting are missing a test for "G" (2007-03-04) http://python.org/sf/1673757 opened by Eric V. Smith 2.4.4 Logging LogRecord attributes broken (2007-02-26) CLOSED http://python.org/sf/1669498 reopened by murrayg Unicode xmlcharrefreplace produces backslash not xml style. (2007-03-05) CLOSED http://python.org/sf/1674223 opened by Stanley Sokolow execfile() response to syntax errors (2007-03-05) CLOSED http://python.org/sf/1674503 opened by Jonathan Griffitts Python 2.5 testsuite sys.path contains system dirs (2007-03-06) http://python.org/sf/1674555 opened by Theodoros V. Kalamatianos Redirect cause invalid descriptor error (2007-03-06) http://python.org/sf/1675026 opened by Jean-Marc Ranger Python still uses broken -xcode option on Solaris/x86 (2007-03-07) http://python.org/sf/1675511 opened by Carson Gaspar random.randint fails on lists (2007-03-07) CLOSED http://python.org/sf/1675516 opened by sardonics setup.py LDFLAGS regexp is wrong (2007-03-07) http://python.org/sf/1675533 opened by Carson Gaspar Python2.5 can't read a (complex) pickle build by python2.4 (2007-03-07) http://python.org/sf/1675967 opened by Michael Vogt Problem linking to readline lib on x86(64) Solaris (2007-03-07) http://python.org/sf/1676121 opened by Alexander Belopolsky scheduler.empty() in module sched appears broken (2007-03-08) CLOSED http://python.org/sf/1676321 opened by Mark Stdlib reference is in italics (2007-03-08) CLOSED http://python.org/sf/1676656 opened by Bj?rn Lindqvist Complex OverflowError has a typo (2007-03-09) CLOSED http://python.org/sf/1676971 opened by John Dong test_timeout refactoring (2007-03-10) http://python.org/sf/1677694 opened by Bj?rn Lindqvist zlib.crc32() not cross-platform (2007-03-10) http://python.org/sf/1678102 opened by Ben Collver 0.0 and -0.0 identified, with surprising results (2007-03-11) http://python.org/sf/1678380 reopened by marketdickinson 0.0 and -0.0 identified, with surprising results (2007-03-11) http://python.org/sf/1678380 opened by Mark Dickinson StringIO / cStringIO inconsistency with unicode (2007-03-11) CLOSED http://python.org/sf/1678411 opened by Armin Rigo Weird behavior Exception with unicode string (2007-03-11) http://python.org/sf/1678647 opened by Leandro Lucarella Bugs Closed ___________ python code fails with "MemoryError" on HP-UX 11.11 PA_RISC (2007-03-01) http://python.org/sf/1671411 closed by loewis Python needs a way to detect implementation (2007-02-27) http://python.org/sf/1669743 closed by loewis Pydoc sets choices for doc locations incorrectly (2007-01-05) http://python.org/sf/1628895 closed by gbrandl shutil.copy2 modifies decimal part of last modification time (2007-03-01) http://python.org/sf/1671965 closed by loewis datetime.fromtimestamp fails with negative fractional times (2007-01-29) http://python.org/sf/1646728 closed by gbrandl datetime.fromtimestamp fails with negative fractional times (2007-01-28) http://python.org/sf/1646728 closed by gvanrossum Clarify PyMem_Realloc and PyMem_Resize docs (2007-02-26) http://python.org/sf/1669304 closed by gbrandl PyMem_Realloc docs don't specifiy out-of-mem behavior (2007-02-24) http://python.org/sf/1668032 closed by gbrandl PyMem_Resize docs don't specify that it modifies an argument (2007-02-24) http://python.org/sf/1668036 closed by gbrandl None-datetime comparison doesn't work (2007-03-04) http://python.org/sf/1673405 closed by tim_one 2.4.4 Logging LogRecord attributes broken (2007-02-26) http://python.org/sf/1669498 closed by vsajip Unicode xmlcharrefreplace produces backslash not xml style. (2007-03-05) http://python.org/sf/1674223 closed by doerwalter curses should reset curses.{COLS,LINES} when term. resized (2007-01-12) http://python.org/sf/1633621 closed by doerwalter execfile() response to syntax errors (2007-03-05) http://python.org/sf/1674503 closed by gbrandl PyType_IsSubtype segfaults (2007-02-19) http://python.org/sf/1663392 closed by sf-robot sys.path is wrong in some cases (2004-05-04) http://python.org/sf/947380 closed by gbrandl inspect.getargspec() fails with keyword-only arguments (2007-02-25) http://python.org/sf/1668565 closed by bcannon os.path.splitext don't handle unix hidden file correctly (2005-02-04) http://python.org/sf/1115886 closed by loewis sys.exit and PYTHONINSPECT (2003-01-18) http://python.org/sf/670311 closed by gbrandl -xcode=pic32 option is not supported on Solaris x86 Sun C (2006-09-19) http://python.org/sf/1561333 closed by gbrandl random.randint fails on lists (2007-03-07) http://python.org/sf/1675516 closed by quiver document that shutil.copyfileobj does not seek() (2007-02-26) http://python.org/sf/1669331 closed by gbrandl shutil.copyfileobj description is incomplete (2007-02-10) http://python.org/sf/1656578 closed by gbrandl Enhance and correct unittest's docs (redux) (2006-09-01) http://python.org/sf/1550263 closed by gbrandl Handle: class MyTest(unittest.TestSuite) (2004-01-16) http://python.org/sf/878275 closed by gbrandl scheduler.empty() in module sched appears broken (2007-03-08) http://python.org/sf/1676321 closed by rhettinger Stdlib reference is in italics (2007-03-08) http://python.org/sf/1676656 closed by gbrandl evaluated code filename changed to instead of ? (2006-06-29) http://python.org/sf/1514617 closed by collinwinter make_table in difflib does not work with unicode (2007-01-17) http://python.org/sf/1637850 closed by rhettinger Complex OverflowError has a typo (2007-03-08) http://python.org/sf/1676971 closed by nnorwitz documentation of email.utils.parsedate is confusing (2007-01-06) http://python.org/sf/1629566 closed by collinwinter ctypes.string_at(buf, 0) is seen as zero-terminated-string (2007-01-28) http://python.org/sf/1646630 closed by theller test_logging fails with reference count-checking enabled (2006-07-04) http://python.org/sf/1516995 closed by vsajip ctypes.Structure formal parameter dies given tuple (2007-02-03) http://python.org/sf/1651235 closed by theller SocketServer.TCPServer returns different ports (2006-07-31) http://python.org/sf/1531963 closed by collinwinter Nested class __name__ (2002-11-05) http://python.org/sf/633930 closed by gvanrossum SocketServer allow_reuse_address checked in constructor (2006-11-13) http://python.org/sf/1595742 closed by collinwinter StringIO / cStringIO inconsistency with unicode (2007-03-11) http://python.org/sf/1678411 closed by arigo New / Reopened RFE __________________ add identity function (2007-03-04) http://python.org/sf/1673203 opened by paul rubin Make threading.Event().wait(timeout=3) return isSet (2007-03-05) http://python.org/sf/1674032 opened by David Fraser funcName and module incorrectly reported in logging (2007-03-05) CLOSED http://python.org/sf/1674315 opened by Michael Garvin RFE Closed __________ funcName and module incorrectly reported in logging (2007-03-05) http://python.org/sf/1674315 closed by gbrandl Interpreter should be interruptable everywhere (2004-10-08) http://python.org/sf/1043446 closed by rhettinger From jon+python-dev at unequivocal.co.uk Mon Mar 12 02:09:37 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Mon, 12 Mar 2007 01:09:37 +0000 Subject: [Python-Dev] datetime module enhancements In-Reply-To: References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <20070310120538.GT32401@snowy.squish.net> <20070310181057.GF32401@snowy.squish.net> <20070311020539.GG32401@snowy.squish.net> Message-ID: <20070312010937.GK32401@snowy.squish.net> Steven Bethard wrote: > I was trying to minimize the extent of this already too long thread, > assuming you could go do the reading I referred you to if you were > really interested in the answer. There's nothing to be gained by being patronising. > I still encourage you to read the reference, but for your sake, > here's a brief example of temporal logic that would break:: > > We know that: > date(2006, 1, 1) *Includes* datetime(2006, 1, 1, 0, 0, 1) Oh dear. You've fallen at the first hurdle. We do not know that. You are assuming the answer you want and then using it as an axiom. Your justification for this assumption was the whole point of my question, but you have again failed to answer. I don't know how much clearer I can make this, given that I have explicitly mentioned it several times already. > Hope that helps, No, it doesn't. Seriously, please try and understand this. I get that you wish Python datetime was the ultra-complete temporal logic module you want it to be. Good for you. I sympathise. If it was, I would agree with you. But it isn't. It's a simple module for dealing with calendar dates and times, not intervals in time. It makes pragmatic assumptions about what people mean when they use date arithmetic. They might not be the assumptions you want, but then it's probably not the module you want. From collinw at gmail.com Mon Mar 12 02:19:27 2007 From: collinw at gmail.com (Collin Winter) Date: Sun, 11 Mar 2007 20:19:27 -0500 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <20070312010937.GK32401@snowy.squish.net> References: <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <20070310120538.GT32401@snowy.squish.net> <20070310181057.GF32401@snowy.squish.net> <20070311020539.GG32401@snowy.squish.net> <20070312010937.GK32401@snowy.squish.net> Message-ID: <43aa6ff70703111819l185a3d0ag1b6059777f32a449@mail.gmail.com> On 3/11/07, Jon Ribbens wrote: [snark] > It makes pragmatic assumptions about > what people mean when they use date arithmetic. They might not be the > assumptions you want, but then it's probably not the module you want. Please find or write a package that makes the assumptions you want, since datetime clearly isn't the module you want. Anyone desperate to discuss the nuances and finer points of dates and time algebras in Python, take it to python-ideas. Collin Winter From glyph at divmod.com Mon Mar 12 02:19:56 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Mon, 12 Mar 2007 01:19:56 -0000 Subject: [Python-Dev] Backports of standard library modules In-Reply-To: References: Message-ID: <20070312011956.7769.669657250.divmod.xquotient.661@joule.divmod.com> On 11 Mar, 10:32 pm, pmaupin at gmail.com wrote: >If this seems useful to others, I could try to start a PEP on how the >process would work (but since I'm fairly new, it would be great if >someone could help out a bit by validating my verbiage against some of >the process requirements). Isn't this PEP 297? This does raise an interesting question, though, since I'm about to get into PEP authorship myself. Have I missed an official way to propose alternatives or resurrect a languishing PEP? I'd like very much to propose to obsolete PEP 355 with twisted's FilePath object that I've previously discussed, but ... does that mean adding text to 355? writing a new PEP and referencing it? Also, the language/library evolution PEP I would like to write is basically an expanded version of PEP 5, but that is apparently moribund (and "not followed", according to MvL). Above all, how can I help to motivate timely yea-or-nay decisions from the BDFL or "his chosen consultants"? PEP 1 seems to defer all of these questions to emailing the PEP editor; is that really the best way to go? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070312/b2e0cd4a/attachment-0001.html From andrewm at object-craft.com.au Mon Mar 12 02:05:47 2007 From: andrewm at object-craft.com.au (Andrew McNamara) Date: Mon, 12 Mar 2007 12:05:47 +1100 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> <20070228062840.17852.238152490.divmod.quotient.727@ohm> <20070306234849.6305.1580733783.divmod.xquotient.44@joule.divmod.com> Message-ID: <20070312010547.400001D403E@longblack.object-craft.com.au> >I wrote two versions of the dict views refactoring. One that turns >d.keys() into list(d.keys()) and d.iterkeys() into iter(d.keys()). >This one is pretty robust except if you have classes that emulate >2.x-style dicts. But it generates ugly code. So I have a >"light-weight" version that leaves d.keys() alone, while turning >d.iterkeys() into d.keys(). This generates prettier code but more >buggy. I probably should have used the heavy-duty one instead. The ugliness is a virtue in this case as it stands out enough to motivate developers to review each case. The pretty/efficient version is tantamount to guessing, and effectively discards information in the transformation ("here be dragons"). -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/ From collinw at gmail.com Mon Mar 12 02:33:23 2007 From: collinw at gmail.com (Collin Winter) Date: Sun, 11 Mar 2007 20:33:23 -0500 Subject: [Python-Dev] Backports of standard library modules In-Reply-To: <20070312011956.7769.669657250.divmod.xquotient.661@joule.divmod.com> References: <20070312011956.7769.669657250.divmod.xquotient.661@joule.divmod.com> Message-ID: <43aa6ff70703111833s3e4bac1bjea759e14ee1cf190@mail.gmail.com> On 3/11/07, glyph at divmod.com wrote: > On 11 Mar, 10:32 pm, pmaupin at gmail.com wrote: > > >If this seems useful to others, I could try to start a PEP on how the > >process would work (but since I'm fairly new, it would be great if > >someone could help out a bit by validating my verbiage against some of > >the process requirements). > > Isn't this PEP 297? I'd say it's almost a subset of PEP 297. > This does raise an interesting question, though, since I'm about to get into > PEP authorship myself. Have I missed an official way to propose > alternatives or resurrect a languishing PEP? I'd like very much to propose > to obsolete PEP 355 with twisted's FilePath object that I've previously > discussed, but ... does that mean adding text to 355? writing a new PEP and > referencing it? The introduction to PEP 342 would be a good starting point; it was built on a number of prior PEPs. > Above all, how can I help to motivate timely yea-or-nay decisions from the > BDFL or "his chosen consultants"? PEP 1 seems to defer all of these > questions to emailing the PEP editor; is that really the best way to go? >From my own experience, the best way is tight, focused discussion that avoids or short-circuits interminable bikeshedding threads; see Ben North's exemplary work on the (ultimately rejected) PEP on "dynamic attribute access" syntax from February 2007 (http://mail.python.org/pipermail/python-dev/2007-February/, search for his name). Collin Winter From jon+python-dev at unequivocal.co.uk Mon Mar 12 02:37:25 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Mon, 12 Mar 2007 01:37:25 +0000 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <43aa6ff70703111819l185a3d0ag1b6059777f32a449@mail.gmail.com> References: <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <20070310120538.GT32401@snowy.squish.net> <20070310181057.GF32401@snowy.squish.net> <20070311020539.GG32401@snowy.squish.net> <20070312010937.GK32401@snowy.squish.net> <43aa6ff70703111819l185a3d0ag1b6059777f32a449@mail.gmail.com> Message-ID: <20070312013725.GL32401@snowy.squish.net> Collin Winter wrote: > Please find or write a package that makes the assumptions you want, > since datetime clearly isn't the module you want. Datetime clearly *is* the module I want. It already makes the assumptions I want, I just want it to make them consistently. From pmaupin at gmail.com Mon Mar 12 02:43:57 2007 From: pmaupin at gmail.com (Patrick Maupin) Date: Sun, 11 Mar 2007 20:43:57 -0500 Subject: [Python-Dev] Backports of standard library modules In-Reply-To: <20070312011956.7769.669657250.divmod.xquotient.661@joule.divmod.com> References: <20070312011956.7769.669657250.divmod.xquotient.661@joule.divmod.com> Message-ID: On 3/11/07, glyph at divmod.com wrote: > > Isn't this PEP 297? Thanks for pointing that out. I looked for a relevant PEP, but didn't notice this one. I think this is complementary, but (if I am reading it correctly) it is not exactly the same. The alternate search path is a necessary part of what I am suggesting, but the meat of my proposal (which Brett correctly discerned would require ongoing effort :) is that, when I go to the downloads page for Python 2.3, in addition to downloading Python, I could download all the compatible libraries which were included in later versions as a single installable file. When 2.6 comes out, this "extras" package would be upgraded to include any new modules in 2.6. Not a lot of fun for people who live on the bleeding edge, but very useful for people who are stuck with an older version for political or other reasons. Regards, Pat From skip at pobox.com Mon Mar 12 03:01:08 2007 From: skip at pobox.com (skip at pobox.com) Date: Sun, 11 Mar 2007 21:01:08 -0500 Subject: [Python-Dev] These csv test cases seem incorrect to me... Message-ID: <17908.46180.251850.822433@montanaro.dyndns.org> I decided it would be worthwhile to have a csv module written in Python (no C underpinnings) for a number of reasons: * It will probably be easier to add Unicode support to a Python version * More people will be able to read/grok/modify/fix bugs in a Python implementation than in the current mixed Python/C implementation. * With alternative implementations of Python available (PyPy, IronPython, Jython) it makes sense to have a Python version they can use. I'm far from having anything which will pass the current test suite, but in diagnosing some of my current failures I noticed a couple test cases which seem wrong. In the TestDialectExcel class I see these two questionable tests: def test_quotes_and_more(self): self.readerAssertEqual('"a"b', [['ab']]) def test_quote_and_quote(self): self.readerAssertEqual('"a" "b"', [['a "b"']]) It seems to me that if a field starts with a quote it *has* to be a quoted field. Any quotes appearing within a quoted field have to be escaped and the field has to end with a quote. Both of these test cases fail on or the other assumption. If they are indeed both correct and I'm just looking at things crosseyed I think they at least deserve comments explaining why they are correct. Both test cases date from the first checkin. I performed the checkin because of the group developing the module I believe I was the only one with checkin privileges at the time, not because I wrote the test cases. Any ideas about why these test cases are in there? I can't imagine Excel generating either one. Thx, Skip From andrewm at object-craft.com.au Mon Mar 12 03:41:40 2007 From: andrewm at object-craft.com.au (Andrew McNamara) Date: Mon, 12 Mar 2007 13:41:40 +1100 Subject: [Python-Dev] These csv test cases seem incorrect to me... In-Reply-To: <17908.46180.251850.822433@montanaro.dyndns.org> References: <17908.46180.251850.822433@montanaro.dyndns.org> Message-ID: <20070312024141.0050F1D403E@longblack.object-craft.com.au> >I decided it would be worthwhile to have a csv module written in Python (no >C underpinnings) for a number of reasons: Several other people have already done this. I will forward you their e-mail address in a separate private e-mail. >I'm far from having anything which will pass the current test suite, but in >diagnosing some of my current failures I noticed a couple test cases which >seem wrong. In the TestDialectExcel class I see these two questionable >tests: > > def test_quotes_and_more(self): > self.readerAssertEqual('"a"b', [['ab']]) > > def test_quote_and_quote(self): > self.readerAssertEqual('"a" "b"', [['a "b"']]) [...] >Any ideas about why these test cases are in there? I can't imagine Excel >generating either one. The point was to produce the same results as Excel. Sure, Excel probably doesn't generate crap like this itself, but 3rd parties do, and people complain if we don't parse it just like Excel (sigh). -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/ From jon+python-dev at unequivocal.co.uk Mon Mar 12 03:49:05 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Mon, 12 Mar 2007 02:49:05 +0000 Subject: [Python-Dev] These csv test cases seem incorrect to me... In-Reply-To: <20070312024141.0050F1D403E@longblack.object-craft.com.au> References: <17908.46180.251850.822433@montanaro.dyndns.org> <20070312024141.0050F1D403E@longblack.object-craft.com.au> Message-ID: <20070312024905.GM32401@snowy.squish.net> Andrew McNamara wrote: > The point was to produce the same results as Excel. Sure, Excel probably > doesn't generate crap like this itself, but 3rd parties do, and people > complain if we don't parse it just like Excel (sigh). The slight problem with copying Excel is that Excel won't parse its own CSV output. From skip at pobox.com Mon Mar 12 03:56:00 2007 From: skip at pobox.com (skip at pobox.com) Date: Sun, 11 Mar 2007 21:56:00 -0500 Subject: [Python-Dev] These csv test cases seem incorrect to me... In-Reply-To: <20070312024141.0050F1D403E@longblack.object-craft.com.au> References: <17908.46180.251850.822433@montanaro.dyndns.org> <20070312024141.0050F1D403E@longblack.object-craft.com.au> Message-ID: <17908.49472.531685.405836@montanaro.dyndns.org> >> I'm far from having anything which will pass the current test suite, >> but in diagnosing some of my current failures I noticed a couple test >> cases which seem wrong. In the TestDialectExcel class I see these >> two questionable tests: >> >> def test_quotes_and_more(self): >> self.readerAssertEqual('"a"b', [['ab']]) >> >> def test_quote_and_quote(self): >> self.readerAssertEqual('"a" "b"', [['a "b"']]) Andrew> The point was to produce the same results as Excel. Sure, Excel Andrew> probably doesn't generate crap like this itself, but 3rd parties Andrew> do, and people complain if we don't parse it just like Excel Andrew> (sigh). (sigh) indeed. Thanks, Skip From skip at pobox.com Mon Mar 12 04:28:21 2007 From: skip at pobox.com (skip at pobox.com) Date: Sun, 11 Mar 2007 22:28:21 -0500 Subject: [Python-Dev] [Csv] These csv test cases seem incorrect to me... In-Reply-To: <45F4C451.5070303@lexicon.net> References: <17908.46180.251850.822433@montanaro.dyndns.org> <45F4C451.5070303@lexicon.net> Message-ID: <17908.51413.179129.169426@montanaro.dyndns.org> John> IMHO these test cases are *WRONG* and it's a worry that they John> "work" with the current csv module :-( That's my take on things as well, though as Andrew pointed out, given those invalid inputs Excel will produce those wacky outputs. I verified that on my Mac a few minutes ago. I'm inclined to just skip those tests in my Python version, but I can understand that for backwards compatibility the current module needs to grok them. Skip From sjmachin at lexicon.net Mon Mar 12 04:09:05 2007 From: sjmachin at lexicon.net (John Machin) Date: Mon, 12 Mar 2007 14:09:05 +1100 Subject: [Python-Dev] [Csv] These csv test cases seem incorrect to me... In-Reply-To: <17908.46180.251850.822433@montanaro.dyndns.org> References: <17908.46180.251850.822433@montanaro.dyndns.org> Message-ID: <45F4C451.5070303@lexicon.net> On 12/03/2007 1:01 PM, skip at pobox.com wrote: > I decided it would be worthwhile to have a csv module written in Python (no > C underpinnings) for a number of reasons: > > * It will probably be easier to add Unicode support to a Python version > > * More people will be able to read/grok/modify/fix bugs in a Python > implementation than in the current mixed Python/C implementation. > > * With alternative implementations of Python available (PyPy, > IronPython, Jython) it makes sense to have a Python version they can > use. > > I'm far from having anything which will pass the current test suite, but in > diagnosing some of my current failures I noticed a couple test cases which > seem wrong. In the TestDialectExcel class I see these two questionable > tests: > > def test_quotes_and_more(self): > self.readerAssertEqual('"a"b', [['ab']]) > > def test_quote_and_quote(self): > self.readerAssertEqual('"a" "b"', [['a "b"']]) > > It seems to me that if a field starts with a quote it *has* to be a quoted > field. Any quotes appearing within a quoted field have to be escaped and > the field has to end with a quote. Both of these test cases fail on or the > other assumption. If they are indeed both correct and I'm just looking at > things crosseyed I think they at least deserve comments explaining why they > are correct. > > Both test cases date from the first checkin. I performed the checkin > because of the group developing the module I believe I was the only one with > checkin privileges at the time, not because I wrote the test cases. > > Any ideas about why these test cases are in there? I can't imagine Excel > generating either one. > Hi Skip, '"a"b' can't be produced by applying minimalist CSV writing rules to 'ab'. A non-minimalist writer could produce '"ab"', but certainly not '"a"b'. The second case is worse -- it's inconsistent; the reader is supposed to remove the quotes from "a" but not from "b"??? IMHO these test cases are *WRONG* and it's a worry that they "work" with the current csv module :-( Regards, John From sjmachin at lexicon.net Mon Mar 12 05:13:25 2007 From: sjmachin at lexicon.net (John Machin) Date: Mon, 12 Mar 2007 15:13:25 +1100 Subject: [Python-Dev] [Csv] These csv test cases seem incorrect to me... In-Reply-To: <20070312024141.0050F1D403E@longblack.object-craft.com.au> References: <17908.46180.251850.822433@montanaro.dyndns.org> <20070312024141.0050F1D403E@longblack.object-craft.com.au> Message-ID: <45F4D365.8090609@lexicon.net> On 12/03/2007 1:41 PM, Andrew McNamara wrote: > > The point was to produce the same results as Excel. Sure, Excel probably > doesn't generate crap like this itself, but 3rd parties do, and people > complain if we don't parse it just like Excel (sigh). Let's put a little flesh on those a's and b's: A typical example of the first case is where a database address line contains a quoted house name e.g. "Dunromin", 123 Main Street and the producer of the CSV file has not done any quoting at all. An example of the 2nd case is a database address line like this: C/o Mrs Jones, "Dunromin", 123 Main Street and the producer of the CSV file has merely wrapped quotes about it without doubling the existing quotes, to emit this: "C/o Mrs Jones, "Dunromin", 123 Main Street" which Excel and adherents would distort to two fields containing: 'C/o Mrs Jones, Dunromin"' and ' 123 Main Street"' -- aarrgghh!! People who complain as described are IMHO misguided; they are accepting crap and losing data (yes, the quotes in the above examples are *DATA*). Why should we heed their complaints? Perhaps we could consider a non-default "dopey_like_Excel" option for csv :-) BTW, it is possible to do a reasonable recovery job when the producer's protocol was to wrap quotes around the data without doubling existing quotes, providing there were an even number of quotes to start with. It just requires a quite different finite state machine. Cheers, John From greg.ewing at canterbury.ac.nz Mon Mar 12 05:43:12 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 12 Mar 2007 17:43:12 +1300 Subject: [Python-Dev] datetime module enhancements In-Reply-To: <20070312010937.GK32401@snowy.squish.net> References: <43aa6ff70703091153s58eca045l3b483e0225eb6bcd@mail.gmail.com> <43aa6ff70703091314k4392c64r4d366c9703ea1a14@mail.gmail.com> <20070310120538.GT32401@snowy.squish.net> <20070310181057.GF32401@snowy.squish.net> <20070311020539.GG32401@snowy.squish.net> <20070312010937.GK32401@snowy.squish.net> Message-ID: <45F4DA60.8050408@canterbury.ac.nz> Jon Ribbens wrote: > Steven Bethard wrote: > >> We know that: >> date(2006, 1, 1) *Includes* datetime(2006, 1, 1, 0, 0, 1) > > Oh dear. You've fallen at the first hurdle. We do not know that. Translated into English, this is saying "The 1st of January 2006 includes the time one minute past midnight on the 1st of January 2006." That doesn't seem a very controversial thing to believe. -- Greg From andrewm at object-craft.com.au Mon Mar 12 05:46:04 2007 From: andrewm at object-craft.com.au (Andrew McNamara) Date: Mon, 12 Mar 2007 15:46:04 +1100 Subject: [Python-Dev] [Csv] These csv test cases seem incorrect to me... In-Reply-To: <45F4C451.5070303@lexicon.net> References: <17908.46180.251850.822433@montanaro.dyndns.org> <45F4C451.5070303@lexicon.net> Message-ID: <20070312044605.27C481D403E@longblack.object-craft.com.au> >IMHO these test cases are *WRONG* and it's a worry that they "work" with >the current csv module :-( Those tests are not "wrong" - they verify that we produce the same result as Excel when presented with those inputs, which was one of the design goals of the module (and is an important consideration for many of it's users). While you might find the Excel team's choices bizare, they are stable, and in the absence of a formal specification for "CSV", Excel's behaviour is what most users want and expect. If you feel like extending the parser to optionally accept some other format, I have no problem. If you want to make this format the default, make sure you stick around to answer all the angry e-mail from users. -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/ From collinw at gmail.com Mon Mar 12 05:50:42 2007 From: collinw at gmail.com (Collin Winter) Date: Sun, 11 Mar 2007 23:50:42 -0500 Subject: [Python-Dev] Folding test_operations in to test_dict Message-ID: <43aa6ff70703112150l6142c1a9u4d8b4a78d39c2c0a@mail.gmail.com> Jerry Seutter posted patch #1678088 yesterday, which converts test_operations over to use unittest and folds the resulting code into test_dict. It struck me as a good idea, because a) it gets away from output/-based tests, and b) test_operations tests certain dictionary operations, something that would seem at home in test_dict. Before I commit this, though, I want to ask: is there a reason test_operations is separate? I notice that it's listed in regrtest.py in the "run these tests before everything else" list, which made me think test_operations is on its own for a reason. Collin Winter From nnorwitz at gmail.com Mon Mar 12 06:06:06 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sun, 11 Mar 2007 21:06:06 -0800 Subject: [Python-Dev] Folding test_operations in to test_dict In-Reply-To: <43aa6ff70703112150l6142c1a9u4d8b4a78d39c2c0a@mail.gmail.com> References: <43aa6ff70703112150l6142c1a9u4d8b4a78d39c2c0a@mail.gmail.com> Message-ID: On 3/11/07, Collin Winter wrote: > Jerry Seutter posted patch #1678088 yesterday, which converts > test_operations over to use unittest and folds the resulting code into > test_dict. It struck me as a good idea, because a) it gets away from > output/-based tests, and b) test_operations tests certain dictionary > operations, something that would seem at home in test_dict. > > Before I commit this, though, I want to ask: is there a reason > test_operations is separate? I notice that it's listed in regrtest.py > in the "run these tests before everything else" list, which made me > think test_operations is on its own for a reason. svn log shows only one 'recent' checkin from last year. Prior to that there were only a few checkins and they are from 1992 (Guido) - 2001 (Tim). Seems like this file is just a historical relic that can go away. n From nnorwitz at gmail.com Mon Mar 12 06:07:48 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sun, 11 Mar 2007 21:07:48 -0800 Subject: [Python-Dev] fmod.c In-Reply-To: References: Message-ID: Unless someone speaks up to save Python/fmod.c, I'll remove this file in a few days. Thanks Paul! n -- On 3/10/07, Paul Hankin wrote: > Hi all, > > Is there any need for Python/fmod.c any more? I can't see how > it can be included because there's no test for fmod in the > ./configure script and grepping all files in the tree for fmod.c > finds nothing except a commented out line in PC/os2vacpp/makefile.omk > > If it is needed, it needs fixing as it gives the wrong answer > for x>=0 and y<0. > Eg: > fmod(0, -1) returns 1 rather than 0. > > (The comment in fmod.c is correct: it should return f such that > x = i*y + f for some integer i, st |f| < |y| and f has the same > sign as x). > > I'm happy to patch it, but I won't if it would be better just > to remove the file. > > -- > Paul Hankin > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com > From talin at acm.org Mon Mar 12 06:56:03 2007 From: talin at acm.org (Talin) Date: Sun, 11 Mar 2007 22:56:03 -0700 Subject: [Python-Dev] pydoc II In-Reply-To: <27d1e6020703110746x4d85b2ffq8b0c6029fe5af29e@mail.gmail.com> References: <27d1e6020703110746x4d85b2ffq8b0c6029fe5af29e@mail.gmail.com> Message-ID: <45F4EB73.4010704@acm.org> Laurent Gautier wrote: > - low-energy barrier for adding support for "other-than-plain-text" docstrings I'd be interested in discussing this offline (I might have already spoken to you about it before, but I don't remember who I spoke to.) I think I've mentioned before about "DocLobster", which is my unpublished prototype of a "subtle" markup language that tries to embed semantic tags in the text, without causing the text to look like it has been marked up. -- Talin From Python.9.OkianWarrior at SpamGourmet.com Mon Mar 12 08:40:31 2007 From: Python.9.OkianWarrior at SpamGourmet.com (Rajstennaj Barrabas) Date: Mon, 12 Mar 2007 07:40:31 +0000 (UTC) Subject: [Python-Dev] Python version incorrect on website? Message-ID: I have an application which runs on 2.4.4 and is known not to run on 2.4.1 or 2.5, and I'm trying to install 2.4.4 from the website. Whenever I download and compile the 2.4.4 sources, the executable shows up as 2.4.1 which will not work for my application. I've downloaded both the bzip and gzip versions, and checked the MD5 sums as posted on the website, but the 2.4.4 system still shows 2.4.1 when run. The local file "buildno" contains the number "1", and the command ./python -V returns the string 2.4.1. (Linux 2.6.8-2-686, everything is apt-get "current". I'm downloading the source tar files because apt-get doesn't have the version I need.) Is there a mismatch on the python website? I'd really like to get a copy of 2.4.4 going on my system - can someone point me to the correct sources for this? From martin at v.loewis.de Mon Mar 12 10:34:02 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Mar 2007 10:34:02 +0100 Subject: [Python-Dev] Backports of standard library modules In-Reply-To: References: Message-ID: <45F51E8A.8010500@v.loewis.de> Patrick Maupin schrieb: > Although it is often quite easy to find and download a module for use > with a prior version of Python, it would be really convenient if all > of these modules were bundled up together and available as a single > download, especially when it comes to Windows users and extension > modules. I don't know how precisely you imagine this to happen: there is clearly zero chance that a future 2.5.x release (say, 2.5.2) will include modules that are about to appear with 2.6 only. If you think of an independent release: who should produce it? I would not expect that any of the current release people are willing to take more load (I know that I wouldn't); so it clearly needs more volunteers. Given the past experience with such proposals (e.g. creation of a Jumbo distribution), I'm skeptical that volunteers will materialize (unless, of course, you are volunteering). If it is indeed you who would volunteer, I'd suggest that you don't have to go through a PEP process. Instead, just do it: bundle the modules that you want to see in earlier Python versions, and publish it. If you want to see it appear on python.org, that could certainly be arranged. Regards, Martin From martin at v.loewis.de Mon Mar 12 10:42:01 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 12 Mar 2007 10:42:01 +0100 Subject: [Python-Dev] Backports of standard library modules In-Reply-To: <20070312011956.7769.669657250.divmod.xquotient.661@joule.divmod.com> References: <20070312011956.7769.669657250.divmod.xquotient.661@joule.divmod.com> Message-ID: <45F52069.6000601@v.loewis.de> glyph at divmod.com schrieb: > This does raise an interesting question, though, since I'm about to get > into PEP authorship myself. Have I missed an official way to propose > alternatives or resurrect a languishing PEP? See PEP 1. The PEP champion is obliged to integrate feedback into the PEP, to avoid repeated discussions. If the PEP champion is unresponsive, I think that ownership should be transferred to a new champion, although PEP 1 doesn't explain how this should happen. Historically, champions who lost interest were always willing to transfer ownership. If the champion merely records negative feedback without actually changing the proposal, you can come up with a counter-proposal, as a separate PEP. For example, I wrote PEP 244 to counter PEP 236. Clearly, one of them will get rejected eventually in favor of the other. Regards, Martin From martin at v.loewis.de Mon Mar 12 10:52:52 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Mar 2007 10:52:52 +0100 Subject: [Python-Dev] Python version incorrect on website? In-Reply-To: References: Message-ID: <45F522F4.1010207@v.loewis.de> Rajstennaj Barrabas schrieb: > I have an application which runs on 2.4.4 and is known not to run on 2.4.1 or > 2.5, and I'm trying to install 2.4.4 from the website. > > Whenever I download and compile the 2.4.4 sources, the executable shows up as > 2.4.1 which will not work for my application. > > I've downloaded both the bzip and gzip versions, and checked the MD5 sums as > posted on the website, but the 2.4.4 system still shows 2.4.1 when run. The > local file "buildno" contains the number "1", and the command ./python -V > returns the string 2.4.1. > > (Linux 2.6.8-2-686, everything is apt-get "current". I'm downloading the source > tar files because apt-get doesn't have the version I need.) > > Is there a mismatch on the python website? It would have helped if you had reported what the precise URL was that you downloaded, and what precise md5sum you got. I can confirm that the official URL for Python 2.4.4 is http://www.python.org/ftp/python/2.4.4/Python-2.4.4.tar.bz2 and the official md5sum of that is 0ba90c79175c017101100ebf5978e906 Regards, Martin From mlobol at gmail.com Mon Mar 12 12:58:38 2007 From: mlobol at gmail.com (Miguel Lobo) Date: Mon, 12 Mar 2007 12:58:38 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45ED8C6A.30901@v.loewis.de> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> Message-ID: <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> Hi list, Sorry for the repetition, but since nobody has commented on patch 1644818 for about a week I thought perhaps I should ask again. Is there anything I need to do before the patch is ready for inclusion? As a remainder, this patch (which can be seen at https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1644818&group_id=5470 ) fixes built-in submodules. At the moment importing built-in submodules does not work whether the parent submodule is itself built-in, a normal module or a frozen module. This small patch fixes all three cases and has been tested to work under Windows. This is useful for example with the PyQt project, which provides a package called PyQt4 containing a number of extension modules (PyQt4.QtCore, PyQt4.QtGui, etc). It should be possible to create a bespoke Python interpreter that has these extensions statically linked, but without this patch importing these extensions fails in such an interpreter. Regards, Miguel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070312/4d5ff79b/attachment.htm From martin at v.loewis.de Mon Mar 12 13:51:12 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Mar 2007 13:51:12 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45ED8C6A.30901@v.loewis.de> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> Message-ID: <45F54CC0.8000700@v.loewis.de> > Sorry for the repetition, but since nobody has commented on patch > 1644818 for about a week I thought perhaps I should ask again. > > Is there anything I need to do before the patch is ready for inclusion? No; basically, you just need to be patient now. Do you have an urgent need to get this patch included? One issue is whether this patch should be backported to the 2.5 branch; as it is arguably a new feature, it probably should not. As 2.6 is still quite some time ahead, I can't see any urgency. Regards, Martin From mlobol at gmail.com Mon Mar 12 14:17:54 2007 From: mlobol at gmail.com (Miguel Lobo) Date: Mon, 12 Mar 2007 14:17:54 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <45F54CC0.8000700@v.loewis.de> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45ED8C6A.30901@v.loewis.de> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> Message-ID: <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> > > No; basically, you just need to be patient now. Do you have an urgent > need to get this patch included? Well, not personally, but I suspect that it is quite possible that other people will waste time trying to figure out why their imports don't work. Also, seeing that there are almost 400 open patches in the patch tracker, I'm concerned that this patch will be forgotten and left to rot, which would be a pity IMO. One issue is whether this patch should be backported to the 2.5 branch; > as it is arguably a new feature, it probably should not. As 2.6 is still > quite some time ahead, I can't see any urgency. I see this more as a fix rather than a new feature, as there is no reason or explanation in the documentation (that I've been able to find) to suggest that importing built-in submodules shouldn't work. The small changes to the code in the patch also "feel" more like a fix than like a new feature. So yes, ideally I would like to see this patch in 2.5.1, although I will of course defer to the people who are in charge of making that decision. Regards, Miguel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070312/1b212871/attachment.html From martin at v.loewis.de Mon Mar 12 15:39:03 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Mar 2007 15:39:03 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45ED8C6A.30901@v.loewis.de> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> Message-ID: <45F56607.6040000@v.loewis.de> Miguel Lobo schrieb: > Also, seeing that there are almost 400 open patches in the patch > tracker, I'm concerned that this patch will be forgotten and left to > rot, which would be a pity IMO. Of course, the submitters of the 300 other patches say the same. It is just too difficult to catch up, so yes, there is a high chance that it will rot for a while. Personally, I can't apply it as-is right now, since a) I would have to check that the test case conditionalization works fine, and b) I would have to come up with a patch for the Windows build process. This requires considerably more time than it requires to write this message. If you want to prioritize it over the other 300 patches, I can offer you a fast-track procedure if you in turn review 5 other patches. Regards, Martin From mlobol at gmail.com Mon Mar 12 15:57:59 2007 From: mlobol at gmail.com (Miguel Lobo) Date: Mon, 12 Mar 2007 15:57:59 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <45F56607.6040000@v.loewis.de> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45ED8C6A.30901@v.loewis.de> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> Message-ID: <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> > > Personally, I can't apply it as-is right now, since a) I would have to > check that the test case conditionalization works fine, and b) I would > have to come up with a patch for the Windows build process. Sorry, I couldn't understand the second point. Why would you have to patch the Windows build process? > If you want to prioritize it over the other 300 patches, I can offer > you a fast-track procedure if you in turn review 5 other patches. Well, it's not really my place to ask that this patch is prioritised above others, since as I said if I'm asking for its inclusion it is in order to benefit other Python users who may easily hit the same problem. As far as I'm concerned personally, I have no problem at all applying the patch manually. Anyway, I'm intrigued about this "review 5 other patches" procedure you suggest. What exactly would be involved in such a review? Please note that I hadn't touched CPython code before I wrote my patch and I haven't been following CPython development closely. Regards, Miguel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070312/4d0de61a/attachment.htm From jason.orendorff at gmail.com Mon Mar 12 16:41:10 2007 From: jason.orendorff at gmail.com (Jason Orendorff) Date: Mon, 12 Mar 2007 11:41:10 -0400 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45ED8C6A.30901@v.loewis.de> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> Message-ID: On 3/12/07, Miguel Lobo wrote: > Anyway, I'm intrigued about this "review 5 other patches" procedure you > suggest. What exactly would be involved in such a review? Please note that > I hadn't touched CPython code before I wrote my patch and I haven't been > following CPython development closely. Hi Miguel, This is how we suck you in... ;) You don't have to be an expert to review patches. The following procedure would qualify you: 1. Find a patch that it appears no one has ever touched (0 comments, assigned to nobody, etc.) 2. Pretty much every patch should include a unit test and documentation. If something is missing from the patch you're looking at, post a comment that says "Incomplete, no docs/tests". 3. Repeat until you've commented on five patches. If you find such clerical work beneath you, you can go further--build Python from source, apply patches, and verify that they work. It's not hard (google "python developer faq"). But it's not required. -j From mlobol at gmail.com Mon Mar 12 16:49:50 2007 From: mlobol at gmail.com (Miguel Lobo) Date: Mon, 12 Mar 2007 16:49:50 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45ED8C6A.30901@v.loewis.de> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> Message-ID: <10b800400703120849m49b13226nf02348e7728caaf7@mail.gmail.com> > > This is how we suck you in... ;) I see :-). Funny I didn't see this procedure mentioned in the patch submission guidelines ;-) You don't have to be an expert to review patches. The following > procedure would qualify you: > > 1. Find a patch that it appears no one has ever touched (0 comments, > assigned to nobody, etc.) > > 2. Pretty much every patch should include a unit test and > documentation. If something is missing from the patch you're looking > at, post a comment that says "Incomplete, no docs/tests". My own patch does not include documentation. I assume documentation would only be needed for patches that add new functionality (as opposed to fixing problems)? 3. Repeat until you've commented on five patches. > > If you find such clerical work beneath you, you can go further--build > Python from source, apply patches, and verify that they work. It's > not hard (google "python developer faq"). But it's not required. > I might try to do this, as time permits. Regards, Miguel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070312/1a59c543/attachment.html From martin at v.loewis.de Mon Mar 12 16:52:16 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Mar 2007 16:52:16 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45ED8C6A.30901@v.loewis.de> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> Message-ID: <45F57730.5040109@v.loewis.de> Miguel Lobo schrieb: > Personally, I can't apply it as-is right now, since a) I would have to > check that the test case conditionalization works fine, and b) I would > have to come up with a patch for the Windows build process. > > > Sorry, I couldn't understand the second point. Why would you have to > patch the Windows build process? I need to integrate the extra test file into a project file (probably pythoncore). > Well, it's not really my place to ask that this patch is prioritised > above others, since as I said if I'm asking for its inclusion it is in > order to benefit other Python users who may easily hit the same > problem. Yet, the same can be said for most other patches: they are all for the benefit of users running into the same respective problems. > Anyway, I'm intrigued about this "review 5 other patches" procedure you > suggest. What exactly would be involved in such a review? Please note > that I hadn't touched CPython code before I wrote my patch and I haven't > been following CPython development closely. There are a number of mostly mechanical steps that should be applied to any patch that hasn't seen these steps done yet: - is it clear what the objective of the patch is? If not, complain to the submitter that they should clarify what problem precisely they try to solve. - if so, does the patch actually achieve that objective? If not, complain to the submitter that the stated objective and the implemented change differ. To determine that, you can either * review the code, to see how it fixes the problem, or * run the code, to verify that it actually does change the problem for the cases tested In your review, state which of these approaches you've used. - As the result of the previous step, it may be that you find that the patch is out-of-date. If so, state that in the report. - Does the patch come with test suite changes, if it changes observable behavior? (irrelevant only for pure documentation changes) - Does the patch come with documentation changes (patches to the Doc directory)? These can be waived only for mere bug fixes (e.g. when the documented behavior already states what the patch achieves) - Does the patch include unnecessary changes, or combine multiple changes in a single one? That should normally not be done. When you have all this information collected, write a brief message into the report indicating what your conclusions are. Ideally, conclude with one of the following recommendations: - accept - reject - "conditional accept", i.e. needs more work, if so, state what that work would be. When you have dealt with 5 reports that way, post a summary message here what reports you've looked at, what your conclusion is, and what patch you want to see expedite processing for. Regards, Martin From facundo at taniquetil.com.ar Mon Mar 12 17:08:26 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Mon, 12 Mar 2007 16:08:26 +0000 (UTC) Subject: [Python-Dev] [ 1673007 ] urllib2 requests history + HEAD support Message-ID: This patch was posted by "koder_ua". I think that Request must have a "request type" parameters, so people can send "HEAD" requests easily. But it seems to me that keeping a request history in the module is bad, because it can easily grow up to thousands and explode (a.k.a. consume too much memory). Fo example, I have a web service, running 7x24, and opening another web service, with around 10 requests per second. This means, keeping the history (around 50bytes each request), 1.2 GB of RAM in only a month! So, I'll close this patch as "Rejected", for this reason, if anyone raises objections. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From collinw at gmail.com Mon Mar 12 17:19:26 2007 From: collinw at gmail.com (Collin Winter) Date: Mon, 12 Mar 2007 11:19:26 -0500 Subject: [Python-Dev] Fwd: Re: [Python-3000] Removing functions from the operator module In-Reply-To: <20070311122050.GA18273@code0.codespeak.net> References: <20070307201628.BAQ33047@ms09.lnh.mail.rcn.net> <43aa6ff70703072153n1fe873fapb17a6421f8d5de8e@mail.gmail.com> <20070311122050.GA18273@code0.codespeak.net> Message-ID: <43aa6ff70703120919x597bf3a3qe788aa8eed568652@mail.gmail.com> On 3/11/07, Armin Rigo wrote: > Hi Collin, > > On Wed, Mar 07, 2007 at 11:53:45PM -0600, Collin Winter wrote: > > bool() and abs() aren't syntax, so I would never look in operator. > > abs() is not syntax but bool() is part of every syntactic construction > that takes a truth value argument (if, while, and, ...) iter() is part of every syntactic construction that takes an iterator argument (for, listcomps, gencomps, ...). Should it go in operator as well? Collin Winter From martin at v.loewis.de Mon Mar 12 18:00:32 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Mar 2007 18:00:32 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <10b800400703120849m49b13226nf02348e7728caaf7@mail.gmail.com> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45ED8C6A.30901@v.loewis.de> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> <10b800400703120849m49b13226nf02348e7728caaf7@mail.gmail.com> Message-ID: <45F58730.5080903@v.loewis.de> Miguel Lobo schrieb: > My own patch does not include documentation. I assume documentation > would only be needed for patches that add new functionality (as opposed > to fixing problems)? Typically, yes. If the bug fix would also change the behavior of existing programs, that change should be documented through the \versionchanged markup. Regards, Martin From guido at python.org Mon Mar 12 18:51:34 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Mar 2007 10:51:34 -0700 Subject: [Python-Dev] Import APIs In-Reply-To: References: Message-ID: what's the situation for Py3k? Should it always use absolute import there? On 3/10/07, Georg Brandl wrote: > Currently, all C code that needs to import a module uses > PyImport_ImportModule which > (1) calls __builtin__.__import__ > (2) attempts relative imports > > Most of the time, though, at least (2) is wrong. > If we want to keep (1), PyImport_ImportModuleLevel can't be > used as a replacement. > So there should be a new API, called PyImport_ImportAbsolute > that gets a flag whether relative import should be allowed. > > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mlobol at gmail.com Mon Mar 12 19:20:56 2007 From: mlobol at gmail.com (Miguel Lobo) Date: Mon, 12 Mar 2007 19:20:56 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <45F57730.5040109@v.loewis.de> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45ED8C6A.30901@v.loewis.de> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> <45F57730.5040109@v.loewis.de> Message-ID: <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> > > I need to integrate the extra test file into a project file (probably > pythoncore). The change to pythoncore.vcproj is already in the patch I posted. Otherwise I wouldn't have been able to run my test under Windows. Yet, the same can be said for most other patches: they are all for the > benefit of users running into the same respective problems. Agreed. What I mean is that this fasttrack system where the submitter has to do some extra work seems to imply that accepting the patch somehow benefits the submitter. In fact I'm probably the person the patch will benefit least, because I have already run into the problem and know how to solve it. I feel responsible for defending the patch since I've written it and I know the problem it fixes and my solution better than anybody else, but I don't see how that responsibility extends to having to do extra unrelated work to have the patch accepted. I'm not complaining or anything, and no offence meant to anyone, just explaining my point of view. I might still try to do the 5 patch review thing, depending on how long it takes me. But if I choose not to do so, leaving my patch to rot only harms CPython, not me. Regards, Miguel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070312/e0e55ebd/attachment.html From dustin at v.igoro.us Mon Mar 12 19:35:02 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Mon, 12 Mar 2007 13:35:02 -0500 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45ED8C6A.30901@v.loewis.de> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> <45F57730.5040109@v.loewis.de> <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> Message-ID: <20070312183502.GH9602@v.igoro.us> On Mon, Mar 12, 2007 at 07:20:56PM +0100, Miguel Lobo wrote: > I'm not complaining or anything, and no offence meant to anyone, just > explaining my point of view. I might still try to do the 5 patch > review thing, depending on how long it takes me. But if I choose not > to do so, leaving my patch to rot only harms CPython, not me. Miguel, last week there was a lengthy conversation on this list on this exact topic. Your point of view, which I hold to be very common, came up a few times, but thanks for stating it so clearly! Dustin P.S. Please note I am *not* trying to re-open that conversation ;-) From guido at python.org Mon Mar 12 19:13:28 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Mar 2007 11:13:28 -0700 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: <20070312010547.400001D403E@longblack.object-craft.com.au> References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> <20070228062840.17852.238152490.divmod.quotient.727@ohm> <20070306234849.6305.1580733783.divmod.xquotient.44@joule.divmod.com> <20070312010547.400001D403E@longblack.object-craft.com.au> Message-ID: Absolutely right. I'll withdraw the lightweight version. It's done enough damage. On 3/11/07, Andrew McNamara wrote: > >I wrote two versions of the dict views refactoring. One that turns > >d.keys() into list(d.keys()) and d.iterkeys() into iter(d.keys()). > >This one is pretty robust except if you have classes that emulate > >2.x-style dicts. But it generates ugly code. So I have a > >"light-weight" version that leaves d.keys() alone, while turning > >d.iterkeys() into d.keys(). This generates prettier code but more > >buggy. I probably should have used the heavy-duty one instead. > > The ugliness is a virtue in this case as it stands out enough to motivate > developers to review each case. The pretty/efficient version is tantamount > to guessing, and effectively discards information in the transformation > ("here be dragons"). > > -- > Andrew McNamara, Senior Developer, Object Craft > http://www.object-craft.com.au/ > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mike.klaas at gmail.com Mon Mar 12 20:09:29 2007 From: mike.klaas at gmail.com (Mike Klaas) Date: Mon, 12 Mar 2007 12:09:29 -0700 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45ED8C6A.30901@v.loewis.de> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> <45F57730.5040109@v.loewis.de> <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> Message-ID: <3d2ce8cb0703121209w67afd9cey645b33911dc183b8@mail.gmail.com> On 3/12/07, Miguel Lobo wrote: > > Yet, the same can be said for most other patches: they are all for the > > benefit of users running into the same respective problems. > > Agreed. What I mean is that this fasttrack system where the submitter has > to do some extra work seems to imply that accepting the patch somehow > benefits the submitter. In fact I'm probably the person the patch will > benefit least, because I have already run into the problem and know how to > solve it. I feel responsible for defending the patch since I've written it > and I know the problem it fixes and my solution better than anybody else, > but I don't see how that responsibility extends to having to do extra > unrelated work to have the patch accepted. It is certainly not your _responsibility_ to review additional patches to get your accepted; without doing so, it likely will be accepted, eventually (assuming it is correct). As far as I understand, Martin's offer is purely a personal one: there is a patch backlog, and if you help clear it out, he will help your patch get processed faster. cheers, -Mike From martin at v.loewis.de Mon Mar 12 20:11:53 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Mar 2007 20:11:53 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45ED8C6A.30901@v.loewis.de> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> <45F57730.5040109@v.loewis.de> <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> Message-ID: <45F5A5F9.5010705@v.loewis.de> Miguel Lobo schrieb: > I need to integrate the extra test file into a project file (probably > pythoncore). > > > The change to pythoncore.vcproj is already in the patch I posted. > Otherwise I wouldn't have been able to run my test under Windows. Ah, ok, I misremembered. It is the UNIX build process that is still lacking, then. > Yet, the same can be said for most other patches: they are all for the > benefit of users running into the same respective problems. > > > Agreed. What I mean is that this fasttrack system where the submitter > has to do some extra work seems to imply that accepting the patch > somehow benefits the submitter. That's true. If there is no benefit for the submitter, the submitter will likely not use that procedure. People have various motives for contributing: scratch-your-own-itch, gain reputation, ... For people that actually do have an interest in their patch being integrated, this offer (and it is just an offer) may be interesting. > In fact I'm probably the person the > patch will benefit least, because I have already run into the problem > and know how to solve it. It won't benefit the Python core either, because we just don't use builtin submodules. In fact, I find the notion of builtin submodules somewhat strange. > I feel responsible for defending the patch > since I've written it and I know the problem it fixes and my solution > better than anybody else, but I don't see how that responsibility > extends to having to do extra unrelated work to have the patch accepted. At this point, there is relatively little that you can do, as you cannot control the priority and the time that people are willing to review and integrate patches. So you either need to be patient until some reviewer picks it up, or you need to involve yourself further into Python. > I'm not complaining or anything, and no offence meant to anyone, just > explaining my point of view. I might still try to do the 5 patch review > thing, depending on how long it takes me. But if I choose not to do so, > leaving my patch to rot only harms CPython, not me. Understood. Regards, Martin From g.brandl at gmx.net Mon Mar 12 20:19:28 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 12 Mar 2007 20:19:28 +0100 Subject: [Python-Dev] Import APIs In-Reply-To: References: Message-ID: IMHO yes, for all occurences in the core code. Guido van Rossum schrieb: > what's the situation for Py3k? Should it always use absolute import there? > > On 3/10/07, Georg Brandl wrote: >> Currently, all C code that needs to import a module uses >> PyImport_ImportModule which >> (1) calls __builtin__.__import__ >> (2) attempts relative imports >> >> Most of the time, though, at least (2) is wrong. >> If we want to keep (1), PyImport_ImportModuleLevel can't be >> used as a replacement. >> So there should be a new API, called PyImport_ImportAbsolute >> that gets a flag whether relative import should be allowed. >> >> Georg From mlobol at gmail.com Mon Mar 12 20:24:30 2007 From: mlobol at gmail.com (Miguel Lobo) Date: Mon, 12 Mar 2007 20:24:30 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <45F5A5F9.5010705@v.loewis.de> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> <45F57730.5040109@v.loewis.de> <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> <45F5A5F9.5010705@v.loewis.de> Message-ID: <10b800400703121224h5dd4cf97w7811f50df1b4a119@mail.gmail.com> > > It won't benefit the Python core either, because we just don't use > builtin submodules. In fact, I find the notion of builtin submodules > somewhat strange. Please excuse my curiosity, but why do you find it strange? P.S. Thanks to all for the considerate responses. Regards, Miguel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070312/4f41d013/attachment.htm From guido at python.org Mon Mar 12 20:49:03 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Mar 2007 12:49:03 -0700 Subject: [Python-Dev] Fwd: Re: [Python-3000] Removing functions from the operator module In-Reply-To: <43aa6ff70703120919x597bf3a3qe788aa8eed568652@mail.gmail.com> References: <20070307201628.BAQ33047@ms09.lnh.mail.rcn.net> <43aa6ff70703072153n1fe873fapb17a6421f8d5de8e@mail.gmail.com> <20070311122050.GA18273@code0.codespeak.net> <43aa6ff70703120919x597bf3a3qe788aa8eed568652@mail.gmail.com> Message-ID: Yes. On 3/12/07, Collin Winter wrote: > On 3/11/07, Armin Rigo wrote: > > Hi Collin, > > > > On Wed, Mar 07, 2007 at 11:53:45PM -0600, Collin Winter wrote: > > > bool() and abs() aren't syntax, so I would never look in operator. > > > > abs() is not syntax but bool() is part of every syntactic construction > > that takes a truth value argument (if, while, and, ...) > > iter() is part of every syntactic construction that takes an iterator > argument (for, listcomps, gencomps, ...). Should it go in operator as > well? > > Collin Winter > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From arigo at tunes.org Mon Mar 12 21:17:24 2007 From: arigo at tunes.org (Armin Rigo) Date: Mon, 12 Mar 2007 21:17:24 +0100 Subject: [Python-Dev] Fwd: Re: [Python-3000] Removing functions from the operator module In-Reply-To: <43aa6ff70703120919x597bf3a3qe788aa8eed568652@mail.gmail.com> References: <20070307201628.BAQ33047@ms09.lnh.mail.rcn.net> <43aa6ff70703072153n1fe873fapb17a6421f8d5de8e@mail.gmail.com> <20070311122050.GA18273@code0.codespeak.net> <43aa6ff70703120919x597bf3a3qe788aa8eed568652@mail.gmail.com> Message-ID: <20070312201724.GA2464@code0.codespeak.net> Hi Collin, On Mon, Mar 12, 2007 at 11:19:26AM -0500, Collin Winter wrote: > iter() is part of every syntactic construction that takes an iterator > argument (for, listcomps, gencomps, ...). Should it go in operator as > well? Historically, things that have a slot go in 'operator'. So that would mean that 'operator' is missing at least: getattr, setattr, delattr, next, repr, str, len, hash, hex, oct, divmod, int, float, long, descr_get, descr_set, descr_delete, getslice, setslice, delslice, call, init. Or else, things in 'operator' are the ones with a "syntactic equivalent". Then we should remove abs and index, and add list, tuple, dict, buildclass, new.function, ... Then there is stuff that doesn't fall in any category, like attrgetter, countOf, indexOf, ... Not-completely-useful'y-yours, Armin From collinw at gmail.com Mon Mar 12 22:09:00 2007 From: collinw at gmail.com (Collin Winter) Date: Mon, 12 Mar 2007 16:09:00 -0500 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: <20070306234849.6305.1580733783.divmod.xquotient.44@joule.divmod.com> References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> <20070228062840.17852.238152490.divmod.quotient.727@ohm> <20070306234849.6305.1580733783.divmod.xquotient.44@joule.divmod.com> Message-ID: <43aa6ff70703121409m176c1480n7d8402453ac9a1c@mail.gmail.com> On 3/6/07, glyph at divmod.com wrote: > On 10:22 pm, guido at python.org wrote: [snip] > >I'm hoping Collin will continue his excellent work on 2to3. Hopefully > >he'll get help from others in writing docs aimed at teaching the > >c.l.py crowd how to use it and what to expect. > > I'm sure he'll hear from me if anything goes wrong with it :). I've started work on "Capabilities" and "Limitations" sections for the 2to3 README: http://svn.python.org/view/sandbox/trunk/2to3/README?view=markup. I intend for them to provide a comprehensive look at what 2to3 can and can't do. Collin Winter From martin at v.loewis.de Mon Mar 12 22:24:45 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Mar 2007 22:24:45 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <10b800400703121224h5dd4cf97w7811f50df1b4a119@mail.gmail.com> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> <45F57730.5040109@v.loewis.de> <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> <45F5A5F9.5010705@v.loewis.de> <10b800400703121224h5dd4cf97w7811f50df1b4a119@mail.gmail.com> Message-ID: <45F5C51D.4020403@v.loewis.de> Miguel Lobo schrieb: > It won't benefit the Python core either, because we just don't use > builtin submodules. In fact, I find the notion of builtin submodules > somewhat strange. > > > Please excuse my curiosity, but why do you find it strange? Normally, the builtin modules are the ones that are shipped in Python core. I know you can get a bigger builtins list through freeze, or through a custom Setup.local, but it is fairly uncommon to do that. Also, having extension modules in a namespace is something that I would normally not do. I couldn't tell off-hand whether having extension modules in a package would even work (but apparently, it does), so I would make them global modules just to be safe. Also, they normally have a canonical naming: if they wrap a library called libfoo, the extension module would normally be called foo.so or foomodule.so (on a Unix system; foo.pyd on Windows). As DLLs don't have hierarchical names, extension modules don't need hierarchical names, either. Putting these two boundary cases (additional builtin modules, and modules in packages) is something that I find unlikely to run into. Regards, Martin From mlobol at gmail.com Mon Mar 12 23:13:57 2007 From: mlobol at gmail.com (Miguel Lobo) Date: Mon, 12 Mar 2007 23:13:57 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <45F5C51D.4020403@v.loewis.de> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> <45F57730.5040109@v.loewis.de> <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> <45F5A5F9.5010705@v.loewis.de> <10b800400703121224h5dd4cf97w7811f50df1b4a119@mail.gmail.com> <45F5C51D.4020403@v.loewis.de> Message-ID: <10b800400703121513h22dc099bk40a83d8c470d90e0@mail.gmail.com> > > Normally, the builtin modules are the ones that are shipped in Python > core. I know you can get a bigger builtins list through freeze, or > through a custom Setup.local, but it is fairly uncommon to do that. > > Also, having extension modules in a namespace is something that I > would normally not do. I couldn't tell off-hand whether having > extension modules in a package would even work (but apparently, > it does), Well, it quite doesn't, without the patch ;-) so I would make them global modules just to be safe. > Also, they normally have a canonical naming: if they wrap a library > called libfoo, the extension module would normally be called > foo.so or foomodule.so (on a Unix system; foo.pyd on Windows). > As DLLs don't have hierarchical names, extension modules don't > need hierarchical names, either. Perhaps one example would help to clarify what I mean. I see that there is an xml.parsers.expat module, with the following content: ---- """Interface to the Expat non-validating XML parser.""" __version__ = '$Revision: 17640 $' from pyexpat import * ---- Then, there is a pyexpat.c module in Modules, which provides the contents for the aforementioned xml.parsers.expat. Wouldn't it be simpler and less invasive of the user's namespace if the Python module at xml.parsers.expat was removed, and pyexpat.c declared a module name of "xml.parsers.expat" instead? Regards, Miguel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070312/9ba3a332/attachment.html From martin at v.loewis.de Mon Mar 12 23:50:17 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Mar 2007 23:50:17 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <10b800400703121513h22dc099bk40a83d8c470d90e0@mail.gmail.com> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> <45F57730.5040109@v.loewis.de> <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> <45F5A5F9.5010705@v.loewis.de> <10b800400703121224h5dd4cf97w7811f50df1b4a119@mail.gmail.com> <45F5C51D.4020403@v.loewis.de> <10b800400703121513h22dc099bk40a83d8c470d90e0@mail.gmail.com> Message-ID: <45F5D929.6060502@v.loewis.de> Miguel Lobo schrieb: > Perhaps one example would help to clarify what I mean. I see that there > is an xml.parsers.expat module, with the following content: > > ---- > > """Interface to the Expat non-validating XML parser.""" > __version__ = '$Revision: 17640 $' > > from pyexpat import * > > ---- > > Then, there is a pyexpat.c module in Modules, which provides the > contents for the aforementioned xml.parsers.expat. > > Wouldn't it be simpler and less invasive of the user's namespace if the > Python module at xml.parsers.expat was removed, and pyexpat.c declared a > module name of "xml.parsers.expat" instead? It certainly wouldn't be simpler: it would break a lot of existing code, which all assumes that pyexpat is a module that you can import. Also, it would noticably complicate the build process, which now suddenly needs to support submodules. It wouldn't be less invasive, either: "pyexpat" is a name that is really unlikely to clash ("expat" itself would already provide that guarantee). As the former PyXML maintainer, I considered that several times, and always concluded that it should be not be done. Does distutils support this kind of setup? Modules/Setup? IOW, I would expect that there are sooo many places where extension modules in packages are not supported that it is just a tiny detail that they don't work in builtin modules (which, as I said, only have top-level modules, anyway). Regards, Martin From greg.ewing at canterbury.ac.nz Tue Mar 13 00:31:01 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 13 Mar 2007 12:31:01 +1300 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45ED8C6A.30901@v.loewis.de> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> <45F57730.5040109@v.loewis.de> <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> Message-ID: <45F5E2B5.2070105@canterbury.ac.nz> Miguel Lobo wrote: > In fact I'm probably the person the > patch will benefit least, because I have already run into the problem > and know how to solve it. For me, the personal benefit of getting a patch applied would be so that I didn't have to keep re-applying it to new versions of Python, and that I could distribute code relying on the patch to others without requiring *them* to use a patched version of Python as well. -- Greg From mlobol at gmail.com Tue Mar 13 01:12:22 2007 From: mlobol at gmail.com (Miguel Lobo) Date: Tue, 13 Mar 2007 01:12:22 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <45F5E2B5.2070105@canterbury.ac.nz> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> <45F57730.5040109@v.loewis.de> <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> <45F5E2B5.2070105@canterbury.ac.nz> Message-ID: <10b800400703121712x26900e6fsa43e87994c8c25c4@mail.gmail.com> > > For me, the personal benefit of getting a patch applied > would be so that I didn't have to keep re-applying it > to new versions of Python, and that I could distribute > code relying on the patch to others without requiring > *them* to use a patched version of Python as well. What you describe is probably fairly common, but in this particular case, the patch is only needed if you are going to build a bespoke Python interpreter. This is a complicated enough process that the difficulty of having to apply a patch is probably insignificant. The potential savings of this patch lay mainly in avoiding the waste of time for people who will face the same problem and not understand why. As anecdotal evidence, just a couple of days after I had figured out what the problem was and had the patch ready, another guy found the same problem completely independently and posted some questions to the PyQt development list about it. Regards, Miguel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070313/7e7bd337/attachment.htm From arigo at tunes.org Tue Mar 13 10:23:22 2007 From: arigo at tunes.org (Armin Rigo) Date: Tue, 13 Mar 2007 10:23:22 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <45F5D929.6060502@v.loewis.de> References: <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> <45F57730.5040109@v.loewis.de> <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> <45F5A5F9.5010705@v.loewis.de> <10b800400703121224h5dd4cf97w7811f50df1b4a119@mail.gmail.com> <45F5C51D.4020403@v.loewis.de> <10b800400703121513h22dc099bk40a83d8c470d90e0@mail.gmail.com> <45F5D929.6060502@v.loewis.de> Message-ID: <20070313092322.GA4215@code0.codespeak.net> Hi Martin, On Mon, Mar 12, 2007 at 11:50:17PM +0100, "Martin v. L?wis" wrote: > Does distutils support this kind of setup? Modules/Setup? distutils does, and I can find many projects which require a combination of C and Python modules being organized as a single package with the extension modules somewhere inside: psyco, pygame, pysqlite2, PIL, Numeric, OpenSSL... in fact, this is the case of the majority of the packages I see in my 'site-packages'. If someone wanted to distribute a Python executable with any of these packages built-in, he would run into exactly the same problem as Miguel did. A bientot, Armin From greg.ewing at canterbury.ac.nz Tue Mar 13 10:49:20 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 13 Mar 2007 22:49:20 +1300 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <45F5C51D.4020403@v.loewis.de> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <10b800400703061240l1eb8993ch55b464c0bb030cf4@mail.gmail.com> <10b800400703120458rce87b52t13870875a343585@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> <45F57730.5040109@v.loewis.de> <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> <45F5A5F9.5010705@v.loewis.de> <10b800400703121224h5dd4cf97w7811f50df1b4a119@mail.gmail.com> <45F5C51D.4020403@v.loewis.de> Message-ID: <45F673A0.3030508@canterbury.ac.nz> Martin v. L?wis wrote: > I couldn't tell off-hand whether having > extension modules in a package would even work It's quite common for a third-party package to consist of some Python code and some extension modules, with the extension modules living inside the package namespace. It works fine. -- Greg From koder.mail at gmail.com Tue Mar 13 11:40:53 2007 From: koder.mail at gmail.com (KoDer) Date: Tue, 13 Mar 2007 12:40:53 +0200 Subject: [Python-Dev] [ 1673007 ] urllib2 requests history + HEAD Message-ID: <6e8b06c90703130340p373ac8afg489c08770d48fe63@mail.gmail.com> > From: Facundo Batista > This patch was posted by "koder_ua". > I think that Request must have a "request type" parameters, so people > can send "HEAD" requests easily. > But it seems to me that keeping a request history in the module is bad, > because it can easily grow up to thousands and explode (a.k.a. consume > too much memory). > Fo example, I have a web service, running 7x24, and opening another web > service, with around 10 requests per second. This means, keeping the > history (around 50bytes each request), 1.2 GB of RAM in only a month! > So, I'll close this patch as "Rejected", for this reason, if anyone > raises objections. > Regards, > -- > . Facundo This is probably a misunderstanding. Request's history don't store in the "module".They store in two places: 1) In Request object (for current request, so they would be destroy?d with it); 2) In HTTPConnection object (while request redirects). In HTTPConnection history stores only for current served Request. Even if You use the same HTTPConnection for many Requests, they (HTTPConnection) clear history every time when new Request starts. # from httplib HTTPConnection.putrequest patched str = '%s %s %s' % (method, url, self._http_vsn_str) self._output(str) self.sended_hdrs = [str] <<< previous history die here ___Full history for all processed request didn't not stored in any place____. --- KDanilov aka koder(aka koder_ua) From jeremy.kloth at 4suite.org Tue Mar 13 17:59:08 2007 From: jeremy.kloth at 4suite.org (Jeremy Kloth) Date: Tue, 13 Mar 2007 09:59:08 -0700 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <45F5D929.6060502@v.loewis.de> References: <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <10b800400703121513h22dc099bk40a83d8c470d90e0@mail.gmail.com> <45F5D929.6060502@v.loewis.de> Message-ID: <200703131059.09134.jeremy.kloth@4suite.org> On Monday 12 March 2007 4:50:17 pm Martin v. L?wis wrote: > Does distutils support this kind of setup? Yes, and it has since its introduction. Just use the dotted name for the extension name. > IOW, I would expect that there are sooo many places where extension > modules in packages are not supported that it is just a tiny detail > that they don't work in builtin modules (which, as I said, only have > top-level modules, anyway). 4Suite has many extension modules that are part of packages and never once has there been a problem reported due to extension modules. The thought that extension modules cannot be in a package has never even cross my mind. I would have thought it a bug if they didn't, FWIW. -- Jeremy Kloth http://4suite.org/ From pje at telecommunity.com Tue Mar 13 18:24:37 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 13 Mar 2007 12:24:37 -0500 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <45F5D929.6060502@v.loewis.de> References: <10b800400703121513h22dc099bk40a83d8c470d90e0@mail.gmail.com> <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> <45F57730.5040109@v.loewis.de> <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> <45F5A5F9.5010705@v.loewis.de> <10b800400703121224h5dd4cf97w7811f50df1b4a119@mail.gmail.com> <45F5C51D.4020403@v.loewis.de> <10b800400703121513h22dc099bk40a83d8c470d90e0@mail.gmail.com> Message-ID: <5.1.1.6.0.20070313122112.0463d658@sparrow.telecommunity.com> At 11:50 PM 3/12/2007 +0100, Martin v. L?wis wrote: >Does distutils support this kind of setup? Modules/Setup? distutils does, and has from its inception, as far as I know. >IOW, I would expect that there are sooo many places where extension >modules in packages are not supported The only thing I know of that doesn't support it very well is py2exe, actually. PEAK has had extensions modules in subpackages for many years now, and never run into any problems. In fact, py2exe's only problem with it is that it still requires such modules to have globally unique names -- you can't have both a foo.speedups extension and a bar.speedups extension. (This actually happened with RuleDispatch and PyProtocols; I ended up renaming them to _p_speedups and _d_speedups to better support py2exe users.) From theller at ctypes.org Tue Mar 13 19:52:26 2007 From: theller at ctypes.org (Thomas Heller) Date: Tue, 13 Mar 2007 19:52:26 +0100 Subject: [Python-Dev] Patch 1644818: Allow importing built-in submodules In-Reply-To: <5.1.1.6.0.20070313122112.0463d658@sparrow.telecommunity.com> References: <10b800400703121513h22dc099bk40a83d8c470d90e0@mail.gmail.com> <10b800400703060628o54c809d3wd954eaaf7a14fe29@mail.gmail.com> <45F54CC0.8000700@v.loewis.de> <10b800400703120617j6d5ff3d5x1608a839923ecf90@mail.gmail.com> <45F56607.6040000@v.loewis.de> <10b800400703120757u636e1e64g68efcde17919786d@mail.gmail.com> <45F57730.5040109@v.loewis.de> <10b800400703121120w29b7a4fcm4792d90ff50845e3@mail.gmail.com> <45F5A5F9.5010705@v.loewis.de> <10b800400703121224h5dd4cf97w7811f50df1b4a119@mail.gmail.com> <45F5C51D.4020403@v.loewis.de> <10b800400703121513h22dc099bk40a83d8c470d90e0@mail.gmail.com> <45F5D929.6060502@v.loewis.de> <5.1.1.6.0.20070313122112.0463d658@sparrow.telecommunity.com> Message-ID: Phillip J. Eby schrieb: > At 11:50 PM 3/12/2007 +0100, Martin v. L?wis wrote: >>Does distutils support this kind of setup? Modules/Setup? > > distutils does, and has from its inception, as far as I know. > > >>IOW, I would expect that there are sooo many places where extension >>modules in packages are not supported > > The only thing I know of that doesn't support it very well is py2exe, > actually. PEAK has had extensions modules in subpackages for many years > now, and never run into any problems. In fact, py2exe's only problem with > it is that it still requires such modules to have globally unique names -- > you can't have both a foo.speedups extension and a bar.speedups > extension. (This actually happened with RuleDispatch and PyProtocols; I > ended up renaming them to _p_speedups and _d_speedups to better support > py2exe users.) > It is kind of funny that I contributed a patch to the McMillan installer to support extension modules in packages years ago, but never bothered to implement that in py2exe. If anyone contributes a patch for py2exe, I can integrate it... Thomas From g.brandl at gmx.net Tue Mar 13 21:25:48 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 13 Mar 2007 21:25:48 +0100 Subject: [Python-Dev] assert(obj) in Py_DECREF and Py_INCREF? Message-ID: Patch #1517947 suggests adding an assert for these. Is that acceptable or just considered a drop in ocean? Georg From g.brandl at gmx.net Tue Mar 13 21:32:47 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 13 Mar 2007 21:32:47 +0100 Subject: [Python-Dev] deprecate commands.getstatus() Message-ID: I'd like to deprecate commands.getstatus() in 2.6. Reason: there is getoutput() and getstatusoutput(). In the latter, "status" means the exit code. getstatus(), however, returns the output of "ls -ld " which is completely nonobvious and confusing. Perhaps the whole commands module can be deprecated in favor of subprocess. Georg From guido at python.org Tue Mar 13 22:16:21 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Mar 2007 14:16:21 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: Message-ID: On 3/13/07, Georg Brandl wrote: > I'd like to deprecate commands.getstatus() in 2.6. > > Reason: there is getoutput() and getstatusoutput(). In the latter, "status" > means the exit code. getstatus(), however, returns the output of "ls -ld > " which is completely nonobvious and confusing. +1. > Perhaps the whole commands module can be deprecated in favor of subprocess. -1. Reason (for both voteS): I have learned that the getoutput() and getstatusoutput() functions in the commands module are exceedingly popular amongst Googlers who write simple Python scripts that occasionally invoke an external command. It's much simpler than using os.popen() or the subprocess module (which is also very popular BTW). But I have found no uses of commands.getstatus() in our entire codebase so I'm okay with seeing that mistake finally eradicated (and it was *my* mistake :-). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Tue Mar 13 22:30:12 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Mar 2007 22:30:12 +0100 Subject: [Python-Dev] assert(obj) in Py_DECREF and Py_INCREF? In-Reply-To: References: Message-ID: <45F717E4.6020202@v.loewis.de> Georg Brandl schrieb: > Patch #1517947 suggests adding an assert for these. > > Is that acceptable or just considered a drop in ocean? I think it's pointless. If they ever be NULL, the INCRE/DECREF will crash right away, anyway, so you'll certainly notice. As you will have a debug build when the assertion triggers, you can just as well fine the location of the crash quickly in the debugger (e.g. using the core file on systems that support that, or the JIT-debugger on other systems). Regards, Martin From g.brandl at gmx.net Tue Mar 13 22:32:15 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 13 Mar 2007 22:32:15 +0100 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: Message-ID: Guido van Rossum schrieb: > On 3/13/07, Georg Brandl wrote: >> I'd like to deprecate commands.getstatus() in 2.6. >> >> Reason: there is getoutput() and getstatusoutput(). In the latter, "status" >> means the exit code. getstatus(), however, returns the output of "ls -ld >> " which is completely nonobvious and confusing. > > +1. Done in rev. 54361. Georg From guido at python.org Wed Mar 14 00:55:07 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Mar 2007 16:55:07 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <20070313223629.GA27802@caltech.edu> References: <20070313223629.GA27802@caltech.edu> Message-ID: On 3/13/07, Titus Brown wrote: > What about reimplementing commands.* using subprocess? Or providing a > commands.*-compatible interface in the subprocess module? What does that buy us? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mal at egenix.com Wed Mar 14 12:11:50 2007 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 14 Mar 2007 12:11:50 +0100 Subject: [Python-Dev] These csv test cases seem incorrect to me... In-Reply-To: <17908.46180.251850.822433@montanaro.dyndns.org> References: <17908.46180.251850.822433@montanaro.dyndns.org> Message-ID: <45F7D876.3010003@egenix.com> Hi Skip, On 2007-03-12 03:01, skip at pobox.com wrote: > I decided it would be worthwhile to have a csv module written in Python (no > C underpinnings) for a number of reasons: > > * It will probably be easier to add Unicode support to a Python version > > * More people will be able to read/grok/modify/fix bugs in a Python > implementation than in the current mixed Python/C implementation. > > * With alternative implementations of Python available (PyPy, > IronPython, Jython) it makes sense to have a Python version they can > use. Lots of good reasons :-) I've written a Python-only Unicode aware CSV module for a client (mostly because CSV data tends to be quirky and I needed a quick way of dealing with corner cases). Perhaps I can get them to donate it to the PSF... > I'm far from having anything which will pass the current test suite, but in > diagnosing some of my current failures I noticed a couple test cases which > seem wrong. In the TestDialectExcel class I see these two questionable > tests: > > def test_quotes_and_more(self): > self.readerAssertEqual('"a"b', [['ab']]) > > def test_quote_and_quote(self): > self.readerAssertEqual('"a" "b"', [['a "b"']]) > > It seems to me that if a field starts with a quote it *has* to be a quoted > field. Any quotes appearing within a quoted field have to be escaped and > the field has to end with a quote. Both of these test cases fail on or the > other assumption. If they are indeed both correct and I'm just looking at > things crosseyed I think they at least deserve comments explaining why they > are correct. > > Both test cases date from the first checkin. I performed the checkin > because of the group developing the module I believe I was the only one with > checkin privileges at the time, not because I wrote the test cases. > > Any ideas about why these test cases are in there? I can't imagine Excel > generating either one. My recommendation: Let the module do whatever Excel does with such data. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 14 2007) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: From amk at amk.ca Wed Mar 14 13:47:31 2007 From: amk at amk.ca (A.M. Kuchling) Date: Wed, 14 Mar 2007 08:47:31 -0400 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070313223629.GA27802@caltech.edu> Message-ID: <20070314124731.GA5052@localhost.localdomain> On Tue, Mar 13, 2007 at 04:55:07PM -0700, Guido van Rossum wrote: > What does that buy us? subprocess offers better error-trapping, I think, and additional features such as closing all file descriptors after forking. At work, I found this fixed a number of bugs in our daemons code because if you were starting something long-running, it might keep a socket open and cause problems later. --amk From fredrik at pythonware.com Wed Mar 14 15:33:23 2007 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 14 Mar 2007 15:33:23 +0100 Subject: [Python-Dev] deprecate commands.getstatus() References: <20070313223629.GA27802@caltech.edu> Message-ID: Guido van Rossum wrote: >> What about reimplementing commands.* using subprocess? Or providing a >> commands.*-compatible interface in the subprocess module? > > What does that buy us? multi-platform support? (commands is Unixoid only, right?) From guido at python.org Wed Mar 14 15:48:31 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 14 Mar 2007 06:48:31 -0800 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070313223629.GA27802@caltech.edu> Message-ID: Alright already! It wasn't a thetorical question, and I'm convinced! :-) On 3/14/07, Fredrik Lundh wrote: > Guido van Rossum wrote: > > >> What about reimplementing commands.* using subprocess? Or providing a > >> commands.*-compatible interface in the subprocess module? > > > > What does that buy us? > > multi-platform support? (commands is Unixoid only, right?) > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From yinyang at eburg.com Wed Mar 14 17:34:41 2007 From: yinyang at eburg.com (Gordon Messmer) Date: Wed, 14 Mar 2007 09:34:41 -0700 Subject: [Python-Dev] thread safe SMTP module Message-ID: <45F82421.1080602@eburg.com> A couple of weeks ago, I posted a message titled "thread safe SMTP module" on python-list. It's oddly split in the archive: http://mail.python.org/pipermail/python-list/2007-March/429067.html http://mail.python.org/pipermail/python-list/2007-March/429172.html A while ago, I wrote an address validation function using smtplib.SMTP for use in a a threaded application. In practice, I found that the application locked up rather frequently. I scratched my head for a while before concluding that the caveat mentioned in the thread module's documentation, noting that some built-in functions may block all threads, was probably at fault. I then subclassed smtplib.SMTP and replaced all of the blocking I/O operations with nonblocking ones, and the function began working properly. After some discussion, Aahz suggested that I discuss the problem here, on python-dev. He seemed to think that the problem I saw may have been an indication of a bug in python. Could anyone take a look at that thread and say whether it looks like a bug, or working with non-blocking sockets was the right thing to do? From pje at telecommunity.com Wed Mar 14 18:08:09 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 14 Mar 2007 12:08:09 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) Message-ID: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> In addition to being made in the face of controversy and opposition, this change is an alteration to *documented and tested* behavior and thus cannot reasonably be considered a mere bug fix. In addition to the prior behavior being explicitly documented in the function docstrings, r54204 shows that it was also *tested* as behaving that way by test_macpath, test_ntpath, and test_posixpath. When combined with the explicit docstrings, this must be considered incontrovertible proof that the previous behavior was either explicitly intended, or at the very least a known, expected, and *accepted* behavior. This backwards-incompatible change is therefore contrary to policy and should be reverted, pending a proper transition plan for the change (such as introduction of an alternative API and deprecation of the existing one.) Some relevant links: http://svn.python.org/view?rev=54204&view=rev http://mail.python.org/pipermail/python-dev/2007-March/071762.html http://mail.python.org/pipermail/python-dev/2007-March/071798.html From martin at v.loewis.de Wed Mar 14 18:47:33 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Mar 2007 18:47:33 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> Message-ID: <45F83535.9080902@v.loewis.de> Phillip J. Eby schrieb: > This backwards-incompatible change is therefore contrary to policy and > should be reverted, pending a proper transition plan for the change > (such as introduction of an alternative API and deprecation of the > existing one.) I'm clearly opposed to this proposal, or else I wouldn't have committed the change in the first place. Regards, Martin From jon+python-dev at unequivocal.co.uk Wed Mar 14 18:48:56 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Wed, 14 Mar 2007 17:48:56 +0000 Subject: [Python-Dev] thread safe SMTP module In-Reply-To: <45F82421.1080602@eburg.com> References: <45F82421.1080602@eburg.com> Message-ID: <20070314174856.GB27107@snowy.squish.net> Gordon Messmer wrote: > After some discussion, Aahz suggested that I discuss the problem here, > on python-dev. He seemed to think that the problem I saw may have been > an indication of a bug in python. Could anyone take a look at that > thread and say whether it looks like a bug, or working with non-blocking > sockets was the right thing to do? Well, not having to faff around with non-blocking IO is one of the major advantages of threading, so if blocking IO is indeed the cause of your problem then it certainly sounds like a bug in Python, or possibly your operating system. From aahz at pythoncraft.com Wed Mar 14 19:26:20 2007 From: aahz at pythoncraft.com (Aahz) Date: Wed, 14 Mar 2007 11:26:20 -0700 Subject: [Python-Dev] thread safe SMTP module In-Reply-To: <20070314174856.GB27107@snowy.squish.net> References: <45F82421.1080602@eburg.com> <20070314174856.GB27107@snowy.squish.net> Message-ID: <20070314182620.GA19184@panix.com> On Wed, Mar 14, 2007, Jon Ribbens wrote: > Gordon Messmer wrote: >> >> After some discussion, Aahz suggested that I discuss the problem here, >> on python-dev. He seemed to think that the problem I saw may have been >> an indication of a bug in python. Could anyone take a look at that >> thread and say whether it looks like a bug, or working with non-blocking >> sockets was the right thing to do? > > Well, not having to faff around with non-blocking IO is one of the > major advantages of threading, so if blocking IO is indeed the cause > of your problem then it certainly sounds like a bug in Python, or > possibly your operating system. One small wrinkle (and the reason I suggested bringing this to python-dev): I suspect that the problem is not a bug, but simply the occasional failure of sockets. When that happens in a threaded app without timeouts, eventually threads "die" (block forever). But you apparently can't use timeouts with file-wrapped sockets, so if that's the problem, we need to pick one of: * switch to non-wrapped sockets * switch to non-blocking I/O for smtplib * make file-wrapped sockets work with timeouts (And please note that if my analysis is correct, we need to make the fix for non-threaded apps, it's just more unusual to encounter the problem there.) This level of design decision is beyond my capabilities. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "I disrespectfully agree." --SJM From jcarlson at uci.edu Wed Mar 14 19:36:52 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 14 Mar 2007 11:36:52 -0700 Subject: [Python-Dev] thread safe SMTP module In-Reply-To: <45F82421.1080602@eburg.com> References: <45F82421.1080602@eburg.com> Message-ID: <20070314112305.FBA6.JCARLSON@uci.edu> Gordon Messmer wrote: > After some discussion, Aahz suggested that I discuss the problem here, > on python-dev. He seemed to think that the problem I saw may have been > an indication of a bug in python. Could anyone take a look at that > thread and say whether it looks like a bug, or working with non-blocking > sockets was the right thing to do? His most recent message in that thread actually said, "No, at this point I think this is neither bug nor about thread blocking on I/O. I think it's about sockets dying and the inability of sockets in blocking mode to recover. I have seen this kind of behavior in single-threaded systems." I would concur with Aahz, more or less. I have also seen this behavior on single-threaded blocking systems. The real issue may be related to blocking sockets breaking coupled with socket.makefile(...).readline() not being able to handle the breakage. If you can, try to dig into the file object implementation returned by socket.makefile(). If you don't want to (I would understand), try replacing smtplib's use of 'self.file.readline()' with a Python variant that handles failures more gracefully. Heck, you could even add timeouts (which are being added to httplib and either urllib or urllib2). - Josiah As an aside, for even minimally loaded systems, I've noticed that file.readline() isn't any faster than just using a while loop with socket.recv() and checking for '\n'. I believe that it would make sense to replace certain libraries' (poplib, smtplib, etc.) use of file.readline() with that of a _readline() method that has a default implementation using either a file.readline() or sock.recv() in a loop. By pulling it out into a single method, the user could easily override it if it isn't doing what they want/expect. From jon+python-dev at unequivocal.co.uk Wed Mar 14 19:40:04 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Wed, 14 Mar 2007 18:40:04 +0000 Subject: [Python-Dev] thread safe SMTP module In-Reply-To: <20070314182620.GA19184@panix.com> References: <45F82421.1080602@eburg.com> <20070314174856.GB27107@snowy.squish.net> <20070314182620.GA19184@panix.com> Message-ID: <20070314184004.GC27107@snowy.squish.net> Aahz wrote: > One small wrinkle (and the reason I suggested bringing this to > python-dev): I suspect that the problem is not a bug, but simply the > occasional failure of sockets. When that happens in a threaded app > without timeouts, eventually threads "die" (block forever). But you > apparently can't use timeouts with file-wrapped sockets, so if that's the > problem, we need to pick one of: > > * switch to non-wrapped sockets > * switch to non-blocking I/O for smtplib > * make file-wrapped sockets work with timeouts In case it's helpful, I've put the file-like-sockets-with-timeouts code that I wrote up here: http://www.unequivocal.co.uk/dl/socketutils.py This is a pure-Python implementation that provides a file-like interface to a socket, with timeouts, that definitely works with threaded programs. Note it was written in 2002, so any part of the Python 'file' interface which was added since then is not in there. From yinyang at eburg.com Wed Mar 14 21:14:23 2007 From: yinyang at eburg.com (Gordon Messmer) Date: Wed, 14 Mar 2007 13:14:23 -0700 Subject: [Python-Dev] thread safe SMTP module In-Reply-To: <20070314182620.GA19184@panix.com> References: <45F82421.1080602@eburg.com> <20070314174856.GB27107@snowy.squish.net> <20070314182620.GA19184@panix.com> Message-ID: <45F8579F.7060104@eburg.com> Aahz wrote: > > One small wrinkle (and the reason I suggested bringing this to > python-dev): I suspect that the problem is not a bug, but simply the > occasional failure of sockets. When that happens in a threaded app > without timeouts, eventually threads "die" (block forever). IIRC, my whole process was locking up, not just individual threads. Tonight I should have time to pull an old copy of the code out of CVS and recreate the test script that I used. Once I have, it should be a matter of feeding a big list of email addresses to the script and waiting a couple of minutes for the script to lock up. The last time I looked, the problem was very easy to observe. From pje at telecommunity.com Wed Mar 14 21:35:31 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 14 Mar 2007 15:35:31 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F83535.9080902@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> At 06:47 PM 3/14/2007 +0100, Martin v. L?wis wrote: >Phillip J. Eby schrieb: > > This backwards-incompatible change is therefore contrary to policy and > > should be reverted, pending a proper transition plan for the change > > (such as introduction of an alternative API and deprecation of the > > existing one.) > >I'm clearly opposed to this proposal, or else I wouldn't have committed >the change in the first place. That much is obvious. But I haven't seen any explanation as to why explicitly-documented and explicitly-tested behavior should be treated as a bug in policy terms, just because people don't like the documented and tested behavior. So far, the only policy justification I've seen you give was along the lines of, "I volunteered to do it, so I get to decide". If this statement were actually a valid policy, then I suppose I could simply volunteer to decide to revert the change. But if it's *not* a valid policy, then there is no policy justification for the change, and therefore it should be reverted. Thus, either way, there needs to be some *other* justification for the original change. From fuzzyman at voidspace.org.uk Wed Mar 14 22:14:14 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Mar 2007 21:14:14 +0000 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> Message-ID: <45F865A6.8080903@voidspace.org.uk> Phillip J. Eby wrote: > At 06:47 PM 3/14/2007 +0100, Martin v. L?wis wrote: > >> Phillip J. Eby schrieb: >> >>> This backwards-incompatible change is therefore contrary to policy and >>> should be reverted, pending a proper transition plan for the change >>> (such as introduction of an alternative API and deprecation of the >>> existing one.) >>> >> I'm clearly opposed to this proposal, or else I wouldn't have committed >> the change in the first place. >> > > That much is obvious. But I haven't seen any explanation as to why > explicitly-documented and explicitly-tested behavior should be treated as a > bug in policy terms, just because people don't like the documented and > tested behavior. > > Because it's clearly a bug and has even been shown to fix bugs in current code ? Honestly it is this sort of pointless prevarication that gives python-dev a bad name. Michael Foord From thomas at python.org Wed Mar 14 22:27:34 2007 From: thomas at python.org (Thomas Wouters) Date: Wed, 14 Mar 2007 22:27:34 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F865A6.8080903@voidspace.org.uk> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <45F865A6.8080903@voidspace.org.uk> Message-ID: <9e804ac0703141427s51b3caa3n16c316ab4d37cbd3@mail.gmail.com> On 3/14/07, Michael Foord wrote: > > Phillip J. Eby wrote: > > At 06:47 PM 3/14/2007 +0100, Martin v. L?wis wrote: > > > >> Phillip J. Eby schrieb: > >> > >>> This backwards-incompatible change is therefore contrary to policy and > >>> should be reverted, pending a proper transition plan for the change > >>> (such as introduction of an alternative API and deprecation of the > >>> existing one.) > >>> > >> I'm clearly opposed to this proposal, or else I wouldn't have committed > >> the change in the first place. > >> > > > > That much is obvious. But I haven't seen any explanation as to why > > explicitly-documented and explicitly-tested behavior should be treated > as a > > bug in policy terms, just because people don't like the documented and > > tested behavior. > > > > > Because it's clearly a bug and has even been shown to fix bugs in > current code ? > > Honestly it is this sort of pointless prevarication that gives > python-dev a bad name. However, changing documented, tested behaviour without warning gives Python an even worse name. I agree with PJE that the change is the wrong thing to do, simply because it sets (yet another) precedent. If providing an alternate API with clearer semantics is too 'heavy-weight' a solution and warning is for some reason unacceptable (I don't see why; all the arguments against warning there go for *any* warning in Python) -- then the problem isn't bad enough to fix it by breaking other code. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070314/76d32d75/attachment.html From tdelaney at avaya.com Wed Mar 14 22:30:17 2007 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Thu, 15 Mar 2007 08:30:17 +1100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1ED15@au3010avexu1.global.avaya.com> Phillip J. Eby wrote: > In addition to being made in the face of controversy and opposition, > this > change is an alteration to *documented and tested* behavior and thus > cannot reasonably be considered a mere bug fix. FWIW, I support Phillip on this. There can be no question that the old behaviour was expected. IMO this is just gratuitous breakage. The only fix that shold be made is to the splitext documentation to match the docstring. A change to the documented behaviour should require a __future__ import for at least one version. That's even assuming that the change is desireable (I don't believe so). We have multiple anecdotes of actual, existing code that *will* break with this change. So far I haven't seen any actual code posted that is currently broken by the existing behaviour. Tim Delaney From pje at telecommunity.com Wed Mar 14 22:39:16 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 14 Mar 2007 16:39:16 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5FF1ED15@au3010avexu1.global .avaya.com> Message-ID: <5.1.1.6.0.20070314163328.047fb9d8@sparrow.telecommunity.com> At 08:30 AM 3/15/2007 +1100, Delaney, Timothy (Tim) wrote: >Phillip J. Eby wrote: > > > In addition to being made in the face of controversy and opposition, > > this > > change is an alteration to *documented and tested* behavior and thus > > cannot reasonably be considered a mere bug fix. > >FWIW, I support Phillip on this. There can be no question that the old >behaviour was expected. > >IMO this is just gratuitous breakage. The only fix that shold be made is >to the splitext documentation to match the docstring. > >A change to the documented behaviour should require a __future__ import >for at least one version. That's even assuming that the change is >desireable (I don't believe so). We have multiple anecdotes of actual, >existing code that *will* break with this change. So far I haven't seen >any actual code posted that is currently broken by the existing behaviour. FWIW, I think that, were we writing splitext() *now*, we should go with the proposed behavior. It's reasonable and justifiable even on Windows (even though Windows Explorer agrees with the current splitext() behavior.) But, that doesn't actually have any bearing on the current discussion, since splitext()'s behavior is existing and documented. Certainly, there *is* code that's broken by the existing behavior -- otherwise the patch would never have been submitted in the first place. However, that doesn't automatically make it a Python bug, especially if the existing behavior is documented and covered by regression tests. I just want to clarify this point, because I don't wish to enter another round of discussion about the merits of one behavior or the other: the merits one way or the other are pretty much irrelevant to the policy issue at hand. From fuzzyman at voidspace.org.uk Wed Mar 14 22:41:36 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Mar 2007 21:41:36 +0000 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5FF1ED15@au3010avexu1.global.avaya.com> References: <2773CAC687FD5F4689F526998C7E4E5FF1ED15@au3010avexu1.global.avaya.com> Message-ID: <45F86C10.4050801@voidspace.org.uk> Delaney, Timothy (Tim) wrote: > Phillip J. Eby wrote: > > >> In addition to being made in the face of controversy and opposition, >> this >> change is an alteration to *documented and tested* behavior and thus >> cannot reasonably be considered a mere bug fix. >> > > FWIW, I support Phillip on this. There can be no question that the old behaviour was expected. > > Expected ? It's perverse. :-) > IMO this is just gratuitous breakage. The only fix that shold be made is to the splitext documentation to match the docstring. > > Agreed. > A change to the documented behaviour should require a __future__ import for at least one version. That's even assuming that the change is desireable (I don't believe so). We have multiple anecdotes of actual, existing code that *will* break with this change. So far I haven't seen any actual code posted that is currently broken by the existing behaviour. > There was code posted that used the (almost entirely sane) pattern : new_filename = os.path.splitext(old_filename)[1] + '.bak' That was broken but is now fixed. It follows the entirely natural assumption that filename without an extension would not have the filename put in the extension half of the tuple. The documentation (not the docstring) actually says : splitext( path) Split the pathname path into a pair (root, ext) such that root + ext == path, and ext is empty or begins with a period and contains at most one period. Even the docstring only states that either part may be empty, hardly documenting what is clearly a misfeature. Michael Foord > Tim Delaney > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > From fuzzyman at voidspace.org.uk Wed Mar 14 22:45:15 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Mar 2007 21:45:15 +0000 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070314163328.047fb9d8@sparrow.telecommunity.com> References: <5.1.1.6.0.20070314163328.047fb9d8@sparrow.telecommunity.com> Message-ID: <45F86CEB.3010507@voidspace.org.uk> Phillip J. Eby wrote: > At 08:30 AM 3/15/2007 +1100, Delaney, Timothy (Tim) wrote: > >> Phillip J. Eby wrote: >> >> >>> In addition to being made in the face of controversy and opposition, >>> this >>> change is an alteration to *documented and tested* behavior and thus >>> cannot reasonably be considered a mere bug fix. >>> >> FWIW, I support Phillip on this. There can be no question that the old >> behaviour was expected. >> >> IMO this is just gratuitous breakage. The only fix that shold be made is >> to the splitext documentation to match the docstring. >> >> A change to the documented behaviour should require a __future__ import >> for at least one version. That's even assuming that the change is >> desireable (I don't believe so). We have multiple anecdotes of actual, >> existing code that *will* break with this change. So far I haven't seen >> any actual code posted that is currently broken by the existing behaviour. >> > > FWIW, I think that, were we writing splitext() *now*, we should go with the > proposed behavior. It's reasonable and justifiable even on Windows (even > though Windows Explorer agrees with the current splitext() behavior.) > > But, that doesn't actually have any bearing on the current discussion, > since splitext()'s behavior is existing and documented. > > Certainly, there *is* code that's broken by the existing behavior -- > otherwise the patch would never have been submitted in the first > place. However, that doesn't automatically make it a Python bug, > especially if the existing behavior is documented and covered by regression > tests. > > I just want to clarify this point, because I don't wish to enter another > round of discussion about the merits of one behavior or the other: the > merits one way or the other are pretty much irrelevant to the policy issue > at hand. > It looks to me like a clear bugfix (the fact that there were unit tests for the insane behaviour doesn't make it any less a bug). The current docstring that states that the first element may be empty hardly counts as it being a 'documented feature'. At the least it is a grey area and not a policy reversal. The policy is that bugfixes can go in with warnings. So, as a debatable issue whether it is a bug (I think it is fairly clear), then it doesn't change or contravene policy. Michael Foord > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > From tdelaney at avaya.com Wed Mar 14 22:47:15 2007 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Thu, 15 Mar 2007 08:47:15 +1100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1ED18@au3010avexu1.global.avaya.com> Michael Foord wrote: > There was code posted that used the (almost entirely sane) pattern : > > new_filename = os.path.splitext(old_filename)[1] + '.bak' > > That was broken but is now fixed. It follows the entirely natural > assumption that filename without an extension would not have the > filename put in the extension half of the tuple. Well, I'd argue the sanity of it, but you're right - it was posted. > The documentation (not the docstring) actually says : > > splitext( path) > > Split the pathname path into a pair (root, ext) such that root + > ext == path, and ext is empty or begins with a period and contains at > most one period. > > Even the docstring only states that either part may be empty, hardly > documenting what is clearly a misfeature. splitext(p) Split the extension from a pathname. Extension is everything from the last dot to the end. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Return (root, ext), either part may be empty. That's pretty explicit. Tim Delaney From fuzzyman at voidspace.org.uk Wed Mar 14 22:51:29 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Mar 2007 21:51:29 +0000 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5FF1ED18@au3010avexu1.global.avaya.com> References: <2773CAC687FD5F4689F526998C7E4E5FF1ED18@au3010avexu1.global.avaya.com> Message-ID: <45F86E61.2010902@voidspace.org.uk> Delaney, Timothy (Tim) wrote: > [snip..] >> >> Even the docstring only states that either part may be empty, hardly >> documenting what is clearly a misfeature. >> > > splitext(p) > Split the extension from a pathname. > > Extension is everything from the last dot to the end. > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > Return (root, ext), either part may be empty. > > That's pretty explicit. > > Hmm... mis-memory on my part. Apologies. Michael > Tim Delaney > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > From martin at v.loewis.de Wed Mar 14 22:54:41 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Mar 2007 22:54:41 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> Message-ID: <45F86F21.8030904@v.loewis.de> Phillip J. Eby schrieb: > That much is obvious. But I haven't seen any explanation as to why > explicitly-documented and explicitly-tested behavior should be treated > as a bug in policy terms, just because people don't like the documented > and tested behavior. It's not "just" that people disliked the behavior. The majority of those that commented agreed that the current behavior is incorrect. Some also observed that the online documentation was underspecified, and indeed allowed for that change. So this is a bug fix, even though the old test case explicitly tested for the presence of the bug, and even though the doc string explicitly documented the old behavior. They were all consistent, but they were consistently wrong. The bug fix may also break existing applications. Hence it cannot go into 2.5.1 but has to wait for 2.6. > So far, the only policy justification I've seen you give was along the > lines of, "I volunteered to do it, so I get to decide". It's more than that. I conducted a poll, and here people were largely in favor of that change. Had they been largely in opposition, I would have rejected the patch. However, *some* action is necessary. The patch was sitting there for a long time already, and it is unfair to the submitter to not act just because you cannot decide. The bug report was even older. > Thus, either way, there needs to be some *other* justification for the > original change. But I said that many times. I changed it because the old behavior (and the old documentation) was buggy, and the (revised version of) the patch fixed that bug. Regards, Martin P.S. If you apply the same effort to all changes that are constantly being made to Python, you will find that you will need to revert many of them. From pmaupin at gmail.com Wed Mar 14 23:02:25 2007 From: pmaupin at gmail.com (Patrick Maupin) Date: Wed, 14 Mar 2007 17:02:25 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> Message-ID: On 3/14/07, Phillip J. Eby wrote: > This backwards-incompatible change is therefore contrary to policy and > should be reverted, pending a proper transition plan for the change (such > as introduction of an alternative API and deprecation of the existing one.) I think the original behavior is simpler, thus easier to document and understand, but even if we want to change the behavior, the fact is that the new behavior can break some old code. As I mentioned in an earlier email, I don't think this is a corner case -- I bet there is a ton of Python code out there which deals with hidden files and directories. Pat From martin at v.loewis.de Wed Mar 14 23:04:10 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 14 Mar 2007 23:04:10 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <9e804ac0703141427s51b3caa3n16c316ab4d37cbd3@mail.gmail.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <45F865A6.8080903@voidspace.org.uk> <9e804ac0703141427s51b3caa3n16c316ab4d37cbd3@mail.gmail.com> Message-ID: <45F8715A.8000208@v.loewis.de> Thomas Wouters schrieb: > However, changing documented, tested behaviour without warning gives > Python an even worse name. I agree with PJE that the change is the wrong > thing to do, simply because it sets (yet another) precedent. If > providing an alternate API with clearer semantics is too 'heavy-weight' > a solution and warning is for some reason unacceptable (I don't see why; > all the arguments against warning there go for *any* warning in Python) > -- then the problem isn't bad enough to fix it by breaking other code. I think producing pointless warnings also gives Python a bad name (I've seen many complaints about Python's warnings in the past, in particular when they fill up Apache log files). However, if everybody (and here I mean everybody) can agree that adding a warning to the current implementation would be an acceptable compromise, I could agree to such a compromise also (although I would prefer if somebody else took the blame for adding that warning. I happily take the blame for changing the behavior). What specific warning would you propose, and in what specific circumstance would it be issued? Regards, Martin From martin at v.loewis.de Wed Mar 14 23:08:34 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Mar 2007 23:08:34 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5FF1ED15@au3010avexu1.global.avaya.com> References: <2773CAC687FD5F4689F526998C7E4E5FF1ED15@au3010avexu1.global.avaya.com> Message-ID: <45F87262.1060307@v.loewis.de> Delaney, Timothy (Tim) schrieb: > A change to the documented behaviour should require a __future__ > import for at least one version. That's even assuming that the change > is desireable (I don't believe so). We have multiple anecdotes of > actual, existing code that *will* break with this change. So far I > haven't seen any actual code posted that is currently broken by the > existing behaviour. Can you please point to them? I'm only aware of a single anecdote (about software that likely will never see Python 2.6). Regards, Martin From tdelaney at avaya.com Wed Mar 14 23:09:29 2007 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Thu, 15 Mar 2007 09:09:29 +1100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) Message-ID: <2773CAC687FD5F4689F526998C7E4E5F074476@au3010avexu1.global.avaya.com> "Martin v. L?wis" wrote: > It's not "just" that people disliked the behavior. The majority of > those > that commented agreed that the current behavior is incorrect. Some > also > observed that the online documentation was underspecified, and indeed > allowed for that change. So this is a bug fix, even though the old > test > case explicitly tested for the presence of the bug, and even though > the > doc string explicitly documented the old behavior. They were all > consistent, but they were consistently wrong. The patch special-cases one edge case where an "extension" does not indicate the type of the file, but metadata on a particular platform. Here's a couple of other edge cases that are not addressed by the patch: On Mac OS (including Mac OS X) you can have a file without an extension, where the type is specified in the metadata. However, such a file could also happen to contain a dot in the file name: splitext("Version 1.2") -> ("Version 1", "2") Also, quite often you have multiple extensions: splitext("file.tar.gz") -> ("file.tar", "gz") To me, the "extension" there is "tar.gz". Basically, changing the behaviour of splitext() is simply trying to make it "do what I mean" (DWIM). But it's never going to DWIM. I think the best we can do is have a simple, unambiguous rule, with no exceptions - which is what the existing behaviour does. Tim Delaney From pje at telecommunity.com Wed Mar 14 23:30:53 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 14 Mar 2007 17:30:53 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F86F21.8030904@v.loewis.de> References: <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> At 10:54 PM 3/14/2007 +0100, Martin v. L?wis wrote: >Phillip J. Eby schrieb: > > That much is obvious. But I haven't seen any explanation as to why > > explicitly-documented and explicitly-tested behavior should be treated > > as a bug in policy terms, just because people don't like the documented > > and tested behavior. > >It's not "just" that people disliked the behavior. The majority of those >that commented agreed that the current behavior is incorrect. And yet, that "incorrect" behavior was clearly intended by the author(s) of the code, test, and docstrings. As it happens, Guido wrote that code (16 years ago) and the docstring (9 years ago), in the case of the posixpath module at least. And in an amusing twist, it appears that you yourself checked in the test, 4 years ago! If this behavior was so *obviously* buggy, why didn't you ask Guido or "fix" it then? But wait, it gets better! Five years ago, you also recommended *rejection* of a similar patch: """I also dislike this patch. The current behaviour completely matches the documented behaviour; changing it might break existing applications. If you need a different behaviour, write a different function.""" http://python.org/sf/536120 So, why is it obviously broken now, but not five years ago? > > So far, the only policy justification I've seen you give was along the > > lines of, "I volunteered to do it, so I get to decide". > >It's more than that. I conducted a poll, and here people were largely >in favor of that change. Had they been largely in opposition, I would >have rejected the patch. By this logic, I could conduct a popularity poll for say, "fixing" the distutils by changing its behavior to match that of setuptools, and go ahead with it if the majority agreed with me that the distutils' behavior was clearly incorrect by retroactive comparison -- despite it being documented and tested behavior, and despite objections of backward incompatibility being presented on Python-Dev. So, I don't understand your reasoning here at all. >However, *some* action is necessary. The patch was sitting there for >a long time already, and it is unfair to the submitter to not act just >because you cannot decide. The bug report was even older. So reject it, or propose to add a new API. >P.S. If you apply the same effort to all changes that are constantly >being made to Python, you will find that you will need to revert many >of them. Then I'm amazed that there is so much desire to *increase* the number of changes being made to Python, if we can't even manage to follow our policies *now*. From arigo at tunes.org Wed Mar 14 23:39:40 2007 From: arigo at tunes.org (Armin Rigo) Date: Wed, 14 Mar 2007 23:39:40 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F86F21.8030904@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <45F86F21.8030904@v.loewis.de> Message-ID: <20070314223940.GA2962@code0.codespeak.net> Hi Martin, On Wed, Mar 14, 2007 at 10:54:41PM +0100, "Martin v. L?wis" wrote: > So this is a bug fix, even though the old test > case explicitly tested for the presence of the bug FWIW, when developing PyPy we found quite a number of tests in CPython that were checking not just obscure implementation details, but things known to be of the kind "don't do that you fool" or "let's check for exactly one of the many things that could reasonably occur here". CPython's test suite should not be regarded as a behavior definition; only the documentation should. I consider this case similarly debatable, and to this debate I will contribute a +1 on keeping Martin's patch. A bientot, Armin From bjourne at gmail.com Thu Mar 15 00:01:21 2007 From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=) Date: Thu, 15 Mar 2007 00:01:21 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F83535.9080902@v.loewis.de> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> Message-ID: <740c3aec0703141601v76c30b2cida1f942fb05c1073@mail.gmail.com> On 3/14/07, Phillip J. Eby wrote: > At 06:47 PM 3/14/2007 +0100, Martin v. L?wis wrote: > >Phillip J. Eby schrieb: > > > This backwards-incompatible change is therefore contrary to policy and > > > should be reverted, pending a proper transition plan for the change > > > (such as introduction of an alternative API and deprecation of the > > > existing one.) > > > >I'm clearly opposed to this proposal, or else I wouldn't have committed > >the change in the first place. > > That much is obvious. But I haven't seen any explanation as to why > explicitly-documented and explicitly-tested behavior should be treated as a > bug in policy terms, just because people don't like the documented and > tested behavior. > > So far, the only policy justification I've seen you give was along the > lines of, "I volunteered to do it, so I get to decide". I guess it maybe also had something to do with that the bug had been left open on the bug tracker for almost two years and the persons favoring the change had a 2:1 advantage (3:1 if you count the patch contributor). There was plenty of time to voice your dissent. I also think that which ever behavior splitext has, it does not matter much in the grand scheme of things. There are bigger fishes to fry. -- mvh Bj?rn From jon+python-dev at unequivocal.co.uk Thu Mar 15 01:03:41 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Thu, 15 Mar 2007 00:03:41 +0000 Subject: [Python-Dev] thread safe SMTP module In-Reply-To: <45F8579F.7060104@eburg.com> References: <45F82421.1080602@eburg.com> <20070314174856.GB27107@snowy.squish.net> <20070314182620.GA19184@panix.com> <45F8579F.7060104@eburg.com> Message-ID: <20070315000341.GD27107@snowy.squish.net> Gordon Messmer wrote: > Tonight I should have time to pull an old copy of the code out of CVS > and recreate the test script that I used. Once I have, it should be a > matter of feeding a big list of email addresses to the script and > waiting a couple of minutes for the script to lock up. The last time I > looked, the problem was very easy to observe. Try it using the alternative makefile code I posted and see if it makes a difference... From glyph at divmod.com Thu Mar 15 01:12:37 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Thu, 15 Mar 2007 00:12:37 -0000 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> Message-ID: <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> On 14 Mar, 05:08 pm, pje at telecommunity.com wrote: >In addition to the prior behavior being explicitly documented in the >function docstrings, r54204 shows that it was also *tested* as behaving >that way by test_macpath, test_ntpath, and test_posixpath. When >combined >with the explicit docstrings, this must be considered incontrovertible >proof that the previous behavior was either explicitly intended, or at >the >very least a known, expected, and *accepted* behavior. I (obviously, I think) agree with all of that. >This backwards-incompatible change is therefore contrary to policy and >should be reverted, pending a proper transition plan for the change >(such >as introduction of an alternative API and deprecation of the existing >one.) I hope that this is true, but *is* this "policy" documented as required somewhere yet? I think it should be reverted regardless, and such a policy instated if one does not exist, but it is my understanding at this point that it is an informal consensus rather than a policy. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070315/5bdb8aec/attachment.html From anthony at interlink.com.au Thu Mar 15 02:16:28 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu, 15 Mar 2007 12:16:28 +1100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F86F21.8030904@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <45F86F21.8030904@v.loewis.de> Message-ID: <200703151216.33674.anthony@interlink.com.au> On Thursday 15 March 2007 08:54, Martin v. L?wis wrote: > The bug fix may also break existing applications. Hence it cannot > go into 2.5.1 but has to wait for 2.6. Steering clear of the rest of the discussion, I'd just like to give a hearty "+1" for this not going into 2.5.x in any way shape or form. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Thu Mar 15 02:22:11 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu, 15 Mar 2007 12:22:11 +1100 Subject: [Python-Dev] Backports of standard library modules In-Reply-To: References: <20070312011956.7769.669657250.divmod.xquotient.661@joule.divmod.com> Message-ID: <200703151222.14713.anthony@interlink.com.au> > :) is that, when I go to the downloads page for Python 2.3, in > addition to downloading Python, I could download all the > compatible libraries which were included in later versions as a > single installable file. When 2.6 comes out, this "extras" > package would be upgraded to include any new modules in 2.6. > > Not a lot of fun for people who live on the bleeding edge, but > very useful for people who are stuck with an older version for > political or other reasons. If you, or someone else, is willing to do the work to do this, I don't think anyone would mind. There is (as Martin also stated) zero chance that I will do this additional work. It scratches no itches for me, and has the potential to add an enormous amount to my workload of doing a new release. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From skip at pobox.com Thu Mar 15 02:37:51 2007 From: skip at pobox.com (skip at pobox.com) Date: Wed, 14 Mar 2007 20:37:51 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <200703151216.33674.anthony@interlink.com.au> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <45F86F21.8030904@v.loewis.de> <200703151216.33674.anthony@interlink.com.au> Message-ID: <17912.41839.787096.396533@montanaro.dyndns.org> Anthony> On Thursday 15 March 2007 08:54, Martin v. L?wis wrote: >> The bug fix may also break existing applications. Hence it cannot go >> into 2.5.1 but has to wait for 2.6. Anthony> Steering clear of the rest of the discussion, I'd just like to Anthony> give a hearty "+1" for this not going into 2.5.x in any way Anthony> shape or form. Given that the topic seems so contentious, maybe it should wait until 3.x. Skip From murman at gmail.com Thu Mar 15 02:46:37 2007 From: murman at gmail.com (Michael Urman) Date: Wed, 14 Mar 2007 20:46:37 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <200703151216.33674.anthony@interlink.com.au> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <45F86F21.8030904@v.loewis.de> <200703151216.33674.anthony@interlink.com.au> Message-ID: On 3/14/07, Anthony Baxter wrote: > Steering clear of the rest of the discussion, I'd just like to give > a hearty "+1" for this not going into 2.5.x in any way shape or > form. Agreed. I'd further vote for keeping this change out until 3.x because it is a behavior change in a corner case predicated on a value judgement. Yes I find the idea of an extension without a filename to be silly. However this change punishes he who checked the corner cases to help he who did not. If this change is primarily geared to help in the case where people want to retrieve the file name without the extension, we should add a function to return this basic name. Who would rather see os.path.dropext(path)? Michael -- Michael Urman http://www.tortall.net/mu/blog From hpk at trillke.net Thu Mar 15 03:47:49 2007 From: hpk at trillke.net (holger krekel) Date: Thu, 15 Mar 2007 03:47:49 +0100 Subject: [Python-Dev] pypy's interpreter/highlevel backend features Message-ID: <20070315024749.GB4635@solar.trillke> Hello Python-dev! we have a document on PyPy's interpreter and translation features that might be of interest to you - we target it at language implementors in general: Includes a discussion on our .NET and also the emerging Java backends, as well as our RPython -> Javascript webapp generating experiments, on security relevant taint propagation and transparent proxies ... IOW quite a bag :) We'd be very happy about feedback and opinions/questions (preferably until Monday, 19th March) http://codespeak.net/pypy/extradoc/eu-report/D12.1_H-L-Backends_and_Feature_Prototypes-interim-2007-03-12.pdf best, Holger -- merlinux GmbH Steinbergstr. 42 31139 Hildesheim http://merlinux.de tel +49 5121 20800 75 (fax 77) From greg.ewing at canterbury.ac.nz Thu Mar 15 06:19:39 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Mar 2007 18:19:39 +1300 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <45F86F21.8030904@v.loewis.de> <200703151216.33674.anthony@interlink.com.au> Message-ID: <45F8D76B.1010204@canterbury.ac.nz> Michael Urman wrote: > Who would rather see os.path.dropext(path)? I'd like to see such a function, and also maybe replaceext(path, new_ext). I often end up coding things like these myself, since indexing the result of splitext all the time is rather ugly. To round off the set, I suggest path.dropext(path) # returns the base name path.getext(path) # returns the extension path.replaceext(path, newext) # adds or replaces the extension -- Greg From tjreedy at udel.edu Thu Mar 15 06:50:18 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 15 Mar 2007 01:50:18 -0400 Subject: [Python-Dev] pypy's interpreter/highlevel backend features References: <20070315024749.GB4635@solar.trillke> Message-ID: "holger krekel" wrote in message news:20070315024749.GB4635 at solar.trillke... | We'd be very happy about feedback and opinions/questions | (preferably until Monday, 19th March) | | http://codespeak.net/pypy/extradoc/eu-report/D12.1_H-L-Backends_and_Feature_Prototypes-interim-2007-03-12.pdf As of this moment, this is not loading. -- times out From sgeiger at ncee.net Thu Mar 15 06:57:06 2007 From: sgeiger at ncee.net (Shane Geiger) Date: Thu, 15 Mar 2007 00:57:06 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F8D76B.1010204@canterbury.ac.nz> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <45F86F21.8030904@v.loewis.de> <200703151216.33674.anthony@interlink.com.au> <45F8D76B.1010204@canterbury.ac.nz> Message-ID: <45F8E032.10708@ncee.net> Good, we have gotten around to discussing a little annoyance I've noticed. I think this should behave similar to the way a Unix admin who is familiar with basename(1) would behave: $ basename -s .py /tmp/foo.py foo $ $ basename /tmp/foo.py .py foo $ basename /tmp/foo.py /tmp/fffff.py foo.py $ basename -a /tmp/foo.py /tmp/fffff.py foo.py fffff.py $ basename -a /tmp/foo.py /tmp/fffff.py .py foo.py fffff.py .py $ basename -s .py -a /tmp/foo.py /tmp/fffff.py .py foo fffff .py $ basename -s .py -a /tmp/foo.py /tmp/fffff.py foo fffff $ Changing the function name shouldn't be the solution, imho. Changing the arguments, however, should be. Compare the documentation yourself: ----- MANPAGE DOCUMENTATION NAME basename, dirname -- return filename or directory portion of pathname SYNOPSIS basename string [suffix] basename [-a] [-s suffix] string [...] dirname string ----- ----- PYTHON DOCUMENTATION: Help on function basename in module posixpath: basename(p) Returns the final component of a pathname ----- Greg Ewing wrote: > Michael Urman wrote: > >> Who would rather see os.path.dropext(path)? >> > > I'd like to see such a function, and also > maybe replaceext(path, new_ext). I often > end up coding things like these myself, > since indexing the result of splitext all > the time is rather ugly. > > To round off the set, I suggest > > path.dropext(path) # returns the base name > path.getext(path) # returns the extension > path.replaceext(path, newext) # adds or replaces the extension > > -- > Greg > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/sgeiger%40ncee.net > -- Shane Geiger IT Director National Council on Economic Education sgeiger at ncee.net | 402-438-8958 | http://www.ncee.net Leading the Campaign for Economic and Financial Literacy -------------- next part -------------- A non-text attachment was scrubbed... Name: sgeiger.vcf Type: text/x-vcard Size: 297 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20070315/a12279d6/attachment.vcf From martin at v.loewis.de Thu Mar 15 07:45:19 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Mar 2007 07:45:19 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> References: <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> Message-ID: <45F8EB7F.6070908@v.loewis.de> Phillip J. Eby schrieb: > And yet, that "incorrect" behavior was clearly intended by the author(s) > of the code, test, and docstrings. > > As it happens, Guido wrote that code (16 years ago) and the docstring (9 > years ago), in the case of the posixpath module at least. I don't find it that clear that it was the intention, AFAICT, it could have been an accident also. Guido added the doc strings as a contribution from Charles G. Waldman; he may just have documented the implemented behavior. In r4493, Sjoerd Mullender changed splitext (in an incompatible way) so that it would split off only the last extension, before, foo.tar.gz would be split into 'foo', '.tar.gz'. So it's clear that the intention was always to "split off the extension", whether or not the behavior on dotfiles was considered I cannot tell. As for Doc/lib, in r6524 Guido changed it to document the actual behavior, from the last component of \var{root} contains no periods, and \var{ext} is empty or begins with a period. to and \var{ext} is empty or begins with a period and contains at most one period. So it seems the original (Guido's) intention was that it splits of all extensions; Sjoerd then changed it to split off only the last extension. > And in an amusing twist, it appears that you yourself checked in the > test, 4 years ago! If this behavior was so *obviously* buggy, why > didn't you ask Guido or "fix" it then? If you look at #536661, you'll see that I committed that as a contribution of Sebastian Keim. The intention here was to rigorously test the implemented behavior, after a failed attempt of challenging it. > But wait, it gets better! Five years ago, you also recommended > *rejection* of a similar patch: > > """I also dislike this patch. The current behaviour completely > matches the documented behaviour; changing it might break > existing applications. If you need a different behaviour, > write a different function.""" > > http://python.org/sf/536120 > > So, why is it obviously broken now, but not five years ago? I apparently took the same position that you now take back then, whereas I'm now leaning towards (or going beyond) the position Tim had back then, who wrote "BTW, if it *weren't* for the code breakage, I'd be in favor of doing this." I now believe that this should be done *despite* having been documented and tested (which, as you can see, was documented and tested only match the implemented behavior). That it keeps popping up is proof that the old behavior is deemed incorrect by many people. >> It's more than that. I conducted a poll, and here people were largely >> in favor of that change. Had they been largely in opposition, I would >> have rejected the patch. > > By this logic, I could conduct a popularity poll for say, "fixing" the > distutils by changing its behavior to match that of setuptools, and go > ahead with it if the majority agreed with me that the distutils' > behavior was clearly incorrect by retroactive comparison -- despite it > being documented and tested behavior, and despite objections of backward > incompatibility being presented on Python-Dev. And indeed, when you go back to the last discussion on distutils, this is precisely what I recommend you to do, instead of putting layers into setuptools to work around what you consider quirks in distutils. > So reject it, or propose to add a new API. Neither is a solution. Rejecting it means it will keep popping up forever. The amount of Python code yet to be written is hopefully larger than the code already written (paraphrasing Guido), so in the long run, it should show the right behavior, not the historical one. Adding a new API has a very high cost: it should ultimately involve removing the old API, so *all* usage of splitext will eventually break, not just those cases which happen to break under the current change. Also, books need to be updated, etc. Furthermore, nobody has proposed a name for a new API, yet. >> P.S. If you apply the same effort to all changes that are constantly >> being made to Python, you will find that you will need to revert many >> of them. > > Then I'm amazed that there is so much desire to *increase* the number of > changes being made to Python, if we can't even manage to follow our > policies *now*. It seems that we disagree on what the policies are. 100% backwards compatibility is none of them. Feature releases may include incompatible changes, and this is just one of them (and a minor one, too). What policy do you think is this change violating? Regards, Martin From martin at v.loewis.de Thu Mar 15 07:51:42 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 15 Mar 2007 07:51:42 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> Message-ID: <45F8ECFE.2030105@v.loewis.de> glyph at divmod.com schrieb: > >This backwards-incompatible change is therefore contrary to policy and > >should be reverted, pending a proper transition plan for the change (such > >as introduction of an alternative API and deprecation of the existing > one.) > > I hope that this is true, but *is* this "policy" documented as required > somewhere yet? I think it should be reverted regardless, and such a > policy instated if one does not exist, but it is my understanding at > this point that it is an informal consensus rather than a policy. I'm not quite sure what "it" is, here. If "it" is that there be no incompatible changes in Python: this is not policy, and not even consensus. Instead, policy (as enforced by the release manager), and consensus is that bug fix releases (2.x.y) must not show incompatible behavior. For feature releases (2.x), incompatible behavior is acceptable (at least this is what I thought consensus is, but apparently I'm wrong). Regards, Martin From mal at egenix.com Thu Mar 15 08:49:18 2007 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 15 Mar 2007 08:49:18 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F8EB7F.6070908@v.loewis.de> References: <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> Message-ID: <45F8FA7E.9090906@egenix.com> On 2007-03-15 07:45, Martin v. L?wis wrote: > Phillip J. Eby schrieb: >> And yet, that "incorrect" behavior was clearly intended by the author(s) >> of the code, test, and docstrings. >> >> As it happens, Guido wrote that code (16 years ago) and the docstring (9 >> years ago), in the case of the posixpath module at least. > > I don't find it that clear that it was the intention, AFAICT, it could > have been an accident also. Guido added the doc strings as a > contribution from Charles G. Waldman; he may just have documented the > implemented behavior. > > In r4493, Sjoerd Mullender changed splitext (in an incompatible way) > so that it would split off only the last extension, before, foo.tar.gz > would be split into 'foo', '.tar.gz'. So it's clear that the intention > was always to "split off the extension", whether or not the behavior > on dotfiles was considered I cannot tell. > > As for Doc/lib, in r6524 Guido changed it to document the actual > behavior, from > > the last component of \var{root} contains no periods, > and \var{ext} is empty or begins with a period. > > to > > and \var{ext} is empty or begins with a period and contains > at most one period. > > So it seems the original (Guido's) intention was that it splits > of all extensions; Sjoerd then changed it to split off only the > last extension. Whatever the intention was or has been: the term "extension" itself is not well-defined, so there's no obvious right way to implement an API that splits off an extension. E.g. in some cases, .tar.gz is considered an extension, in others, the .gz part is just a transfer encoding and .tar the extension. Then you have .tgz which is a bit of both. It also depends on the platform, e.g. on Windows, only the very last part of a filename is used as extension by the OS to determine the (MIME) type of a file. As always, it's best to just right your own application-specific code to get defined behavior. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 15 2007) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: From cfbolz at gmx.de Thu Mar 15 11:09:17 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 15 Mar 2007 11:09:17 +0100 Subject: [Python-Dev] pypy's interpreter/highlevel backend features In-Reply-To: References: <20070315024749.GB4635@solar.trillke> Message-ID: <45F91B4D.8090002@gmx.de> Terry Reedy wrote: > "holger krekel" wrote in message > news:20070315024749.GB4635 at solar.trillke... > | We'd be very happy about feedback and opinions/questions > | (preferably until Monday, 19th March) > | > | > http://codespeak.net/pypy/extradoc/eu-report/D12.1_H-L-Backends_and_Feature_Prototypes-interim-2007-03-12.pdf > > As of this moment, this is not loading. -- times out Works for me now, in the last days there were routing problems with the provider, maybe that was it. Cheers, Carl Friedrich From facundo at taniquetil.com.ar Thu Mar 15 13:48:09 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Thu, 15 Mar 2007 12:48:09 +0000 (UTC) Subject: [Python-Dev] pypy's interpreter/highlevel backend features References: <20070315024749.GB4635@solar.trillke> Message-ID: holger krekel wrote: > Hello Python-dev! Hello Holger! > We'd be very happy about feedback and opinions/questions > (preferably until Monday, 19th March) > > http://codespeak.net/pypy/extradoc/eu-report/D12.1_H-L-Backends_and_Feature_Prototypes-interim-2007-03-12.pdf It seems quite interesting. I'll read it later. BTW, I downloaded it ok. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From martin at v.loewis.de Thu Mar 15 14:34:15 2007 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Mar 2007 14:34:15 +0100 Subject: [Python-Dev] Status of thread cancellation Message-ID: <45F94B57.6080507@v.loewis.de> I just proposed to implement thread cancellation for the SoC. Is there any prior work where one could start? Regards, Martin From exarkun at divmod.com Thu Mar 15 14:58:24 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Thu, 15 Mar 2007 08:58:24 -0500 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: <45F94B57.6080507@v.loewis.de> Message-ID: <20070315135824.18920.724506537.divmod.quotient.2760@ohm> On Thu, 15 Mar 2007 14:34:15 +0100, "\"Martin v. L?wis\"" wrote: >I just proposed to implement thread cancellation for the SoC. >Is there any prior work where one could start? The outcome of some prior work, at least: http://java.sun.com/j2se/1.4.2/docs/guide/misc/threadPrimitiveDeprecation.html Jean-Paul From skip at pobox.com Thu Mar 15 15:41:31 2007 From: skip at pobox.com (skip at pobox.com) Date: Thu, 15 Mar 2007 09:41:31 -0500 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: <20070315135824.18920.724506537.divmod.quotient.2760@ohm> References: <45F94B57.6080507@v.loewis.de> <20070315135824.18920.724506537.divmod.quotient.2760@ohm> Message-ID: <17913.23323.449154.329106@montanaro.dyndns.org> >> I just proposed to implement thread cancellation for the SoC. Is >> there any prior work where one could start? Jean-Paul> The outcome of some prior work, at least: Jean-Paul> http://java.sun.com/j2se/1.4.2/docs/guide/misc/threadPrimitiveDeprecation.html I responded to that. I got the impression reading that page that the killed thread doesn't regain control so it can't clean up its potentially inconsistent data structures. I inferred from Martin's proposal that he expected the thread to be able to catch the exception. Perhaps he can elaborate on what cleanup actions the dying thread will be allowed to perform. Skip From exarkun at divmod.com Thu Mar 15 16:12:36 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Thu, 15 Mar 2007 10:12:36 -0500 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: <17913.23323.449154.329106@montanaro.dyndns.org> Message-ID: <20070315151236.18920.1909125356.divmod.quotient.2790@ohm> On Thu, 15 Mar 2007 09:41:31 -0500, skip at pobox.com wrote: > > >> I just proposed to implement thread cancellation for the SoC. Is > >> there any prior work where one could start? > > Jean-Paul> The outcome of some prior work, at least: > > Jean-Paul> http://java.sun.com/j2se/1.4.2/docs/guide/misc/threadPrimitiveDeprecation.html > >I responded to that. I got the impression reading that page that the killed >thread doesn't regain control so it can't clean up its potentially >inconsistent data structures. The second question on the page: Couldn't I just catch the ThreadDeath exception and fix the damaged object? Addresses this. >I inferred from Martin's proposal that he >expected the thread to be able to catch the exception. Perhaps he can >elaborate on what cleanup actions the dying thread will be allowed to >perform. Perhaps he can. Hopefully, he can specifically address these points: 1. A thread can throw a ThreadDeath exception almost anywhere. All synchronized methods and blocks would have to be studied in great detail, with this in mind. 2. A thread can throw a second ThreadDeath exception while cleaning up from the first (in the catch or finally clause). Cleanup would have to repeated till it succeeded. The code to ensure this would be quite complex. Jean-Paul From martin at v.loewis.de Thu Mar 15 17:24:31 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Mar 2007 17:24:31 +0100 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: <20070315151236.18920.1909125356.divmod.quotient.2790@ohm> References: <20070315151236.18920.1909125356.divmod.quotient.2790@ohm> Message-ID: <45F9733F.7030409@v.loewis.de> Jean-Paul Calderone schrieb: >> I inferred from Martin's proposal that he >> expected the thread to be able to catch the exception. Perhaps he can >> elaborate on what cleanup actions the dying thread will be allowed to >> perform. > > Perhaps he can. Hopefully, he can specifically address these points: > > 1. A thread can throw a ThreadDeath exception almost anywhere. All > synchronized methods and blocks would have to be studied in great > detail, with this in mind. > > 2. A thread can throw a second ThreadDeath exception while cleaning up > from the first (in the catch or finally clause). Cleanup would have > to repeated till it succeeded. The code to ensure this would be quite > complex. Clearly, a thread need to have its finally blocks performed in response to a cancellation request. These issues are real, however, they apply to any asynchronous exception, not just to thread cancellation. In Python, we already have an asynchronous exception: KeyboardInterrupt. This suffers from the same problems: a KeyboadInterrupt also can occur at any point, interrupting code in the middle of its finally-blocks. The other exception that is nearly-asynchronous is OutOfMemoryError, which can occur at nearly any point (but of course, never occurs in practice). So yes, it would be good if Python's exception handling supported asynchronous exceptions in a sensible way. I have to research somewhat more, but I think the standard solution to the problem in operating system (i.e. disabling interrupts at certain points, explicitly due to code or implicitly as a result of entering the interrupt handler) may apply. Regards, Martin From facundo at taniquetil.com.ar Thu Mar 15 18:04:46 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Thu, 15 Mar 2007 17:04:46 +0000 (UTC) Subject: [Python-Dev] Status of thread cancellation References: <20070315151236.18920.1909125356.divmod.quotient.2790@ohm> <45F9733F.7030409@v.loewis.de> Message-ID: Martin v. L?wis wrote: > asynchronous exceptions in a sensible way. I have to research somewhat > more, but I think the standard solution to the problem in operating > system (i.e. disabling interrupts at certain points, explicitly > due to code or implicitly as a result of entering the interrupt > handler) may apply. Two already working schemes, that are similar, comes to my mind. One is signals in Linux/Unix, where you can send SIGTERM, and the process can handle it and do whatever it takes. But also you can send SIGKILL, which can not be blocked. The other is microprocessors, where you have interrupts, and when the interrupt is received, you disable it (are there processors that support "reentrant" interrupts? I don't know of any, but I'm no specialist here). To me, is natural this behaviour: One can send ThreadDeath to the thread, and it can handle it or no. If not, it dies. If yes, it does some stuff and dies. But if I send a second ThreadDeath to the same thread, when it's still "dying", for me it's ok to receive an answer like "Ok, ok, I heard you, I'm on it". But, in that scenario, should be a way to say to the thread "Die, die now, no matter what"? Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From pje at telecommunity.com Thu Mar 15 18:51:09 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 15 Mar 2007 12:51:09 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F8EB7F.6070908@v.loewis.de> References: <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> At 07:45 AM 3/15/2007 +0100, Martin v. L?wis wrote: >I apparently took the same position that you now take back then, >whereas I'm now leaning towards (or going beyond) the position >Tim had back then, who wrote "BTW, if it *weren't* for the code breakage, >I'd be in favor of doing this." If it weren't for the code breakage, I'd be in favor too. That's not the point. The point is that how can Python be stable as a language if precedents can be reversed without a migration plan, just because somebody changes their mind? In another five years, will you change your mind again, and decide to put this back the way it was? Speaking as a business person, that seems to me... unwise. When I found out that this change had been checked in despite all the opposition, my gut reaction was, "I guess I can't rely on Python any more", despite 10 years of working with it, developing open source software with it, and contributing to its development. Because from a *business* perspective, this sort of flip-flopping means that moving from one "minor" Python version to another is potentially *very* costly. The process of having warnings at least ensures that I can *discover* whether my programs depend on some behavior that has changed - rather than having something that used to work and now doesn't. >I now believe that this should be done *despite* having been >documented and tested (which, as you can see, was documented >and tested only match the implemented behavior). That it keeps >popping up is proof that the old behavior is deemed incorrect >by many people. But as you are so fond of pointing out, there is no "many people". There are only individual people. That a majority want it one way, means that there is a minority who want it another. If next year, it becomes more popular to have it the other way, will we switch again? If a majority of people want braces and required type declarations, will we add them? After all, there is *substantial* support for some proposals along those lines! Yet, one of the appeals of Python is that it has some sense of what is "right" or "wrong", and some explanation for that rightness or wrongness that doesn't change with the ebb and flow of popular opinion and the current population of a mailing list. IMO, Python is not -- or at least should not be -- a popularity contest. >>So reject it, or propose to add a new API. > >Neither is a solution. Rejecting it means it will keep popping up >forever. Like requests to remove whitespace sensitivity and add braces? That a request may keep popping up forever is not an argument for changing it NOW. As Tim put it, "Never is often better than *right* now," and it seems to me that this is *exactly* the sort of change for which that saying was coined. >The amount of Python code yet to be written is hopefully larger >than the code already written (paraphrasing Guido), so in the long run, >it should show the right behavior, not the historical one. Sure - but by that argument, the amount of code that will be written in 3.0 or 3.1 is larger still, and if this behavior's been mostly okay for 9+ years, then fixing it in a year or two should be quite prompt, if you want to look at the historical scale. In any case, my main concern about this change isn't whether it's right or wrong -- it's about whether Python provides a stable platform for software development with reasonable migration paths. *This* change won't actually hurt *me* -- but what will the next change be? Must everyone who wants some form of stability maintain a constant watch over Python's source changes? I gather that your answer is "yes", and that's what disturbs me here. From glyph at divmod.com Thu Mar 15 19:25:23 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Thu, 15 Mar 2007 18:25:23 -0000 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: <45F9733F.7030409@v.loewis.de> References: <20070315151236.18920.1909125356.divmod.quotient.2790@ohm> <45F9733F.7030409@v.loewis.de> Message-ID: <20070315182523.7769.715364850.divmod.xquotient.887@joule.divmod.com> On 04:24 pm, martin at v.loewis.de wrote: >Jean-Paul Calderone schrieb: >>>I inferred from Martin's proposal that he >>>expected the thread to be able to catch the exception. Perhaps he >>>can >>>elaborate on what cleanup actions the dying thread will be allowed to >>>perform. >> >>Perhaps he can. Hopefully, he can specifically address these points: >> >> 1. A thread can throw a ThreadDeath exception almost anywhere. All >> synchronized methods and blocks would have to be studied in >>great >> detail, with this in mind. >> >> 2. A thread can throw a second ThreadDeath exception while cleaning >>up >> from the first (in the catch or finally clause). Cleanup would >>have >> to repeated till it succeeded. The code to ensure this would be >>quite >> complex. > >Clearly, a thread need to have its finally blocks performed in response >to a cancellation request. These issues are real, however, they apply >to any asynchronous exception, not just to thread cancellation. To be sure, the problem does apply to all asynchronous exceptions. That's why it is generally understood that a program which has received an asynchronous exception cannot continue. >In Python, we already have an asynchronous exception: >KeyboardInterrupt. >This suffers from the same problems: a KeyboadInterrupt also can occur >at any point, interrupting code in the middle of its finally-blocks. >The other exception that is nearly-asynchronous is OutOfMemoryError, >which can occur at nearly any point (but of course, never occurs in >practice). KeyboardInterrupt and MemoryError share a common feature which forced thread termination does not: nobody reasonably expects the program to keep running after they have been raised. Indeed, programs are written with the expectation that MemoryError will never occur, and if it does, the user is not surprised to find them in an inconsistent state. In any situation where a MemoryError may reasonably be expected - that is to say, a specific, large allocation of a single block of memory - it can be trapped as if it were not asynchronous. Long-running Python programs which expect to need to do serious clean-up in the face of interrupts, in fact, block KeyboardInterrupt by registering their own interrupt handlers (Zope, Twisted). Developers who think they want thread cancellation inevitably believe they can, if they are "sufficiently careful", implement operating- system-like scheduling features by starting arbitrary user code and then providing "terminate", "pause", and "resume" commands. That was the original intent of these (now removed) Java APIs, and that is why they were removed: you can't do this. It's impossible. Asynchronous exceptions are better than immediate termination because they allow for code which is allocating scarce or fragile resources to have a probabilistically better chance of cleaning up. However, nobody writes code like this: def addSomeStuff(self, volume, mass): self.volume += volume try: self.mass += mass except AsynchronousInterrupt: while 1: try: self.volume -= volume break except AsynchronousInterrupt: pass and nobody is going to start if the language provides thread termination. Async-Exception-Safe Python code is, and will be, as rare as POSIX Async-Safe functions, which means at best you will be able to call a thread cancellation API and have it _appear_ to work in some circumstances. In any system which uses Python code not explicitly designed to support asynchronous exceptions (let's say, the standard library) it will be completely impossible to write correct code. I'm not a big fan of shared-state-threading, but it does allow for a particular programming model. Threading provides you some guarantees. You can't poke around on the heap, but you know that your stack, and your program counter, are inviolate. You can reason about, if not quite test, the impact of sharing a piece of state on the heap; its destructive methods need to be synchronized along with the read methods that interact with it. Asynchronous exceptions destroy all hope of sanity. Your program might suddenly perform a nonlocal exit _anywhere_ except, maybe, inside a "finally" block. This literally encourages some people that program in environments where asynchronous exceptions are possible (.NET, in particular) to put huge chunks of application code inside finally blocks. They generally look like this: try {} finally { // entire application here } because that is really the only way you can hope to write code that will function robustly in such an environment. >So yes, it would be good if Python's exception handling supported >asynchronous exceptions in a sensible way. I have to research somewhat >more, but I think the standard solution to the problem in operating >system (i.e. disabling interrupts at certain points, explicitly >due to code or implicitly as a result of entering the interrupt >handler) may apply. For one thing, the Python core code is not operating system kernel code and it is unlikely that the techniques found there will apply. Interrupts, in particular, are nothing at all like exceptions, and have a completely different impact on running code (which, since it is kernel code, is written much more carefully and under an entirely different set of constraints than Python application, or even framework, code). Can you suggest any use-cases for thread termination which will *not* result in a completely broken and unpredictable heap after the thread has died? If you can think of such a case, are you sure it wouldn't be better served by a set of threads communicating over queues and sending 'Stop' objects to each other to indicate termination at a point in queue rather than a forced termination via exception? Just in case it's not clear from the other things I've said: this is a terrible, terrible idea, and I am shocked that it is even being *considered* for inclusion in Python. As a foolish youth, I wasted many months trying to get a program that used Java's (then not deprecated) asynchronous exception APIs to behave properly. It wasn't possible then, and it isn't possible now. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070315/6a356b17/attachment-0001.html From martin at v.loewis.de Thu Mar 15 19:35:44 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Mar 2007 19:35:44 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> References: <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> Message-ID: <45F99200.7040303@v.loewis.de> Phillip J. Eby schrieb: > If it weren't for the code breakage, I'd be in favor too. That's not the > point. > > The point is that how can Python be stable as a language if precedents can > be reversed without a migration plan, just because somebody changes their > mind? In another five years, will you change your mind again, and decide > to put this back the way it was? I'm still wondering what policy you think I have violated. If we both agree that the old behavior was erroneous, then I cannot understand why you want to see the patch reverted. Nobody has so far proposed any alternative fix (not even as a specification, let alone as a specific patch - just with some hinting), and the majority of the people polled thought that it ought to be fixed. > The process of having warnings at least ensures that I can *discover* > whether my programs depend on some behavior that has changed - rather than > having something that used to work and now doesn't. So you would agree to the change if a warning was generated at run-time? Notice that there is already a warning there: the documentation clearly states that the behavior has changed, and, of course, Misc/NEWS lists it as changed behavior. So you can certainly discover *now* that the behavior has changed: read the documentation. If you want to discover it without reading documentation, we can discuss that. > But as you are so fond of pointing out, there is no "many people". There > are only individual people. That a majority want it one way, means that > there is a minority who want it another. If next year, it becomes more > popular to have it the other way, will we switch again? This is highly theoretical. > If a majority of > people want braces and required type declarations, will we add them? PEP 3099 explicitly rules out the introduction of braces. As for type declarations: it would require a PEP, being a major feature. It then depends on the details. PEP 245 was rejected by BDFL pronouncement. This is how things are ultimately decided: by BDFL pronouncement. > Yet, one of the appeals of Python is that it has some sense of what is > "right" or "wrong", and some explanation for that rightness or wrongness > that doesn't change with the ebb and flow of popular opinion and the > current population of a mailing list. In this specific case, it seems that people had agree on "right" for a long time, and had just accepted that the current implementation is "wrong". You also agree to that, and many other long-time Python contributors have agreed. So as long as those people are around, it is unlikely that they change their minds again on this specific question. >>> So reject it, or propose to add a new API. >> Neither is a solution. Rejecting it means it will keep popping up >> forever. > > Like requests to remove whitespace sensitivity and add braces? No, unlike that. See above, plus the people contributing to Python believe that the current behavior is right (although the view on using tabs-vs-spaces has changed over time). In this case it's different: all long-time contributors seem to agree that the new behavior is the desirable one, on a green field. > In any case, my main concern about this change isn't whether it's right or > wrong -- it's about whether Python provides a stable platform for software > development with reasonable migration paths. *This* change won't actually > hurt *me* -- but what will the next change be? Must everyone who wants > some form of stability maintain a constant watch over Python's source changes? > > I gather that your answer is "yes", and that's what disturbs me here. No. I firmly believe that even with this change, Python provides "some form of stability". All the alternatives that had been proposed, except for the "never" variant, provide less stability. This is the most stable patch to solve the problem that I could think of. Regards, Martin From steve at holdenweb.com Thu Mar 15 19:35:39 2007 From: steve at holdenweb.com (Steve Holden) Date: Thu, 15 Mar 2007 14:35:39 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> References: <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> Message-ID: <45F991FB.20906@holdenweb.com> Phillip J. Eby wrote: > At 07:45 AM 3/15/2007 +0100, Martin v. L?wis wrote: >> I apparently took the same position that you now take back then, >> whereas I'm now leaning towards (or going beyond) the position >> Tim had back then, who wrote "BTW, if it *weren't* for the code breakage, >> I'd be in favor of doing this." > > If it weren't for the code breakage, I'd be in favor too. That's not the > point. > > The point is that how can Python be stable as a language if precedents can > be reversed without a migration plan, just because somebody changes their > mind? In another five years, will you change your mind again, and decide > to put this back the way it was? > > Speaking as a business person, that seems to me... unwise. When I found > out that this change had been checked in despite all the opposition, my gut > reaction was, "I guess I can't rely on Python any more", despite 10 years > of working with it, developing open source software with it, and > contributing to its development. Because from a *business* perspective, > this sort of flip-flopping means that moving from one "minor" Python > version to another is potentially *very* costly. > > The process of having warnings at least ensures that I can *discover* > whether my programs depend on some behavior that has changed - rather than > having something that used to work and now doesn't. > > >> I now believe that this should be done *despite* having been >> documented and tested (which, as you can see, was documented >> and tested only match the implemented behavior). That it keeps >> popping up is proof that the old behavior is deemed incorrect >> by many people. > > But as you are so fond of pointing out, there is no "many people". There > are only individual people. That a majority want it one way, means that > there is a minority who want it another. If next year, it becomes more > popular to have it the other way, will we switch again? If a majority of > people want braces and required type declarations, will we add them? > > After all, there is *substantial* support for some proposals along those lines! > > Yet, one of the appeals of Python is that it has some sense of what is > "right" or "wrong", and some explanation for that rightness or wrongness > that doesn't change with the ebb and flow of popular opinion and the > current population of a mailing list. > > IMO, Python is not -- or at least should not be -- a popularity contest. > > >>> So reject it, or propose to add a new API. >> Neither is a solution. Rejecting it means it will keep popping up >> forever. > > Like requests to remove whitespace sensitivity and add braces? > > That a request may keep popping up forever is not an argument for changing > it NOW. As Tim put it, "Never is often better than *right* now," and it > seems to me that this is *exactly* the sort of change for which that saying > was coined. > > >> The amount of Python code yet to be written is hopefully larger >> than the code already written (paraphrasing Guido), so in the long run, >> it should show the right behavior, not the historical one. > > Sure - but by that argument, the amount of code that will be written in 3.0 > or 3.1 is larger still, and if this behavior's been mostly okay for 9+ > years, then fixing it in a year or two should be quite prompt, if you want > to look at the historical scale. > > In any case, my main concern about this change isn't whether it's right or > wrong -- it's about whether Python provides a stable platform for software > development with reasonable migration paths. *This* change won't actually > hurt *me* -- but what will the next change be? Must everyone who wants > some form of stability maintain a constant watch over Python's source changes? > > I gather that your answer is "yes", and that's what disturbs me here. > The impact of this one little change certainly isn't the only issue at stake here. But as Mr. Creosote knows, even one little "wafer thin" change can lead to a chaotic transformation. Since 2.0 serious efforts have been made to maintain, and even promote, the stability and backwards compatibility of Python. This has benefited the language. This particular change looks like gratuitous breakage, no matter how sound the reasons for it, and putting it in to 2.6 with 3.0 "just around the corner" (though not for production purposes) is guaranteed to upset some people and cause adverse reaction. Now, you might feel (as Guido does) that it doesn't matter what people write in their blogs, but I personally want people to perceive Python as a language whose development is carefully managed. Consequently I am disturbed when a change of this nature is made and it becomes apparent that there is no consensus for it. This is not "prevarication", it's a serious discussion about how such issues should be managed. The current glaring lack is of a sound decision-making process. Such breakage-inducing change should be reserved for major versions (as was the fix to the socket addressing wart). regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From steve at holdenweb.com Thu Mar 15 19:35:39 2007 From: steve at holdenweb.com (Steve Holden) Date: Thu, 15 Mar 2007 14:35:39 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> References: <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> Message-ID: <45F991FB.20906@holdenweb.com> Phillip J. Eby wrote: > At 07:45 AM 3/15/2007 +0100, Martin v. L?wis wrote: >> I apparently took the same position that you now take back then, >> whereas I'm now leaning towards (or going beyond) the position >> Tim had back then, who wrote "BTW, if it *weren't* for the code breakage, >> I'd be in favor of doing this." > > If it weren't for the code breakage, I'd be in favor too. That's not the > point. > > The point is that how can Python be stable as a language if precedents can > be reversed without a migration plan, just because somebody changes their > mind? In another five years, will you change your mind again, and decide > to put this back the way it was? > > Speaking as a business person, that seems to me... unwise. When I found > out that this change had been checked in despite all the opposition, my gut > reaction was, "I guess I can't rely on Python any more", despite 10 years > of working with it, developing open source software with it, and > contributing to its development. Because from a *business* perspective, > this sort of flip-flopping means that moving from one "minor" Python > version to another is potentially *very* costly. > > The process of having warnings at least ensures that I can *discover* > whether my programs depend on some behavior that has changed - rather than > having something that used to work and now doesn't. > > >> I now believe that this should be done *despite* having been >> documented and tested (which, as you can see, was documented >> and tested only match the implemented behavior). That it keeps >> popping up is proof that the old behavior is deemed incorrect >> by many people. > > But as you are so fond of pointing out, there is no "many people". There > are only individual people. That a majority want it one way, means that > there is a minority who want it another. If next year, it becomes more > popular to have it the other way, will we switch again? If a majority of > people want braces and required type declarations, will we add them? > > After all, there is *substantial* support for some proposals along those lines! > > Yet, one of the appeals of Python is that it has some sense of what is > "right" or "wrong", and some explanation for that rightness or wrongness > that doesn't change with the ebb and flow of popular opinion and the > current population of a mailing list. > > IMO, Python is not -- or at least should not be -- a popularity contest. > > >>> So reject it, or propose to add a new API. >> Neither is a solution. Rejecting it means it will keep popping up >> forever. > > Like requests to remove whitespace sensitivity and add braces? > > That a request may keep popping up forever is not an argument for changing > it NOW. As Tim put it, "Never is often better than *right* now," and it > seems to me that this is *exactly* the sort of change for which that saying > was coined. > > >> The amount of Python code yet to be written is hopefully larger >> than the code already written (paraphrasing Guido), so in the long run, >> it should show the right behavior, not the historical one. > > Sure - but by that argument, the amount of code that will be written in 3.0 > or 3.1 is larger still, and if this behavior's been mostly okay for 9+ > years, then fixing it in a year or two should be quite prompt, if you want > to look at the historical scale. > > In any case, my main concern about this change isn't whether it's right or > wrong -- it's about whether Python provides a stable platform for software > development with reasonable migration paths. *This* change won't actually > hurt *me* -- but what will the next change be? Must everyone who wants > some form of stability maintain a constant watch over Python's source changes? > > I gather that your answer is "yes", and that's what disturbs me here. > The impact of this one little change certainly isn't the only issue at stake here. But as Mr. Creosote knows, even one little "wafer thin" change can lead to a chaotic transformation. Since 2.0 serious efforts have been made to maintain, and even promote, the stability and backwards compatibility of Python. This has benefited the language. This particular change looks like gratuitous breakage, no matter how sound the reasons for it, and putting it in to 2.6 with 3.0 "just around the corner" (though not for production purposes) is guaranteed to upset some people and cause adverse reaction. Now, you might feel (as Guido does) that it doesn't matter what people write in their blogs, but I personally want people to perceive Python as a language whose development is carefully managed. Consequently I am disturbed when a change of this nature is made and it becomes apparent that there is no consensus for it. This is not "prevarication", it's a serious discussion about how such issues should be managed. The current glaring lack is of a sound decision-making process. Such breakage-inducing change should be reserved for major versions (as was the fix to the socket addressing wart). regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From martin at v.loewis.de Thu Mar 15 19:39:16 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 15 Mar 2007 19:39:16 +0100 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: <20070315182523.7769.715364850.divmod.xquotient.887@joule.divmod.com> References: <20070315151236.18920.1909125356.divmod.quotient.2790@ohm> <45F9733F.7030409@v.loewis.de> <20070315182523.7769.715364850.divmod.xquotient.887@joule.divmod.com> Message-ID: <45F992D4.8090401@v.loewis.de> glyph at divmod.com schrieb: > Just in case it's not clear from the other things I've said: this is a > terrible, terrible idea, and I am shocked that it is even being > *considered* for inclusion in Python. As a foolish youth, I wasted many > months trying to get a program that used Java's (then not deprecated) > asynchronous exception APIs to behave properly. It wasn't possible > then, and it isn't possible now. Ok, I withdraw this SoC project idea. Regards, Martin From facundo at taniquetil.com.ar Thu Mar 15 19:42:53 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Thu, 15 Mar 2007 18:42:53 +0000 (UTC) Subject: [Python-Dev] Adding timeout to socket.py and httplib.py References: Message-ID: Facundo Batista wrote: > I studied Skip patch, and I think he is in good direction: add a > NetworkConnection object to socket.py, and then use it from the other > modules. As of discussion in the patch tracker, this class is now a function in socket.py. This function connect() does the connection to an address, and can receive, optionally, a timeout. > I opened a new patch (#1676823) with the changes I propose regarding > socket.py, because the goal from the two patches are different (my plan The timeout.diff in this patch was updated to reflect these changes. If nobody raises objections, I'll commit these changes. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From martin at v.loewis.de Thu Mar 15 19:50:34 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Mar 2007 19:50:34 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F991FB.20906@holdenweb.com> References: <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> Message-ID: <45F9957A.1080204@v.loewis.de> Steve Holden schrieb: > This is not "prevarication", it's a serious discussion about how such > issues should be managed. The current glaring lack is of a sound > decision-making process. Such breakage-inducing change should be > reserved for major versions (as was the fix to the socket addressing wart). Please take a look at Misc/NEWS and review all changes that had been made since 2.5 to find out what other changes are breakage-inducing. Regards, Martin From martin at v.loewis.de Thu Mar 15 19:58:04 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Mar 2007 19:58:04 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F991FB.20906@holdenweb.com> References: <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> Message-ID: <45F9973C.1030909@v.loewis.de> > This particular change looks like gratuitous breakage, no matter how > sound the reasons for it, and putting it in to 2.6 with 3.0 "just around > the corner" (though not for production purposes) is guaranteed to upset > some people and cause adverse reaction. > > This is not "prevarication", it's a serious discussion about how such > issues should be managed. The current glaring lack is of a sound > decision-making process. Such breakage-inducing change should be > reserved for major versions (as was the fix to the socket addressing wart). I just like to point out that I disagree with this classification. The change is not gratuitous breakage (it's neither gratuitous, nor is it breakage), nor is it breakage-inducing. Regards, Martin From glyph at divmod.com Thu Mar 15 20:17:49 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Thu, 15 Mar 2007 19:17:49 -0000 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> References: <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> Message-ID: <20070315191749.7769.1592757264.divmod.xquotient.985@joule.divmod.com> On 05:51 pm, pje at telecommunity.com wrote: >At 07:45 AM 3/15/2007 +0100, Martin v. L?wis wrote: >>I apparently took the same position that you now take back then, >>whereas I'm now leaning towards (or going beyond) the position >>Tim had back then, who wrote "BTW, if it *weren't* for the code >>breakage, >>I'd be in favor of doing this." > >If it weren't for the code breakage, I'd be in favor too. That's not >the >point. > >The point is that how can Python be stable as a language if precedents >can >be reversed without a migration plan, just because somebody changes >their >mind? In another five years, will you change your mind again, and >decide >to put this back the way it was? Hear, hear. Python is _not_ stable as a language. I have Java programs that I wrote almost ten years ago which still run perfectly on the latest runtime. There is python software I wrote two years ago which doesn't work right on 2.5, and some of the Python stuff contemporary with that Java code won't even import. >Speaking as a business person, that seems to me... unwise. When I >found >out that this change had been checked in despite all the opposition, my >gut >reaction was, "I guess I can't rely on Python any more", despite 10 >years >of working with it, developing open source software with it, and >contributing to its development. Because from a *business* >perspective, >this sort of flip-flopping means that moving from one "minor" Python >version to another is potentially *very* costly. And indeed it is. Python's advantages in terms of rapidity of development have, thus far, made up the difference for me, but it is threatening to become a close thing. This is a severe problem and something needs to be done about it. >But as you are so fond of pointing out, there is no "many people". >There >are only individual people. That a majority want it one way, means >that >there is a minority who want it another. If next year, it becomes more >popular to have it the other way, will we switch again? If a majority >of >people want braces and required type declarations, will we add them? And, in fact, there is not even a majority. There is a *perception* of a majority. There isn't even a *perception* of a majority of Python users, but a perception of a majority of python-dev readers, who are almost by definition less risk-averse when it comes to language change than anyone else! If we actually care about majorities, let's set up a voting application and allow Python users to vote on each and every feature, and publicize it each time such a debate comes up. Here, I'll get it started: http://jyte.com/cl/python-should-have-a-strict-backward-compatibility- policy-to-guide-its-development According to that highly scientific study, at this point in time, "Nobody disagrees" :). (One in favor, zero against.) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070315/c86a6948/attachment.html From guido at python.org Thu Mar 15 20:25:20 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Mar 2007 12:25:20 -0700 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: References: Message-ID: I need to shed load; I've asked Georg to review this. If he's fine with it, Facundo can check it in. On 3/15/07, Facundo Batista wrote: > Facundo Batista wrote: > > > I studied Skip patch, and I think he is in good direction: add a > > NetworkConnection object to socket.py, and then use it from the other > > modules. > > As of discussion in the patch tracker, this class is now a function in > socket.py. > > This function connect() does the connection to an address, and can > receive, optionally, a timeout. > > > > I opened a new patch (#1676823) with the changes I propose regarding > > socket.py, because the goal from the two patches are different (my plan > > The timeout.diff in this patch was updated to reflect these changes. > > If nobody raises objections, I'll commit these changes. > > Regards, > > -- > . Facundo > . > Blog: http://www.taniquetil.com.ar/plog/ > PyAr: http://www.python.org/ar/ > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mbk.lists at gmail.com Thu Mar 15 21:10:55 2007 From: mbk.lists at gmail.com (Mike Krell) Date: Thu, 15 Mar 2007 13:10:55 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F99200.7040303@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> Message-ID: On 3/15/07, "Martin v. L?wis" wrote: > ... the majority of the people polled thought that it ought to be fixed. Personally, I didn't respond to your "poll" because I didn't think this particular issue would come down to a silly head count of self-selecting responders. When I first needed to use splitext in my code, I tested the relevant corner case in question at the interactive prompt. I also read the docstring which explicitly documented the behavior. I then wrote my code accordingly. Knowing that this was well-defined and documented behavior and having followed this list during previous backward compatibility discussions, I "knew" that there was no way your proposed patch would make it into a minor release because many long-time active developers would rightfully point out that it gratuitously breaks code. In your radical departure from the common-sense approach to code-breaking changes that typically prevails here, you proved me wrong. So now I'm speaking up. FWIW, I agree completely with PJE's and glyph's remarks with respect to expectations of stability, especially in a minor release. Sorry, updating the NEWS file isn't good enough, because as has been amply demonstrated here, many people cannot be bothered to read the documentation. +10000 on reverting the patch and not punishing those users who bothered to the documentation or test the corner cases themselves. Mike From martin at v.loewis.de Thu Mar 15 21:13:37 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Mar 2007 21:13:37 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> Message-ID: <45F9A8F1.6070209@v.loewis.de> Mike Krell schrieb: > When I first needed to use splitext in my code, I tested the relevant > corner case in question at the interactive prompt. I also read the > docstring which explicitly documented the behavior. I then wrote my > code accordingly. Can you show us the relevant fragment of your code? Regards, Martin From martin at v.loewis.de Thu Mar 15 21:21:13 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Mar 2007 21:21:13 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> Message-ID: <45F9AAB9.4040408@v.loewis.de> Mike Krell schrieb: > FWIW, I agree completely with PJE's and glyph's remarks with respect > to expectations of stability, especially in a minor release. Not sure what you mean by "minor release". The change isn't proposed for the next bug fix release (2.5.1), but for the next major release (2.6). See PEP 6. Regards, Martin From tjreedy at udel.edu Thu Mar 15 21:24:25 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 15 Mar 2007 16:24:25 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) References: <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com><5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com><5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com><5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com><5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com><5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> Message-ID: "Phillip J. Eby" wrote in message news:5.1.1.6.0.20070315105608.02ba5f58 at sparrow.telecommunity.com... > The process of having warnings at least ensures that I can *discover* > whether my programs depend on some behavior that has changed - rather > than > having something that used to work and now doesn't. I am not familiar with the warning system, but it seems plausible that one could add to the end of .splitext (before it returns) an optional warning something like if not ext and base[0] == '.': warn("Before 2.6, this would have returned (%s,%s) instead of (%s,%s)" % (ext, base, base, ext)) where base and ext have the obvious contents. Is this what you want? Terry Jan Reedy From jeremy at alum.mit.edu Thu Mar 15 21:43:10 2007 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Thu, 15 Mar 2007 16:43:10 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <20070315191749.7769.1592757264.divmod.xquotient.985@joule.divmod.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <20070315191749.7769.1592757264.divmod.xquotient.985@joule.divmod.com> Message-ID: On 3/15/07, glyph at divmod.com wrote: > On 05:51 pm, pje at telecommunity.com wrote: > >At 07:45 AM 3/15/2007 +0100, Martin v. L?wis wrote: > >>I apparently took the same position that you now take back then, > >>whereas I'm now leaning towards (or going beyond) the position > >>Tim had back then, who wrote "BTW, if it *weren't* for the code breakage, > >>I'd be in favor of doing this." > > > >If it weren't for the code breakage, I'd be in favor too. That's not the > >point. > > > >The point is that how can Python be stable as a language if precedents can > >be reversed without a migration plan, just because somebody changes their > >mind? In another five years, will you change your mind again, and decide > >to put this back the way it was? > > Hear, hear. Python is _not_ stable as a language. I have Java programs > that I wrote almost ten years ago which still run perfectly on the latest > runtime. There is python software I wrote two years ago which doesn't work > right on 2.5, and some of the Python stuff contemporary with that Java code > won't even import. I think the problem has less to do with bug fixing than with lack of any clear specifications or documentation about what developers can depend on. You could probably make a case that any change that doesn't fix a crash bug is likely to cause some particular program to behave differently. Take bug 1504333 which lead to a change in sgmllib behavior for angle brackets in quoted attribute values. Did the sgmllib documentation explain that the fixed behavior was incorrect? Might a programmer working with sgmllib have written code that depended on this bug? Do you object to this bug fix? For many of these bugs, some people will have written code against the documentation and some people against the implementation or behavior. (In this case, the documentation is vague or conflicting.) I don't think I know how to balance the important of these two classes of users. Some code is going to break the first time they run into the under-specific edge case, some code is going to break when the specification and implementation are clarified. You have to weigh which you think is more likely and which will benefit users the most. I think everyone wants to do the "right thing" by Python's users, but it's not clear what that right thing is. > >Speaking as a business person, that seems to me... unwise. When I found > >out that this change had been checked in despite all the opposition, my gut > >reaction was, "I guess I can't rely on Python any more", despite 10 years > >of working with it, developing open source software with it, and > >contributing to its development. Because from a *business* perspective, > >this sort of flip-flopping means that moving from one "minor" Python > >version to another is potentially *very* costly. > > And indeed it is. Python's advantages in terms of rapidity of development > have, thus far, made up the difference for me, but it is threatening to > become a close thing. This is a severe problem and something needs to be > done about it. Could you point out a few such programs that people on python-dev can look at? I think it would be useful to gather some data about the kind of migration pains real users are having. I believe Martin and others are trying to do the right thing. Real data is more likely to convince them than passionate arguments on python-dev. > >But as you are so fond of pointing out, there is no "many people". There > >are only individual people. That a majority want it one way, means that > >there is a minority who want it another. If next year, it becomes more > >popular to have it the other way, will we switch again? If a majority of > >people want braces and required type declarations, will we add them? > > And, in fact, there is not even a majority. There is a *perception* of a > majority. There isn't even a *perception* of a majority of Python users, > but a perception of a majority of python-dev readers, who are almost by > definition less risk-averse when it comes to language change than anyone > else! I think you missed the point here. The hypothetical question was not about any particular majority, but rather that regardless of which group you poll, the majority decision may not be the right one. Even a majority of Twised users :-). Jeremy > If we actually care about majorities, let's set up a voting application and > allow Python users to vote on each and every feature, and publicize it each > time such a debate comes up. Here, I'll get it started: > http://jyte.com/cl/python-should-have-a-strict-backward-compatibility-policy-to-guide-its-development > > According to that highly scientific study, at this point in time, "Nobody > disagrees" :). (One in favor, zero against.) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu > > From mbk.lists at gmail.com Thu Mar 15 21:47:48 2007 From: mbk.lists at gmail.com (Mike Krell) Date: Thu, 15 Mar 2007 13:47:48 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F9A8F1.6070209@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> Message-ID: On 3/15/07, "Martin v. L?wis" wrote: > Can you show us the relevant fragment of your code? Sure: for f in files: try: (root, ext) = os.path.splitext(f) os.rename(f, '%s.%s%s' % (root, index, ext)) except OSError: die('renaming %s failed' % f) Background: This is a little utility that runs on windows that archives arbitrary files. index is an integer. For index == 1, I want "a.txt" to be renamed to "a.1.txt", and I want ".emacs" to be renamed to ".1.emacs", thus preserving the extensions. Under the new patch, the second file would be renamed to ".emacs.1", gratuitously breaking the extension preservation. Mike From glyph at divmod.com Thu Mar 15 21:57:36 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Thu, 15 Mar 2007 20:57:36 -0000 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F9AAB9.4040408@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9AAB9.4040408@v.loewis.de> Message-ID: <20070315205736.7769.1178617927.divmod.xquotient.1001@joule.divmod.com> On 08:21 pm, martin at v.loewis.de wrote: >Mike Krell schrieb: >>FWIW, I agree completely with PJE's and glyph's remarks with respect >>to expectations of stability, especially in a minor release. > >Not sure what you mean by "minor release". The change isn't proposed >for the next bug fix release (2.5.1), but for the next major release >(2.6). See PEP 6. Common parlance for the parts of a version number is: major.minor.micro See: http://twistedmatrix.com/documents/current/api/twisted.python.versions.Version.html#__init__ Changing this terminology about Python releases to be more consistent with other projects would be a a subtle, but good shift towards a generally better attitude of the expectations of "minor" releases. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070315/a88224d5/attachment.htm From g.brandl at gmx.net Thu Mar 15 21:59:24 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 15 Mar 2007 21:59:24 +0100 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: References: Message-ID: I'll review it tomorrow. Georg Guido van Rossum schrieb: > I need to shed load; I've asked Georg to review this. If he's fine > with it, Facundo can check it in. > > On 3/15/07, Facundo Batista wrote: >> Facundo Batista wrote: >> >> > I studied Skip patch, and I think he is in good direction: add a >> > NetworkConnection object to socket.py, and then use it from the other >> > modules. >> >> As of discussion in the patch tracker, this class is now a function in >> socket.py. >> >> This function connect() does the connection to an address, and can >> receive, optionally, a timeout. >> >> >> > I opened a new patch (#1676823) with the changes I propose regarding >> > socket.py, because the goal from the two patches are different (my plan >> >> The timeout.diff in this patch was updated to reflect these changes. >> >> If nobody raises objections, I'll commit these changes. >> >> Regards, >> >> -- >> . Facundo >> . >> Blog: http://www.taniquetil.com.ar/plog/ >> PyAr: http://www.python.org/ar/ >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > From g.brandl at gmx.net Thu Mar 15 22:03:32 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 15 Mar 2007 22:03:32 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F9957A.1080204@v.loewis.de> References: <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> Message-ID: Martin v. L?wis schrieb: > Steve Holden schrieb: >> This is not "prevarication", it's a serious discussion about how such >> issues should be managed. The current glaring lack is of a sound >> decision-making process. Such breakage-inducing change should be >> reserved for major versions (as was the fix to the socket addressing wart). > > Please take a look at Misc/NEWS and review all changes that had been > made since 2.5 to find out what other changes are breakage-inducing. For example, I committed a fix for urllib that made it raise IOError instead of an AttributeError (which wasn't explicitly raised, of course) if a certain error condition occurs. This is changed behavior too, but if we are to postpone all these fixes to 3.0, we won't have half of the fixes in Python 2.6 that are there now. Georg From g.brandl at gmx.net Thu Mar 15 22:06:37 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 15 Mar 2007 22:06:37 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F99200.7040303@v.loewis.de> References: <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> Message-ID: Martin v. L?wis schrieb: >> The process of having warnings at least ensures that I can *discover* >> whether my programs depend on some behavior that has changed - rather than >> having something that used to work and now doesn't. > > So you would agree to the change if a warning was generated at run-time? > > Notice that there is already a warning there: the documentation clearly > states that the behavior has changed, and, of course, Misc/NEWS lists > it as changed behavior. As a sidenote, this item should be included in the 2.6 "What's new"'s "porting" section. Perhaps it would be a good "policy" to automatically list potentially breaking fixes there instead of rolling off that task to Andrew. Georg From brett at python.org Thu Mar 15 22:17:44 2007 From: brett at python.org (Brett Cannon) Date: Thu, 15 Mar 2007 14:17:44 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F9973C.1030909@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9973C.1030909@v.loewis.de> Message-ID: On 3/15/07, "Martin v. L?wis" wrote: > > This particular change looks like gratuitous breakage, no matter how > > sound the reasons for it, and putting it in to 2.6 with 3.0 "just around > > the corner" (though not for production purposes) is guaranteed to upset > > some people and cause adverse reaction. > > > > This is not "prevarication", it's a serious discussion about how such > > issues should be managed. The current glaring lack is of a sound > > decision-making process. Such breakage-inducing change should be > > reserved for major versions (as was the fix to the socket addressing wart). > > I just like to point out that I disagree with this classification. The > change is not gratuitous breakage (it's neither gratuitous, nor is it > breakage), nor is it breakage-inducing. > First off, I should say I totally agree with Martin's thinking in this whole matter. If I had been in his situation there is a good chance I would have done what he did based on prior history of when underspecified stuff has what could considered poor behaviour. But the key point I want to get across is people should not being getting mad at Martin. The people who are getting all bent out of shape over this should be upset at python-dev as a whole for not having a clear policy on this sort of thing. Martin just happened to be the guy who made a change that sparked this and he is explaining his thinking behind it (which also happens to mirror my thinking on this whole situation). It could have easily been someone else. But since Martin does so much work clearing out patches (and we all owe Martin and everyone else who consistently tries to close bugs and patches a HUGE thank you). I am sorry it happened to be Martin, but I also think he has done a great job keeping his composure in this as I would have lost my top at by now had I not been ignoring this thread. And I would hope that people are not explicitly mad at Martin (I suspect people aren't). But from my viewpoint people are getting the point of yelling and that is not going to get us anywhere. Bottom line, let's work together as a group to come up with a policy in a civil, positive manner (in a new thread!) and let the result of that decision guide what is done with this fix. Yelling at poor Martin about one patch when we could be spending this effort on trying to discuss what kind of policy we want is not getting us anywhere. -Brett From martin at v.loewis.de Thu Mar 15 22:24:31 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Mar 2007 22:24:31 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> Message-ID: <45F9B98F.5060102@v.loewis.de> Mike Krell schrieb: > Sure: > > for f in files: > try: > (root, ext) = os.path.splitext(f) > os.rename(f, '%s.%s%s' % (root, index, ext)) > except OSError: > die('renaming %s failed' % f) Thanks! Looking more closely, it's not entirely clear where index comes from - what if you already have "a.1.txt". Will you set it to 2? Will that then produce a.1.2.txt? > This is a little utility that runs on windows that archives arbitrary > files. index is an integer. > For index == 1, I want "a.txt" to be renamed to "a.1.txt", and I want > ".emacs" to be renamed to ".1.emacs", thus preserving the extensions. > Under the new patch, the second file would be renamed to ".emacs.1", > gratuitously breaking the extension preservation. I can see that it breaks the behavior you intended it to have. However, I disagree that it broke "extension preservation". Rather, it *provides* extension preservation, something that the old code did not do. I also like to point out that the primary objective of the code ("archive arbitrary files") is still preserved - it still does that, but in different manner. (disclaimer: I don't fully understand the index part) Regards, Martin From apt.shansen at gmail.com Thu Mar 15 22:28:26 2007 From: apt.shansen at gmail.com (Stephen Hansen) Date: Thu, 15 Mar 2007 14:28:26 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> Message-ID: <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> > For example, I committed a fix for urllib that made it raise IOError > instead > of an AttributeError (which wasn't explicitly raised, of course) if a > certain > error condition occurs. > > This is changed behavior too, but if we are to postpone all these fixes > to 3.0, we won't have half of the fixes in Python 2.6 that are there now. There's a big difference between that change and this one; that change is 'loud'. It makes noise. It's raising an exception: that exception will either be handled or will propagate up the stack and be noticed somewhere. I *think* (ahem.. I read minds...) the problem people are having with this particular change is the fact that the behavior of this function is being changed in a way that is completely silent. Code written to expect one kind of result are now getting a different kind of result... instead of having an error thrown, a warning given, or something explicit... it's just different now. And it'd be so easy to do it in a way which wouldn't be silent... just throw out a warning, and defer the actual change until the next release. Expecting people to keep on top of Misc/NEWS and re-read the documentation for every function in their code is a tad unreasonable. I don't personally find it unreasonable for people to ask for a bit more of an extended migration path when changes that are being implemented will cause *silent* changes in behavior. It's been very hard for my company to move from 2.3 to 2.4 as a development platform as it is, which we're just barely doing now... for this reason I'm paying a lot more attention to -dev lately to be prepared for 2.6 and beyond. Not everyone has the time to do that.. there's a lot of messages :) And Misc/NEWS is *huge*. Warnings are a very useful mechanism for semi-painless migrations and upgrades... (And, if I thought it'd have any chance of going in, I'd submit a patch to add a warning and adjust docs/tests/etc... but this issue seems ever so divided...) --S -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070315/4924c086/attachment-0001.html From martin at v.loewis.de Thu Mar 15 22:29:23 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 15 Mar 2007 22:29:23 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <20070315205736.7769.1178617927.divmod.xquotient.1001@joule.divmod.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9AAB9.4040408@v.loewis.de> <20070315205736.7769.1178617927.divmod.xquotient.1001@joule.divmod.com> Message-ID: <45F9BAB3.9090907@v.loewis.de> glyph at divmod.com schrieb: > >Not sure what you mean by "minor release". The change isn't proposed > >for the next bug fix release (2.5.1), but for the next major release > >(2.6). See PEP 6. > > Common parlance for the parts of a version number is: > major.minor.micro > See: > > http://twistedmatrix.com/documents/current/api/twisted.python.versions.Version.html#__init__ > > Changing this terminology about Python releases to be more consistent > with other projects would be a a subtle, but good shift towards a > generally better attitude of the expectations of "minor" releases. When PEP 6 was originally written, it said "feature release", and "bug fix release". This was then changed at some point (too lazy to look up subversion log now) to say "major release" and "bugfix release", indicating that the major releases (in the sense of the common expectation) *are* the 2.x releases. At that time, it wasn't clear whether there ever would be a 3.0 release. This is where my understanding of policy comes from: bug fix releases are for bug fixes *only*, major releases can add new features, and correct problems that may break existing applications (using parallel APIs, warnings, etc, as appropriate). Regards, Martin From martin at v.loewis.de Thu Mar 15 22:31:06 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Mar 2007 22:31:06 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> Message-ID: <45F9BB1A.30309@v.loewis.de> Georg Brandl schrieb: > As a sidenote, this item should be included in the 2.6 "What's new"'s "porting" > section. > > Perhaps it would be a good "policy" to automatically list potentially breaking > fixes there instead of rolling off that task to Andrew. I would do that, except that Andrew explicitly reserved the right to change whatsnew.tex. I believe he does go over Misc/NEWS in doing so. Regards, Martin From g.brandl at gmx.net Thu Mar 15 22:38:26 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 15 Mar 2007 22:38:26 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> Message-ID: Stephen Hansen schrieb: > And it'd be so easy to do it in a way which wouldn't be silent... just > throw out a warning, and defer the actual change until the next release. > > Expecting people to keep on top of Misc/NEWS and re-read the > documentation for every function in their code is a tad unreasonable. I > don't personally find it unreasonable for people to ask for a bit more > of an extended migration path when changes that are being implemented > will cause *silent* changes in behavior. Which is why there is the "Porting" section in the What's New document, which is typically not longer than a page and should list these changes. Georg From martin at v.loewis.de Thu Mar 15 22:39:19 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 15 Mar 2007 22:39:19 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> Message-ID: <45F9BD07.2010605@v.loewis.de> Stephen Hansen schrieb: > And it'd be so easy to do it in a way which wouldn't be silent... just > throw out a warning, and defer the actual change until the next release. I disagree that it easy to do that. Implementation-wise, it probably is. However, I feel that warnings have a *very* high cost, and cause more harm than good. They primarily don't appear at the machine of the author of the code. Instead, they appear at the end-user's machine, causing them problems in that they see messages from a software they didn't know they are even running. They all have to chase down the source of the program, only to eventually have the author of the software to tell them that it is safe to ignore this warning (which I believe it is). I specifically got complaints that Python fills Apache log files with garbage warnings. It won't help to issue the warning only once (although that will help in other cases): you then get the warning once per HTTP request (assuming CGI), which still can be a lot of warnings. That said, if it makes people more comfortable with having a warning added, I won't object. It's just that I don't want to be the one to take the blame for issuing the warning, because deep in my heart I feel that warnings are a bad thing, unless they are right most of the time (which they won't be in this case). > (And, if I thought it'd have any chance of going in, I'd submit a patch > to add a warning and adjust docs/tests/etc... but this issue seems ever > so divided...) You need to find a committer to commit such a change, but otherwise, I think it's a good idea. Contributing is always a good idea. Martin From glyph at divmod.com Thu Mar 15 22:46:17 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Thu, 15 Mar 2007 21:46:17 -0000 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <20070315191749.7769.1592757264.divmod.xquotient.985@joule.divmod.com> Message-ID: <20070315214617.7769.191452689.divmod.xquotient.1091@joule.divmod.com> On 08:43 pm, jeremy at alum.mit.edu wrote: >On 3/15/07, glyph at divmod.com wrote: >>On 05:51 pm, pje at telecommunity.com wrote: >> >At 07:45 AM 3/15/2007 +0100, Martin v. L?wis wrote: >> >>I apparently took the same position that you now take back then, >> >>whereas I'm now leaning towards (or going beyond) the position >> >>Tim had back then, who wrote "BTW, if it *weren't* for the code >>breakage, >> >>I'd be in favor of doing this." >> > >> >If it weren't for the code breakage, I'd be in favor too. That's not >>the >> >point. >> > >> >The point is that how can Python be stable as a language if >>precedents can >> >be reversed without a migration plan, just because somebody changes >>their >> >mind? In another five years, will you change your mind again, and >>decide >> >to put this back the way it was? >> >>Hear, hear. Python is _not_ stable as a language. I have Java >>programs >>that I wrote almost ten years ago which still run perfectly on the >>latest >>runtime. There is python software I wrote two years ago which doesn't >>work >>right on 2.5, and some of the Python stuff contemporary with that Java >>code >>won't even import. > >I think the problem has less to do with bug fixing than with lack of >any clear specifications or documentation about what developers can >depend on. You could probably make a case that any change that >doesn't fix a crash bug is likely to cause some particular program to >behave differently. Absolutely. One of the reasons I'm working on documenting this process is that, of course, *everything* can't be made compatible. The mere act of adding a function or module adds a detectable change in behavior that some program *might* insanely depend on. A clear understanding of what is meant by "backwards compatibility" is equally important to developers trying to future-proof their code as it is to those trying to make sure they don't break code which has been future-proofed. This is a form of social contract, and both sides need to know about it. >Take bug 1504333 which lead to a change in sgmllib behavior for angle >brackets in quoted attribute values. Did the sgmllib documentation >explain that the fixed behavior was incorrect? Might a programmer >working with sgmllib have written code that depended on this bug? Do >you object to this bug fix? I don't know enough about the specification to say for sure, but I suspect that it is a legitimate bug fix, because sgmllib is implementing an externally-determined spec. In cases where the spec is flagrantly violated, it seems like it should be fixed to adhere to it. >For many of these bugs, some people will have written code against the >documentation and some people against the implementation or behavior. >(In this case, the documentation is vague or conflicting.) I don't >think I know how to balance the important of these two classes of >users. Some code is going to break the first time they run into the >under-specific edge case, some code is going to break when the >specification and implementation are clarified. You have to weigh >which you think is more likely and which will benefit users the most. If the documentation is vague and conflicting, then it seems likely that a parsing option could be added. I am not advocating perfect, 100% backwards compatibility, merely some standards for what happens when a (potentially) incompatible change is made. For example, you could add a flag to the parser which tweaks the treatment of quoted angle brackets, and warn if the argument is not passed that the default will change in the future (or, better yet, that the argument will be required in the future). Or, you could provide a separate name for invoking the different behavior. >I think everyone wants to do the "right thing" by Python's users, but >it's not clear what that right thing is. I really think that starting with the "golden rule" would be a good idea. Would Python core developers mind if something analogous in the C runtime changed? How would they cope with it? What kind of feedback would you expect the C compiler or runtime to provide in such a case? Python should do unto others, etc. >Could you point out a few such programs that people on python-dev can >look at? I think it would be useful to gather some data about the >kind of migration pains real users are having. I believe Martin and >others are trying to do the right thing. Real data is more likely to >convince them than passionate arguments on python-dev. (I assume you're responding to my other comment about my programs not running, even though that's not what you quoted.) I don't think these programs would contribute much to the discussion. I've probably got them archived somewhere, but they were broken circa 2.3 and I don't think I've run them since. I doubt they would make any sense to anyone here, and we would all get into a heated debate as to whether my usage of Python was valid or not (hint: it was *REALLY* gross). In fact, let's back up a step. These programs were never released as open-source. Pretend for a moment that it is beyond my power to reveal the highly proprietary for loops and if statements contained therein to the general public. Why should python-dev get to see my source code? Microsoft doesn't have access to millions of enterprises' code all over the world, and yet, they manage to provide a (relatively) stable platform for developing such code. The Mozilla foundation doesn't receive sources from Adobe, but the flash plugin manages to run on their browser for years at a time with no change. Sun does not require IBM reveal their source code to keep WebSphere working with new releases of Java. When was the last time you needed to approach the Linux kernel mailing list with some of your code to discuss their rapid evolution which was breaking all your applications? I am sympathetic to the Python team's desire to evolve and collaborate with the community, so I am overstating the case a bit here. I think that this experiment in a platform that actually has meaningful changes and deprecations going on that require developer participation is interesting, and probably the way that all software will eventually be moving. My point is that Python is going to be operating at a massive disadvantage to other languages and environments by being a living, dynamic, evolving environment in the first place. To combat that, it needs to be as conservative as possible about the types of change that are accepted. The process of evolution needs to be tightly managed, and very good, highly explicit channels of communication are necessary in the code, on the website, in the releases, on the mailing lists, and so on. I keep hearing the "NEWS" file bandied about as the solution here, and that isn't going to cut it. The change that caused the most angst to fix in Twisted for Python 2.5 was buried on line 751 of the NEWS file, in the "what's new" for one of the betas. We discovered it because of our test coverage, we would never have found it in the NEWS file. There were HUNDREDS of other issues in that file, many of which have absolutely no observable effect on the semantics of _any_ working python program, like minor fixes for obvious bugs like memory-management issues. These are mixed in with no prioritization amongst and zingers like new-style exceptions and id()'s return value changing. Of course, there is *some* code I can disclose. Have a look here: http://twistedmatrix.com/trac/ticket/1842#comment:5 Migrating Twisted from Python *2.4* to *2.5* took over a week, and several people. There were 10 separate issues that needed to be fixed. This is very recent, very active code, that had already had some preparation from a few people following beta releases, not some gnarly old legacy application. >I think you missed the point here. The hypothetical question was not >about any particular majority, but rather that regardless of which >group you poll, the majority decision may not be the right one. Even >a majority of Twised users :-). Erm... perhaps I was insufficiently clear. My comments there were that this is yet another example of letting muddy thinking and vague feelings drive the development process. If the development process were democratic (and let me be crystal clear, I think that democratic development is a _VERY BAD_ idea) then at least it would be an explicit process, not just one guy thinking that maybe a lot of people like a particular change. >>According to that highly scientific study, at this point in time, >>"Nobody >>disagrees" :). (One in favor, zero against.) (And *this* was just a joke...) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070315/0e5abe6d/attachment-0001.htm From glyph at divmod.com Thu Mar 15 22:54:28 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Thu, 15 Mar 2007 21:54:28 -0000 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9973C.1030909@v.loewis.de> Message-ID: <20070315215428.7769.91408526.divmod.xquotient.1105@joule.divmod.com> On 09:17 pm, brett at python.org wrote: >But the key point I want to get across is people should not being >getting mad at Martin. The people who are getting all bent out of >shape over this should be upset at python-dev as a whole for not >having a clear policy on this sort of thing. Martin just happened to >be the guy who made a change that sparked this and he is explaining >his thinking behind it (which also happens to mirror my thinking on >this whole situation). It could have easily been someone else. On part of this point, I have to agree. Nullum crimen, nulla poena sine praevia lege poenali. However, the decision was a bad one regardless of the existing policy, and sets a bad precedent while we are discussing this policy. I could be wrong, but I think it would be reasonable to assume that if Martin strongly supports such a change, Martin would oppose a policy which would strictly forbid such changes, and it is just such a policy that Python needs. >Bottom line, let's work together as a group to come up with a policy >in a civil, positive manner (in a new thread!) and let the result of >that decision guide what is done with this fix. Yelling at poor >Martin about one patch when we could be spending this effort on trying >to discuss what kind of policy we want is not getting us anywhere. I *am* working on that on the side and I hope to have something coherent and whole to present here, in that different thread, very soon. The point, for me, of participating in *this* thread is (A) to continue to keep the issue high-visibility, because in my opinion, compatibility policy is _THE_ issue that python-dev needs to deal with now, (B) to deal with the aforementioned strongly implied opposition to such a policy, and (C) last but not least, to actually get the patch reverted, since, while it is not the larger problem, it is, itself, a problem that needs to be fixed. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070315/d65c11fa/attachment.html From mbk.lists at gmail.com Thu Mar 15 22:56:52 2007 From: mbk.lists at gmail.com (Mike Krell) Date: Thu, 15 Mar 2007 14:56:52 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F9B98F.5060102@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> <45F9B98F.5060102@v.loewis.de> Message-ID: On 3/15/07, "Martin v. L?wis" wrote: > Mike Krell schrieb: > > Sure: > > > > for f in files: > > try: > > (root, ext) = os.path.splitext(f) > > os.rename(f, '%s.%s%s' % (root, index, ext)) > > except OSError: > > die('renaming %s failed' % f) > > Thanks! Looking more closely, it's not entirely clear where index > comes from - what if you already have "a.1.txt". Will you set it > to 2? Will that then produce a.1.2.txt? A bit more background: This runs periodically in a setting where the files in the file list are regenerated in between invocations of this code. Each time a renaming occurs, the index is incremented (it is preserved in a file in between invocations). Thus, various incarnations of "a.txt" would be archived as "a.1.txt", "a.2.txt", etc. Similarly, copies of ".emacs" would be made as ".1.emacs", ".2.emacs", etc. If "b.1.txt" appeared in the list of files to be archived, it would be archived as "b.1.1.txt", "b.1.2.txt", etc. > > Under the new patch, [.emacs] would be renamed to ".emacs.1", > > gratuitously breaking the extension preservation. > > I can see that it breaks the behavior you intended it to have. However, > I disagree that it broke "extension preservation". Rather, it *provides* > extension preservation, something that the old code did not do. Here is a point of confusion. Bear in mind I'm running this under windows, so explorer happily reports that ".emacs" has a type of "emacs". (In windows, file types are registered in the system based on the extension -- all the characters following the last dot. An unregistered extension is listed as its own type. Thus files ending in .txt are listed as type "Text Document", but files ending in ".emacs" are listed as type "emacs" because it's an unregistered extension.) I often sort files in the explorer based on type, and I want a file and all its backups to appear next to each other in such a sorted list. That's exactly why I rename the files the way I do. Thus, ".1.emacs" is what I want, and ".emacs.1" is a markedly inferior and unacceptable alternative. That's what I'm referring to by extension preservation. > I also like to point out that the primary objective of the code > ("archive arbitrary files") is still preserved - it still does that, > but in different manner. (disclaimer: I don't fully understand the > index part) See above. BTW, I want to echo Brett Cannon's comments about the tone. I've been a bit testy about this breakage, however, upon reflection, it's clear that it's not Martin's fault, but rather a shortcoming of the process. Sorry, Martin. If you or anyone else was offended, please accept my apologies. Mike From mike.klaas at gmail.com Thu Mar 15 23:09:04 2007 From: mike.klaas at gmail.com (Mike Klaas) Date: Thu, 15 Mar 2007 15:09:04 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> <45F9B98F.5060102@v.loewis.de> Message-ID: <3d2ce8cb0703151509g1de38c1evaf080d3a94ec08e2@mail.gmail.com> On 3/15/07, Mike Krell wrote: > Here is a point of confusion. Bear in mind I'm running this under > windows, so explorer happily reports that ".emacs" has a type of > "emacs". (In windows, file types are registered in the system based > on the extension -- all the characters following the last dot. An > unregistered extension is listed as its own type. Thus files ending > in .txt are listed as type "Text Document", but files ending in > ".emacs" are listed as type "emacs" because it's an unregistered > extension.) Unix-derived files prepended with a dot (like .emacs) are not meant to be interpreted as a "file type". It may be useful on occasion when using windows, but it certainly is not the intent of a dotfile. The following files reside in my /tmp: .X0-lock .X100-lock .X101-lock .X102-lock .X103-lock .X104-lock .X105-lock .X106-lock .X11-unix .X99-lock ...which are certainly not all unnamed files of different type. > I often sort files in the explorer based on type, and I want a file > and all its backups to appear next to each other in such a sorted > list. That's exactly why I rename the files the way I do. > Thus, ".1.emacs" is what I want, and ".emacs.1" is a markedly inferior > and unacceptable alternative. That's what I'm referring to by > extension preservation. Unacceptable? You code fails in (ISTM) the more common case of an extensionless file. -Mike From mbk.lists at gmail.com Thu Mar 15 23:16:11 2007 From: mbk.lists at gmail.com (Mike Krell) Date: Thu, 15 Mar 2007 15:16:11 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <3d2ce8cb0703151509g1de38c1evaf080d3a94ec08e2@mail.gmail.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> <45F9B98F.5060102@v.loewis.de> <3d2ce8cb0703151509g1de38c1evaf080d3a94ec08e2@mail.gmail.com> Message-ID: On 3/15/07, Mike Klaas wrote: > Unacceptable? You code fails in (ISTM) the more common case of an > extensionless file. I'm well aware of that limitation. However, what seems to you as a more common case is, in the context of this particular application, a case that never occurs. I wrote the code with my particular use cases in mind. Mike From greg.ewing at canterbury.ac.nz Fri Mar 16 00:27:48 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Mar 2007 12:27:48 +1300 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: References: <20070315151236.18920.1909125356.divmod.quotient.2790@ohm> <45F9733F.7030409@v.loewis.de> Message-ID: <45F9D674.70601@canterbury.ac.nz> Facundo Batista wrote: > are there processors that support "reentrant" interrupts? The PDP11 had seven priority levels for interrupts. When an interrupt was handled, interrupts with priorities less than or equal to the current level were blocked, but the handler could be interrupted by a higher priority interrupt. Also, on any processor I know about, there's nothing to stop an interrupt handler re-enabling interrupts once it's ensured that the particular one it's handling isn't going to happen again. You can use this to implement an interrupt priority scheme in software if the hardware doesn't support it. So yes, re-entrant interrupts do make sense in some situations. The thing to model this on, I think, would be the BSD sigmask mechanism, which lets you selectively block certain signals to create a critical section of code. A context manager could be used to make its use easier and less error-prone (i.e. harder to block async exceptions and then forget to unblock them). > But, in that scenario, should be a way to say to the thread "Die, die > now, no matter what"? Unconditionally killing a whole process is no big problem because all the resources it's using get cleaned up by the OS, and the effect on other processes is minimal and well-defined (pipes and sockets get EOF, etc.). But killing a thread can leave the rest of the program in an awkward state. I'm inclined to think that there should be some way to do it, and any locks held by the killed thread should be broken. It's then up to the program to deal with the consequences. If it's not willing to do that, then it shouldn't use the instant-death mechanism. -- Greg From martin at v.loewis.de Fri Mar 16 00:34:35 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 16 Mar 2007 00:34:35 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <20070315215428.7769.91408526.divmod.xquotient.1105@joule.divmod.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9973C.1030909@v.loewis.de> <20070315215428.7769.91408526.divmod.xquotient.1105@joule.divmod.com> Message-ID: <45F9D80B.7050902@v.loewis.de> glyph at divmod.com schrieb: > However, the decision was a bad one regardless of the existing policy, > and sets a bad precedent while we are discussing this policy. I could > be wrong, but I think it would be reasonable to assume that if Martin > strongly supports such a change, Martin would oppose a policy which > would strictly forbid such changes, and it is just such a policy that > Python needs. I still can't guess what policy you have in mind, so I can't object to it :-) I may accept a policy that rejects this change, but allows another change to fix the problem. I would oppose a policy that causes this bug to be unfixable forever. Regards, Martin From martin at v.loewis.de Fri Mar 16 00:45:12 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 16 Mar 2007 00:45:12 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> <45F9B98F.5060102@v.loewis.de> Message-ID: <45F9DA88.7080609@v.loewis.de> Mike Krell schrieb: > Here is a point of confusion. Bear in mind I'm running this under > windows, so explorer happily reports that ".emacs" has a type of > "emacs". (In windows, file types are registered in the system based > on the extension -- all the characters following the last dot. Is it really that Emacs registered .emacs as an extension on Windows? (I see you answer that below) I would be surprised - I'd rather expect that the Emacs authors *don't* consider .emacs to be a file with an empty filename and a .emacs extension. They also (alternatively) support a directory called .emacs.d for startup files, and I would be equally surprised if they registered .d as extension (about the only extension Emacs might register is .el/.elc). The reason the file is called .emacs on Windows is *not* because it should have that extension, but because it is called .emacs on Unix, and it is called that way because the Unix shell and ls suppress dotfiles unless explicitly asked to display them. > I often sort files in the explorer based on type, and I want a file > and all its backups to appear next to each other in such a sorted > list. That's exactly why I rename the files the way I do. > Thus, ".1.emacs" is what I want, and ".emacs.1" is a markedly inferior > and unacceptable alternative. That's what I'm referring to by > extension preservation. Ok, I see why that would break. What do you do with files that really have no extension whatsoever (i.e. no dot at all)? Regards, Martin From greg.ewing at canterbury.ac.nz Fri Mar 16 01:06:56 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Mar 2007 13:06:56 +1300 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: <20070315182523.7769.715364850.divmod.xquotient.887@joule.divmod.com> References: <20070315151236.18920.1909125356.divmod.quotient.2790@ohm> <45F9733F.7030409@v.loewis.de> <20070315182523.7769.715364850.divmod.xquotient.887@joule.divmod.com> Message-ID: <45F9DFA0.80504@canterbury.ac.nz> glyph at divmod.com wrote: > Can you suggest any use-cases for thread termination which will *not* > result in a completely broken and unpredictable heap after the thread > has died? Suppose you have a GUI and you want to launch a long-running computation without blocking the user interface. You don't know how long it will take, so you want the user to be able to cancel it if he gets bored. There's no single place in the code where you could put in a check for cancellation. Sprinkling such checks all over the place would be tedious, or even impossible if large amounts of time are spent in calls to a third-party library that wasn't designed for such things. Interaction with the rest of the program is extremely limited -- some data is passed in, it churns away, and some data is returned. It doesn't matter what happens to its internal state if it gets interrupted, as it's all going to be thrown away. In that situation, it doesn't seem unreasonable to me to want to be able to just kill the thread. I don't see how it could do any more harm than using KeyboardInterrupt to kill a program, because that's all it is -- a subprogram running inside your main program. How would you handle this situation? > If you can think of such a case, are you sure it wouldn't be > better served by a set of threads communicating over queues and sending > 'Stop' objects to each other If the thread is guaranteed to return to reading from the queue within a bounded time, that's fine, and it's the solution I would recommend in that case. But not all cases are like that. -- Greg From mbk.lists at gmail.com Fri Mar 16 01:13:18 2007 From: mbk.lists at gmail.com (Mike Krell) Date: Thu, 15 Mar 2007 17:13:18 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F9DA88.7080609@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> <45F9B98F.5060102@v.loewis.de> <45F9DA88.7080609@v.loewis.de> Message-ID: On 3/15/07, "Martin v. L?wis" wrote: > *don't* consider .emacs to be a file with an empty filename and > a .emacs extension. They also (alternatively) support a directory > called .emacs.d for startup files, and I would be equally surprised > if they registered .d as extension (about the only extension Emacs > might register is .el/.elc). Agreed on both counts. I'm sure neither of these are registered extensions, but for what I care about the operative question is what windows explorer does with (what it considers to be) unregistered extensions. > The reason the file is called .emacs on Windows is *not* because > it should have that extension, but because it is called .emacs > on Unix, and it is called that way because the Unix shell and ls > suppress dotfiles unless explicitly asked to display them. Yes. > Ok, I see why that would break. What do you do with files that really > have no extension whatsoever (i.e. no dot at all)? That use case doesn't come up for this application -- see my response to Mike Klass. I actually muddied the waters here by using ".emacs" as an example. In practice, this app would never copy a .emacs file since its used to copy files used by itself. Mike From anthony at interlink.com.au Fri Mar 16 01:13:58 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri, 16 Mar 2007 11:13:58 +1100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <20070315205736.7769.1178617927.divmod.xquotient.1001@joule.divmod.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F9AAB9.4040408@v.loewis.de> <20070315205736.7769.1178617927.divmod.xquotient.1001@joule.divmod.com> Message-ID: <200703161114.01440.anthony@interlink.com.au> On Friday 16 March 2007 07:57, glyph at divmod.com wrote: > Common parlance for the parts of a version number is: > major.minor.micro > See: > http://twistedmatrix.com/documents/current/api/twisted.python.ver >sions.Version.html#__init__ > > Changing this terminology about Python releases to be more > consistent with other projects would be a a subtle, but good > shift towards a generally better attitude of the expectations of > "minor" releases. I disagree entirely. Python has major releases, and bugfix releases. At the moment, the major releases are of the form 2.x, and bugfix 2.x.y. Trying to say that 2.4->2.5 is a "minor" release is just nonsensical. That suggests that very little new language development would go on, at all. Now, you might be arguing that in fact very little should go on (I know some people have argued this, I can't remember if you were one of these). My standard response to this is that people who really feel like this are welcome to pick a release, say, 2.3, and take on the process of backporting the relevant bugfixes back to that release, and cutting new releases, &c. -- Anthony Baxter It's never too late to have a happy childhood. From greg.ewing at canterbury.ac.nz Fri Mar 16 01:30:43 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Mar 2007 13:30:43 +1300 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> Message-ID: <45F9E533.7000306@canterbury.ac.nz> Mike Krell wrote: > I want > ".emacs" to be renamed to ".1.emacs", thus preserving the extensions. > Under the new patch, the second file would be renamed to ".emacs.1", > gratuitously breaking the extension preservation. This argument presupposes that ".emacs" on its own should be considered an extension, which is the very thing under dispute. If it's not considered an extension, it gets renamed to ".emacs.1", just the same as any other file with no extension, e.g. "foo" to "foo.1". So it's equally consistent whichever way you think of it. -- Greg From jcarlson at uci.edu Fri Mar 16 01:53:09 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 15 Mar 2007 17:53:09 -0700 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: <45F9DFA0.80504@canterbury.ac.nz> References: <20070315182523.7769.715364850.divmod.xquotient.887@joule.divmod.com> <45F9DFA0.80504@canterbury.ac.nz> Message-ID: <20070315174053.FBD8.JCARLSON@uci.edu> Greg Ewing wrote: > > glyph at divmod.com wrote: > > > Can you suggest any use-cases for thread termination which will *not* > > result in a completely broken and unpredictable heap after the thread > > has died? > > Suppose you have a GUI and you want to launch a > long-running computation without blocking the > user interface. You don't know how long it will > take, so you want the user to be able to cancel > it if he gets bored. If the code is in Python, you can use sys.settrace to handle this. If the code is in an extension module that a user has control over, having a cancel_thread() function that is made available to Python, and having your C code check the value of a single variable every few seconds could do the same thing (even checking the value in a tight loop shouldn't slow computations down significantly, branch prediction should be able to make it a more or less zero-cost operation). Yes, it can be tedious, but at least the programmer can actually control cleanup in a reasonable manner. The only case that I have been able to come up with that is not covered with these two is if you have no control over the C-level code, which would be the case in a precompiled 3rd party extension or system call. In the system call case, I'm not sure there is a sane way to abort it on all platforms, and I can just about guarantee that even if you *could* kill a thread, doing so in 3rd party code (depending on the code) could leave you in a questionable state (memory leaks, temporary files, broken data structures, etc.). It seems better to write to allow for cancellation, rather than adding a big red STOP button to threads. - Josiah From greg.ewing at canterbury.ac.nz Fri Mar 16 01:55:02 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Mar 2007 13:55:02 +1300 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> <45F9B98F.5060102@v.loewis.de> Message-ID: <45F9EAE6.1040907@canterbury.ac.nz> Mike Krell wrote: > copies of ".emacs" would be made as ".1.emacs", > ".2.emacs", etc. But that's not going to work for other extensionless files that don't begin with a dot. The fact that it happens to work for .emacs files and the like is just a fluke due to Windows' ignorance of Unix file naming conventions. -- Greg From greg.ewing at canterbury.ac.nz Fri Mar 16 02:04:31 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Mar 2007 14:04:31 +1300 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <200703161114.01440.anthony@interlink.com.au> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F9AAB9.4040408@v.loewis.de> <20070315205736.7769.1178617927.divmod.xquotient.1001@joule.divmod.com> <200703161114.01440.anthony@interlink.com.au> Message-ID: <45F9ED1F.5030407@canterbury.ac.nz> Anthony Baxter wrote: > Python has major releases, and bugfix releases. > At the moment, the major releases are of the form 2.x, and bugfix > 2.x.y. Yes, and from history so far there's no particular semantics attached to first-digit transitions. 1.x -> 2.x was nothing to write home about, and 2.x -> 3.x is going to be something like a "mega" release. -- Greg From murman at gmail.com Fri Mar 16 02:28:36 2007 From: murman at gmail.com (Michael Urman) Date: Thu, 15 Mar 2007 20:28:36 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F99200.7040303@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> Message-ID: On 3/15/07, "Martin v. L?wis" wrote: > If we both agree that the old behavior was erroneous, then I > cannot understand why you want to see the patch reverted. I think at least part of the disagreement is over the classification of the earlier behavior as "erroneous". Both unexpected and undesirable have certainly been common classifications, but as not everyone agrees, and a very visible example in Windows Explorer disagree, it's hard to settle on this behavior being simply incorrect. Thus it's a value judgement. Unlike other value judgements reflected in Misc/NEWS, there are no similar APIs with which we can compare behavior and match to increase consistency. Michael -- Michael Urman http://www.tortall.net/mu/blog From pje at telecommunity.com Fri Mar 16 02:32:19 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 15 Mar 2007 20:32:19 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F9E533.7000306@canterbury.ac.nz> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> Message-ID: <5.1.1.6.0.20070315200047.028365a0@sparrow.telecommunity.com> At 01:30 PM 3/16/2007 +1300, Greg Ewing wrote: >Mike Krell wrote: > > I want > > ".emacs" to be renamed to ".1.emacs", thus preserving the extensions. > > Under the new patch, the second file would be renamed to ".emacs.1", > > gratuitously breaking the extension preservation. > >This argument presupposes that ".emacs" on its own >should be considered an extension, which is the >very thing under dispute. It's not under dispute for *him*, nor is it under dispute that Windows Explorer treats it that way. What's *actually* under dispute here is whether it's acceptable to classify this perfectly useful-as-is behavior, that was documented and tested in released versions of Python for several years (with patches to change its behavior explicitly rejected in the past), as a "bug". Unfortunately, people who are on the other side of the issue seem unable to conceive of the possibility that there are people who legitimately *don't* think this is a bug. However, just because someone doesn't like it, doesn't make it a bug. Design flaw? Wart? Oversight? Perhaps. But bug? When it's explicitly documented and tested, and there exist legitimate uses of the existing behavior, even among Python-dev'ers? Heck no. Unfortunately, because some people have it in their heads that "'.emacs' is not a file extension" (to *them*), they aren't able to handle the idea that on Windows, it bloody well *is* a file extension, and some people would like to treat it as such *even on non-Windows platforms*. They don't seem to understand that it doesn't matter how many posts they write "explaining" to us poor deluded souls that their interpretation of "extension" is the only correct interpretation, it isn't going to change the fact that there are *many* valid interpretations. Reasonable people can therefore dispute what "splitext" should consider to be an "extension" - and it's been further pointed out that at one time, splitext() could and did consider '.tar.gz' to be the "extension"! So, no matter how many times people call this a "bug", it is *not* a bug. It is merely a feature that "more" people (in a straw poll of Python-dev) dislike than like. However, a straw poll of Python users at large might reveal that Python's "explicit self" pattern is unpopular. Should we consider that a bug, then, and "fix" it too, if someone offered a patch for it, because they wrote a program using Java-style implicit self, and it didn't work? Yes, let's change Python so that methods and attributes don't need a self parameter or a "self.", and silently change the meaning of existing programs, because clearly anybody who *likes* explicit self is wrong and wrote a bad program. So let's just "fix" their program by silently changing its meaning underneath them, because surely they meant 'self' to be some other function argument, and now anybody who writes new programs in this style will have them work! (And if you think that this isn't a fair comparison, it's because you're not looking at it from the POV of those who like the current splitext() behavior.) As Glyph says, this change is *punishing the conscientious* (i.e., everyone who actually read the docstrings and/or tested their code in the last decade and a half) to reward people who were NOT conscientious. That seems backwards to me. From pje at telecommunity.com Fri Mar 16 02:39:23 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 15 Mar 2007 20:39:23 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F9BD07.2010605@v.loewis.de> References: <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> Message-ID: <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> At 10:39 PM 3/15/2007 +0100, Martin v. L??wis wrote: >That said, if it makes people more comfortable with having a warning >added, I won't object. It's just that I don't want to be the one to >take the blame for issuing the warning, because deep in my heart I >feel that warnings are a bad thing, unless they are right most of the >time (which they won't be in this case). Some other options: 1. Deprecate splitext() and remove it in 3.0 2. Add an optional flag argument to enable the new behavior 3. Create a new function with the new behavior (as you proposed the last time there was a patch submitted for this) From steve at holdenweb.com Fri Mar 16 03:54:59 2007 From: steve at holdenweb.com (Steve Holden) Date: Thu, 15 Mar 2007 22:54:59 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> Message-ID: Michael Urman wrote: > On 3/15/07, "Martin v. L?wis" wrote: >> If we both agree that the old behavior was erroneous, then I >> cannot understand why you want to see the patch reverted. > > I think at least part of the disagreement is over the classification > of the earlier behavior as "erroneous". Both unexpected and > undesirable have certainly been common classifications, but as not > everyone agrees, and a very visible example in Windows Explorer > disagree, it's hard to settle on this behavior being simply incorrect. > Thus it's a value judgement. Unlike other value judgements reflected > in Misc/NEWS, there are no similar APIs with which we can compare > behavior and match to increase consistency. The fact remains that those who have used the existing functionality as it is implemented and documented will, of this change isn't reverted, have to make a gratuitous change to their currently working programs. I'm not really too concerned with whether the behavior is "erroneous" - that is indeed in the realm of value judgment. I'm concerned about the impression the user community will get as they see such changes made to the language with no more apparent justification than the whim of an individual developer. Because some immoderate language has been used in this thread I would like to underline that these remarks are not directed personally at Martin, for whom I have the utmost respect as a developer, but at the lack of process that allows circumstances like this to arise. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From pmaupin at gmail.com Fri Mar 16 04:19:58 2007 From: pmaupin at gmail.com (Patrick Maupin) Date: Thu, 15 Mar 2007 22:19:58 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> Message-ID: On 3/15/07, Steve Holden wrote: > The fact remains that those who have used the existing functionality as > it is implemented and documented will, of this change isn't reverted, > have to make a gratuitous change to their currently working programs. The worst part is, if they are relying on that specific behavior and have to rely on the new specific behavior, and want to support old and new versions of Python, they are potentially left with some very unattractive options -- check the Python version to figure out how splitext works, or just roll their own and stop calling splitext entirely, because its behavior is not consistent across versions. > Because some immoderate language has been used in this thread I would > like to underline that these remarks are not directed personally at > Martin, for whom I have the utmost respect as a developer, but at the > lack of process that allows circumstances like this to arise. I second this sentiment. Regards, Pat From shansen at advpubtech.com Fri Mar 16 05:25:14 2007 From: shansen at advpubtech.com (Stephen Hansen) Date: Thu, 15 Mar 2007 21:25:14 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> Message-ID: <7a9c25c20703152125v417b99c6of49a2c082991031f@mail.gmail.com> For anyone who is interested, I've submitted a patch (source + docs + tests) to SF as 1681842, which re-establishes the previous behavior, but adds a keyword argument to obtain the new behavior and a warning promising the new behavior will become default in the future. ...which would be my second contribution ever. And the first one to be more then a line and a half :P -- Stephen Hansen Development Advanced Prepress Technology shansen at advpubtech.com (818) 748-9282 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070315/c411b99f/attachment.htm From tjreedy at udel.edu Fri Mar 16 06:21:38 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 16 Mar 2007 01:21:38 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com><5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com><5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com><45F99200.7040303@v.loewis.de><45F9A8F1.6070209@v.loewis.de><45F9B98F.5060102@v.loewis.de><45F9DA88.7080609@v.loewis.de> Message-ID: "Mike Krell" wrote in message news:da7032ce0703151713j1b38dcb2l1ac63b1d1bd46407 at mail.gmail.com... >I actually muddied the waters here by using ".emacs" as an example. In >practice, this app would never copy a .emacs file since its used to >copy files used by itself. Do you actually save any files 'named' '.xxx'? From tjreedy at udel.edu Fri Mar 16 06:47:23 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 16 Mar 2007 01:47:23 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com><5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com><5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com><5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com><45F99200.7040303@v.loewis.de><45F9A8F1.6070209@v.loewis.de> <45F9E533.7000306@canterbury.ac.nz> <5.1.1.6.0.20070315200047.028365a0@sparrow.telecommunity.com> Message-ID: "Phillip J. Eby" wrote in message news:5.1.1.6.0.20070315200047.028365a0 at sparrow.telecommunity.com... As to the usefulness of current behavior, the only supposed use-case code posted, that I have noticed, was that it made it easy to turn '.emacs' into '1.emacs', but then MK said the app does not really do that. As for comparison with changing 'self'... | (And if you think that this isn't a fair comparison, I think it is a ludicrous comparison, not even in the same ballpark, that tends to discredit the valid points you have made. | it's because you're | not looking at it from the POV of those who like the current splitext() | behavior.) I get annoyed when people tell me why I think the way I do, especially when they are so badly mistaken. Terry Jan Reedy From mbk.lists at gmail.com Fri Mar 16 09:02:51 2007 From: mbk.lists at gmail.com (Mike Krell) Date: Fri, 16 Mar 2007 01:02:51 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> <45F9E533.7000306@canterbury.ac.nz> <5.1.1.6.0.20070315200047.028365a0@sparrow.telecommunity.com> Message-ID: On 3/15/07, Terry Reedy wrote: > As to the usefulness of current behavior, the only supposed use-case code > posted, that I have noticed, was that it made it easy to turn '.emacs' into > '1.emacs', but then MK said the app does not really do that. I said the name ".emacs" was used as an example. For that matter, the name "a.txt" was also used as an example. The use cases are real. Mike From ncoghlan at gmail.com Fri Mar 16 09:40:17 2007 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Mar 2007 18:40:17 +1000 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: <20070315174053.FBD8.JCARLSON@uci.edu> References: <20070315182523.7769.715364850.divmod.xquotient.887@joule.divmod.com> <45F9DFA0.80504@canterbury.ac.nz> <20070315174053.FBD8.JCARLSON@uci.edu> Message-ID: <45FA57F1.6020903@gmail.com> Josiah Carlson wrote: > Greg Ewing wrote: >> glyph at divmod.com wrote: >> >>> Can you suggest any use-cases for thread termination which will *not* >>> result in a completely broken and unpredictable heap after the thread >>> has died? >> Suppose you have a GUI and you want to launch a >> long-running computation without blocking the >> user interface. You don't know how long it will >> take, so you want the user to be able to cancel >> it if he gets bored. > > If the code is in Python, you can use sys.settrace to handle this. If > the code is in an extension module that a user has control over, having > a cancel_thread() function that is made available to Python, and having > your C code check the value of a single variable every few seconds could > do the same thing (even checking the value in a tight loop shouldn't > slow computations down significantly, branch prediction should be able > to make it a more or less zero-cost operation). Yes, it can be tedious, > but at least the programmer can actually control cleanup in a reasonable > manner. Option 3, farm the long running operation out to another process and use the OS-provided facilities to abort and cleanup if the user changes their mind. It's the only way to be sure the aborted operation doesn't leave the main process in a dodgy state. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From rasky at develer.com Fri Mar 16 09:55:03 2007 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 16 Mar 2007 09:55:03 +0100 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: <45F9DFA0.80504@canterbury.ac.nz> References: <20070315151236.18920.1909125356.divmod.quotient.2790@ohm> <45F9733F.7030409@v.loewis.de> <20070315182523.7769.715364850.divmod.xquotient.887@joule.divmod.com> <45F9DFA0.80504@canterbury.ac.nz> Message-ID: On 16/03/2007 1.06, Greg Ewing wrote: >> Can you suggest any use-cases for thread termination which will *not* >> result in a completely broken and unpredictable heap after the thread >> has died? > > Suppose you have a GUI and you want to launch a > long-running computation without blocking the > user interface. You don't know how long it will > take, so you want the user to be able to cancel > it if he gets bored. > > There's no single place in the code where you > could put in a check for cancellation. Sprinkling > such checks all over the place would be tedious, > or even impossible if large amounts of time are > spent in calls to a third-party library that > wasn't designed for such things. > > Interaction with the rest of the program is > extremely limited -- some data is passed in, > it churns away, and some data is returned. It > doesn't matter what happens to its internal > state if it gets interrupted, as it's all going > to be thrown away. > > In that situation, it doesn't seem unreasonable > to me to want to be able to just kill the thread. > I don't see how it could do any more harm than > using KeyboardInterrupt to kill a program, > because that's all it is -- a subprogram running > inside your main program. > > How would you handle this situation? It's really simple: don't use threads, use processes! Spawn an external process which does the calculation, pass data to it through pipe/socket/namedpipe/xmlrpc/whatever and read data back from it when it's done. If you need to kill it, just kill it away, at any asynchronous time: the OS will clean up after it. After many years working with these issues, I came to the personal conclusion of avoiding threads as much as possible. Threads are processes with shared memory, but in many real-world use cases I faced, there is really only a very little chunk of memory which is shared, and Python makes it incredibly easy to marshal data to a process (pickle or whatever). So in many cases there's really little excuses for going mad with threads. -- Giovanni Bajo From p.f.moore at gmail.com Fri Mar 16 11:34:36 2007 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 16 Mar 2007 10:34:36 +0000 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070315200047.028365a0@sparrow.telecommunity.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> <45F9E533.7000306@canterbury.ac.nz> <5.1.1.6.0.20070315200047.028365a0@sparrow.telecommunity.com> Message-ID: <79990c6b0703160334p507b7024ma8fb5dd332880dc5@mail.gmail.com> On 16/03/07, Phillip J. Eby wrote: > What's *actually* under dispute here is whether it's acceptable to classify > this perfectly useful-as-is behavior, that was documented and tested in > released versions of Python for several years (with patches to change its > behavior explicitly rejected in the past), as a "bug". Just to put this into context, the word "bug" is probably not the best to use here. The orignal behaviour was described as a bug, certainly, but that's not how the change has been treated. If the behaviour was being deemed a "bug", it would be acceptable in a "bugfix" release (ie. 2.5.1). No-one is advocating that. Rather, the change is being treated as a behaviour change (which it is) and submitted for a *feature* release (2.6). Whether the behaviour change is good, reasonable, acceptable - that's the question here. (And one on which I don't have an opinion!) Paul From thomas at python.org Fri Mar 16 11:52:22 2007 From: thomas at python.org (Thomas Wouters) Date: Fri, 16 Mar 2007 11:52:22 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F8715A.8000208@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <45F865A6.8080903@voidspace.org.uk> <9e804ac0703141427s51b3caa3n16c316ab4d37cbd3@mail.gmail.com> <45F8715A.8000208@v.loewis.de> Message-ID: <9e804ac0703160352n1280c927h4e546091cabf2695@mail.gmail.com> On 3/14/07, "Martin v. L?wis" wrote: > > Thomas Wouters schrieb: > > However, changing documented, tested behaviour without warning gives > > Python an even worse name. I agree with PJE that the change is the wrong > > thing to do, simply because it sets (yet another) precedent. If > > providing an alternate API with clearer semantics is too 'heavy-weight' > > a solution and warning is for some reason unacceptable (I don't see why; > > all the arguments against warning there go for *any* warning in Python) > > -- then the problem isn't bad enough to fix it by breaking other code. > > I think producing pointless warnings also gives Python a bad name > (I've seen many complaints about Python's warnings in the past, in > particular when they fill up Apache log files). I would be more pissed if my apache logfiles were full of errors, instead :-) But perhaps we should be more forward about the use of warnings: warn people (so to speak) about warnings, and tell them about the -W option for making them louder/quieter. However, if everybody (and here I mean everybody) can agree that adding > a warning to the current implementation would be an acceptable > compromise, I could agree to such a compromise also (although I > would prefer if somebody else took the blame for adding that warning. > I happily take the blame for changing the behavior). > > What specific warning would you propose, and in what specific > circumstance would it be issued? Hah, everyone agree? They weren't agreeing when you changed it, either :) But no, we don't add a warning *and* change the API. We add a warning *about* changing the API. 2.6 sees no semantic change, just a warning when os.path.splitext is used on a dotfile with no extension (or a file with an extension but no name, so you will.) 2.7/3.0 see the change in semantics. We do this for three reasons: - People who rely on the documented, tested, ages-old behaviour will get fair warning that the behaviour will change. I don't mean just programmers. I mean users, too. Yes, users will get to see the warning and many of them might not be able to do something about it. Well, considering this class of users would get a behavioural change, and quite likely a bug, giving them a warning hardly seems intrusive. - People who (inadvertently) rely on the new behaviour get a warning that their code is currently bugged. This includes users, too, of course: they get a warning that this program is bugged in older versions of Python. If there was a way to telepathically warn the actual programmer, that would be better, but there isn't, so we can't. We will have to use the user as the messenger. Furthermore, even if the original programmer is a user of his or her own program and uses Python 2.6, he or she may never trigger the erroneous behaviour himself. The user who got the warning is the only one who can tell him or her that there's a problem with dotfiles. (Even though only a small fraction of the actual users will send in a bug -- it's still the best we can do.) - Most importantly, people who don't care about the change, whose code works acceptibly with either version of os.path.splitext, will get warned about the change in behaviour. If, as in one of the given examples in this thread, files are renamed based on their 'extension', it may work either way, and it may make sense either way, but it will *change*. Files may end up being differently renamed. I don't see that as acceptible behaviour, for upgrading python to cause a subtle but noticeable change in how a program does its work -- without error. That's why we warn. There is a big difference between fixing this, and fixing bugs that are obviously bugs: functions that behave differently from the documentation or (if not documented) in obviously wrong ways. If you need to wonder what 'obvious' means: if the "average programmer" using the function does not realize he's getting 'incorrect' behaviour, it's not obviously wrong. This entire thread should make it obvious that os.path.splitext's old behaviour, while nonsensical if you think about it, is not *obviously* wrong. If os.path.splitext(".dotfile") were to return (".dotfile", ".dotfile"), that would be obviously wrong. What it does now is not. Changing it is the right thing, but changing it without first warning about it is not. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070316/9711676e/attachment.htm From amk at amk.ca Fri Mar 16 12:22:23 2007 From: amk at amk.ca (A.M. Kuchling) Date: Fri, 16 Mar 2007 07:22:23 -0400 Subject: [Python-Dev] What's New maintenance In-Reply-To: <45F9BB1A.30309@v.loewis.de> References: <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9BB1A.30309@v.loewis.de> Message-ID: <20070316112223.GA3460@andrew-kuchlings-computer.local> On Thu, Mar 15, 2007 at 10:31:06PM +0100, "Martin v. L?wis" wrote: > I would do that, except that Andrew explicitly reserved the right to > change whatsnew.tex. I believe he does go over Misc/NEWS in doing so. It's actually OK to add things to whatsnew.tex, subject to the warning that: 1) I will rewrite the text as necessary. 2) some changes I consider too obscure to bother describing in whatsnew.tex. If such a change is added to the text, I'll just remove it. I would feel bad if someone were to spend a long time writing a paragraph or section, and then I decided to delete the text. So... add changes as you wish, but please don't work very hard on the text, and writing in note form is fine -- I'll rewrite as needed. You don't need to comment out your added text; we're so far from release I don't care if the formatting is incorrect. Where should I add a note about this: a comment at the top of whatsnew.tex, maybe? I do go over Misc/NEWS every so often. --amk From martin at v.loewis.de Fri Mar 16 13:31:20 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 16 Mar 2007 13:31:20 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> References: <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> Message-ID: <45FA8E18.4040805@v.loewis.de> Phillip J. Eby schrieb: > Some other options: > > 1. Deprecate splitext() and remove it in 3.0 How would that help the problem? Isn't it useful to have a function that strips off the extension? > 2. Add an optional flag argument to enable the new behavior How would that help backwards compatibility? > 3. Create a new function with the new behavior (as you proposed the last > time there was a patch submitted for this) What to do with the old function in this case? Martin From martin at v.loewis.de Fri Mar 16 13:51:24 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 16 Mar 2007 13:51:24 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <9e804ac0703160352n1280c927h4e546091cabf2695@mail.gmail.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <45F865A6.8080903@voidspace.org.uk> <9e804ac0703141427s51b3caa3n16c316ab4d37cbd3@mail.gmail.com> <45F8715A.8000208@v.loewis.de> <9e804ac0703160352n1280c927h4e546091cabf2695@mail.gmail.com> Message-ID: <45FA92CC.7020307@v.loewis.de> Thomas Wouters schrieb: > Hah, everyone agree? They weren't agreeing when you changed it, either > :) But no, we don't add a warning *and* change the API. We add a warning > *about* changing the API. 2.6 sees no semantic change, just a warning > when os.path.splitext is used on a dotfile with no extension (or a file > with an extension but no name, so you will.) 2.7/3.0 see the change in > semantics. Would you like to work on that? Feel free to undo my changes as needed, although I think the merging of the various implementations of splitext can be kept, as should the additional test cases (just with a different outcome). The tracker reports need to be updated to indicate the change, too. > - People who rely on the documented, tested, ages-old behaviour will > get fair warning that the behaviour will change. I don't mean just > programmers. I mean users, too. Yes, users will get to see the warning > and many of them might not be able to do something about it. Well, > considering this class of users would get a behavioural change, and > quite likely a bug, giving them a warning hardly seems intrusive. Here I disagree. I believe many people will see the warning that won't see any behavior change (except for temporarily getting a warning). Much code will do for fn in os.listdir(path): if os.path.splitext(fn)[1] in ('.c', '.h', '.vcproj'): some_action This code will be unaffected by the change, unless people have a file called .c in a directory. > - People who (inadvertently) rely on the new behaviour get a warning > that their code is currently bugged. This includes users, too, of > course: they get a warning that this program is bugged in older versions > of Python. If there was a way to telepathically warn the actual > programmer, that would be better, but there isn't, so we can't. We will > have to use the user as the messenger. But we do warn the programmer: there is a change in the documentation (not just Misc/NEWS). > What it does > now is not. Changing it is the right thing, but changing it without > first warning about it is not. Ok, I can accept a solution that will allow it to be changed eventually, although I'm not happy with producing a warning. So, as I said, if somebody wants to commit such a change, go ahead. If you want me to review it first, I can do that as well. Regards, Martin From martin at v.loewis.de Fri Mar 16 14:00:30 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 16 Mar 2007 14:00:30 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> Message-ID: <45FA94EE.5080503@v.loewis.de> Patrick Maupin schrieb: > The worst part is, if they are relying on that specific behavior and have > to rely on the new specific behavior, and want to support old and new > versions of Python, they are potentially left with some very unattractive > options -- check the Python version to figure out how splitext works, or > just roll their own and stop calling splitext entirely, because its behavior > is not consistent across versions. Somebody has pointed out that it is fairly easy to write a wrapper around splitext that explicitly produces the old behavior on all versions, or the new behavior on all versions, depending on what precisely is desired. Users that have coded for a specific behavior will have to write a wrapper - whether they explicitly code for the old behavior or the new one. Regards, Martin From martin at v.loewis.de Fri Mar 16 14:05:18 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 16 Mar 2007 14:05:18 +0100 Subject: [Python-Dev] What's New maintenance In-Reply-To: <20070316112223.GA3460@andrew-kuchlings-computer.local> References: <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9BB1A.30309@v.loewis.de> <20070316112223.GA3460@andrew-kuchlings-computer.local> Message-ID: <45FA960E.9050505@v.loewis.de> A.M. Kuchling schrieb: > Where should I add a note about this: a comment at the top of > whatsnew.tex, maybe? That would be good. Regards, Martin From steve at holdenweb.com Fri Mar 16 14:10:51 2007 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Mar 2007 09:10:51 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45FA8E18.4040805@v.loewis.de> References: <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <45FA8E18.4040805@v.loewis.de> Message-ID: Martin v. L?wis wrote: > Phillip J. Eby schrieb: >> Some other options: >> >> 1. Deprecate splitext() and remove it in 3.0 > > How would that help the problem? Isn't it useful to have a function > that strips off the extension? > >> 2. Add an optional flag argument to enable the new behavior > > How would that help backwards compatibility? > By providing it! The suggestion would retain the same behavior unless a newly-specified aspect of the API is exercised, therefore avoiding gratuitous change to existing programs' functionality. Since the default would be to behave as the existing function does then you would have to specify a True value for the "strange-and-incomprehensible-treatment-of-dotfiles" to get the behavior as specified in the patch you just applied. This seems like the best option to me, as clearly there are enough different opinions about whether the old or the new behavior is a bug that a user-selectable behavior is actually desirable. My suspicion is that most users just won't care about dotfiles, and will continue to use splitext as is. Windows users are always surprised to see them appearing, but they are becoming more common as open source functionality migrates to Windows. But those who do care (as you obviously do) can use bizarreAndInexplicableDotfileBehavior=True ;-) >> 3. Create a new function with the new behavior (as you proposed the last >> time there was a patch submitted for this) > > What to do with the old function in this case? > Presumably keep it, thereby adding to the bloat in the language - definitely NOT my preferred option. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From steve at holdenweb.com Fri Mar 16 14:14:12 2007 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Mar 2007 09:14:12 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45FA94EE.5080503@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45FA94EE.5080503@v.loewis.de> Message-ID: Martin v. L?wis wrote: > Patrick Maupin schrieb: >> The worst part is, if they are relying on that specific behavior and have >> to rely on the new specific behavior, and want to support old and new >> versions of Python, they are potentially left with some very unattractive >> options -- check the Python version to figure out how splitext works, or >> just roll their own and stop calling splitext entirely, because its behavior >> is not consistent across versions. > > Somebody has pointed out that it is fairly easy to write a wrapper > around splitext that explicitly produces the old behavior on all > versions, or the new behavior on all versions, depending on what > precisely is desired. Users that have coded for a specific behavior > will have to write a wrapper - whether they explicitly code for the > old behavior or the new one. > How is forcing people to write such a wrapper better than providing an optional argument (defaulting to current behavior) to specify the behavior they want? Presumably people who already care enough to want the patched behavior have already written such a wrapper around the current version. This should continue to work, albeit with less than exemplary efficiency. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From pje at telecommunity.com Fri Mar 16 16:13:08 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 16 Mar 2007 10:13:08 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45FA8E18.4040805@v.loewis.de> References: <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070316100847.0289ba88@sparrow.telecommunity.com> At 01:31 PM 3/16/2007 +0100, Martin v. L?wis wrote: >Phillip J. Eby schrieb: > > Some other options: > > > > 1. Deprecate splitext() and remove it in 3.0 > >How would that help the problem? Isn't it useful to have a function >that strips off the extension? Not if there's no consensus as to what "extension" means. Proposals to add sorted dictionaries or routines to add "non-duplicate" items to a list or to "flatten" data structures are routinely rejected for the same reason: users are recommended to write a routine that does what their particular application needs. > > 2. Add an optional flag argument to enable the new behavior > >How would that help backwards compatibility? All existing calls to splitext() would work the same way they've done for several years, and the documentation would now make it obvious to new users of the function that there's a choice about how to handle dotfiles. Heck, we could throw in another optional argument to strip multiple extensions like .tar.gz. For that matter, the documentation should address the issue that no matter what options you use, you may *still* end up with unexpected results, for files like "Release 1.2", due to the fuzzy nature of the concept of a file "extension" on modern OSes. > > 3. Create a new function with the new behavior (as you proposed the last > > time there was a patch submitted for this) > >What to do with the old function in this case? Leave it alone - it's not broken. If people have buggy programs because they assumed '.foo' files were handled in a way that the docstrings and tests clearly indicate they are *not*, and they didn't test their *own* program, it's not Python's responsibility to fix their programs. From pje at telecommunity.com Fri Mar 16 16:23:46 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 16 Mar 2007 10:23:46 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> <45F9E533.7000306@canterbury.ac.nz> <5.1.1.6.0.20070315200047.028365a0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070316101518.04c14030@sparrow.telecommunity.com> At 01:47 AM 3/16/2007 -0400, Terry Reedy wrote: >I think it is a ludicrous comparison, not even in the same ballpark, that >tends to discredit the valid points you have made. Of course it's not in the same ballpark. The point was to show how ludicrous the *logic* is, by applying it to something that's more obviously disagreeable. The problem (as I see it) is that people who favor the change seem to be trying to use the specifics of this case to justify it -- but my point is that special cases aren't special enough to break the rules. And, that the rules shouldn't be changed to "majority wins, if they think it's 'obviously' a bug". (Aside from all that, I also couldn't think of any milder examples of a popular, yet controversial change at that particular moment.) From ncoghlan at gmail.com Fri Mar 16 17:43:25 2007 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 17 Mar 2007 02:43:25 +1000 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070316100847.0289ba88@sparrow.telecommunity.com> References: <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <5.1.1.6.0.20070316100847.0289ba88@sparrow.telecommunity.com> Message-ID: <45FAC92D.3090307@gmail.com> Phillip J. Eby wrote: > At 01:31 PM 3/16/2007 +0100, Martin v. L?wis wrote: >> Phillip J. Eby schrieb: >>> 2. Add an optional flag argument to enable the new behavior >> How would that help backwards compatibility? > > All existing calls to splitext() would work the same way they've done > for several years, and the documentation would now make it obvious to > new users of the function that there's a choice about how to handle > dotfiles. Heck, we could throw in another optional argument to strip > multiple extensions like .tar.gz. > That may actually be a genuinely useful approach: splitext(name, ignore_leading_dot=False, all_ext=False) Split the extension from a pathname. Returns "(root, ext)". By default, the extension is all characters from the last dot to the end of the string. "ext" will be empty if there are no dots in the name and "root" will be empty if the characters starts with a single dot and that is the only dot in the name. If ignore_leading_dot=True, then a leading dot is always considered part of "root", and is ignored when determining the extension. "root" will never be empty in this case. If all_ext=True, the extension is all characters from the first dot to the end. > For that matter, the documentation should address the issue that no matter > what options you use, you may *still* end up with unexpected results, > for files like "Release 1.2", due to the fuzzy nature of the concept > of a file > "extension" on modern OSes. Not sure what can be done about that... although such filenames are likely a big reason why grabbing 'all extensions' will typically be a bad idea. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From guido at python.org Fri Mar 16 17:50:29 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Mar 2007 09:50:29 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <20070316163704.GC3107@caltech.edu> References: <20070313223629.GA27802@caltech.edu> <20070316163704.GC3107@caltech.edu> Message-ID: On 3/16/07, Titus Brown wrote: > -> > >> What about reimplementing commands.* using subprocess? Or providing a > -> > >> commands.*-compatible interface in the subprocess module? > > OK, so as I understand it, the next step would be for me to provide a > patch implementing this, right? Or is this PEP-required (please no...)? No pep. > What do people think of this basic interface? > > (status, output) = subprocess.get_status_output(cmd) > > output = subprocess.get_output(cmd) > > Here 'status' is the 'returncode' from subprocess.Popen, and 'output' > would be the combined stdout/stderr. 'commands.getstatus' would be > removed entirely [0]. > > This mimics 'commands' fairly closely, while adhering to PEP 8 > guidelines; it's a simple API; and it should be dead easy to implement. Right. Does it also match the style of the API provided by the subprocess module? > It will also have the various advantages people have mentioned: > > * better error trapping; > * better post-fork behavior; > * multi-platform support; > > If this sort of thing goes in, I guess commands.* would then be > deprecated with a note saying "go look at these similar commands in > subprocess", right? Yes. Another intermediate step might be to rewrite the commands module to call the new APIs in the subprocess module. > An additional function that I would personally like is: > > (status, output, errout) = subprocess.get_status_output_err(cmd) > > although the name is hideous. I'd like to change 'get_status_output' > to return a triple, but I'm not sure how big a change that would be > relative to the expected behavior from the ancestral commands function. Call it get_status_output_errors and you have my blessings. If at all possible, get_status_output should *not* just concatenate the output and errors from this API, but attempt to really *merge* the stdout and stderr stream so that if they are interleaved they come out in the right order. That's what the old getstatusoutput() did by putting '2>&1' around the command. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Mar 16 18:27:01 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Mar 2007 10:27:01 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <20070316171904.GA23991@caltech.edu> References: <20070313223629.GA27802@caltech.edu> <20070316163704.GC3107@caltech.edu> <20070316171904.GA23991@caltech.edu> Message-ID: On 3/16/07, Titus Brown wrote: > -> Yes. Another intermediate step might be to rewrite the commands module > -> to call the new APIs in the subprocess module. > > My preference would be to push people towards subprocess, and the more > capable & complex API there, by deprecating commands completely. I'm > happy to do it either way. Yeah, but the deprecation rules require that the commands module has to stay around for at least one release; the quickest we can go is to deprecate it in 2.6 and remove it in 2.7. However, it would be fine to reimplement it in 2.6 using calls to subprocess. Anyway, don't sweat it. Either way will be fine. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Fri Mar 16 20:04:57 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 16 Mar 2007 20:04:57 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45FA94EE.5080503@v.loewis.de> Message-ID: <45FAEA59.4000804@v.loewis.de> Steve Holden schrieb: >> Somebody has pointed out that it is fairly easy to write a wrapper >> around splitext that explicitly produces the old behavior on all >> versions, or the new behavior on all versions, depending on what >> precisely is desired. Users that have coded for a specific behavior >> will have to write a wrapper - whether they explicitly code for the >> old behavior or the new one. >> > How is forcing people to write such a wrapper better than providing an > optional argument (defaulting to current behavior) to specify the > behavior they want? If they pass the flag to the function, the code will stop running on 2.5 and earlier. This is worse than having code that works on all versions. This is also whz I wondered how the flag helps backwards compatibility: when people add the flag, the code stops working on old versions, so it will *not* be backwards compatible. Regards, Martin From mbk.lists at gmail.com Fri Mar 16 20:27:06 2007 From: mbk.lists at gmail.com (Mike Krell) Date: Fri, 16 Mar 2007 12:27:06 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45FAC92D.3090307@gmail.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <5.1.1.6.0.20070316100847.0289ba88@sparrow.telecommunity.com> <45FAC92D.3090307@gmail.com> Message-ID: On 3/16/07, Nick Coghlan wrote: > splitext(name, ignore_leading_dot=False, all_ext=False) +1. ISTM this is a reasonable way to go in the face of our existing backward compatibility issue and the differing definitions of extensions across OS's. Mike From mbk.lists at gmail.com Fri Mar 16 20:30:44 2007 From: mbk.lists at gmail.com (Mike Krell) Date: Fri, 16 Mar 2007 12:30:44 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45FAEA59.4000804@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45FA94EE.5080503@v.loewis.de> <45FAEA59.4000804@v.loewis.de> Message-ID: On 3/16/07, "Martin v. L?wis" wrote: > If they pass the flag to the function, the code will stop running on > 2.5 and earlier. This is worse than having code that works on all > versions. This is also whz I wondered how the flag helps backwards > compatibility: when people add the flag, the code stops working > on old versions, so it will *not* be backwards compatible. I don't understand. Under Nick's proposal, calling splitext with no keyword parameters results in the exact behavior we have today, so it's obviously backward compatible. If you use a keyword parameter, you're using a new feature implemented in 2.6, so there is no expectation of backward compatibility unless and until the keyword parameters are backported. Mike From stephen at xemacs.org Fri Mar 16 20:59:51 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 17 Mar 2007 04:59:51 +0900 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <79990c6b0703160334p507b7024ma8fb5dd332880dc5@mail.gmail.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> <45F9E533.7000306@canterbury.ac.nz> <5.1.1.6.0.20070315200047.028365a0@sparrow.telecommunity.com> <79990c6b0703160334p507b7024ma8fb5dd332880dc5@mail.gmail.com> Message-ID: <17914.63287.299639.142225@uwakimon.sk.tsukuba.ac.jp> Paul Moore writes: > On 16/03/07, Phillip J. Eby wrote: > > What's *actually* under dispute here is whether it's acceptable to classify > > this perfectly useful-as-is behavior, that was documented and tested in > > released versions of Python for several years (with patches to change its > > behavior explicitly rejected in the past), as a "bug". > > Just to put this into context, the word "bug" is probably not the > best to use here. The orignal behaviour was described as a bug, > certainly, but that's not how the change has been treated. [...] > Rather, the change is being treated as a behaviour change (which it > is) and submitted for a *feature* release (2.6). Very well put, and the main point. > Whether the behaviour change is good, reasonable, acceptable - > that's the question here. (And one on which I don't have an > opinion!) That definition of the question is open-minded of you. However, Phillip's point remains valid. Eg, Martin's clear preference for not changing API and mild resistance to a warning suggests that the design of this change is strongly influenced by the feeling that current behavior is a "bug". I think that's inappropriate. Note that Phillip (and glyph, AIUI) is not opposing a behavior change in Python; he simply wants the current API to keep the same spec (here I mean the docstring). I have seen many discussions like this one on various Emacs-related lists, and I would not use a mild phrase like "the word 'bug' is probably not the best to use here". In my experience, use of the word "bug" to describe behavior that is consistent with all documented specifications is usually a political play. When such a change is described as a feature, the majority of programmers who haven't careful studied the spec will say "I thought it already does that now, and I wish it would." When it is described as a bugfix, that changes to "I thought it already does that now, so I demand that it do so." (All this is much less true of Python, but the dynamic can be seen here, too.) That is, even in a feature release backward compatibility is very important, it's just not paramount any more. If you describe it as a new feature, then people are very likely to accept the admittedly expensive process of defining a new API for the improved behavior, and where necessary deprecating the old behavior, and finally removing it. If you describe it as a bugfix, many will not. My opinion is that this shift of atmosphere leads to bad decisions. In cases where behavior conforms to spec, the process of define new API -- deprecate -- remove, though expensive, is almost always a good investment. It both lowers compatibility costs, and makes them much more plan-able. These costs are both large and easy to underestimate, so that's a very good thing. From barry at python.org Fri Mar 16 20:49:53 2007 From: barry at python.org (Barry Warsaw) Date: Fri, 16 Mar 2007 15:49:53 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45FA94EE.5080503@v.loewis.de> <45FAEA59.4000804@v.loewis.de> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mar 16, 2007, at 3:30 PM, Mike Krell wrote: > On 3/16/07, "Martin v. L?wis" wrote: > >> If they pass the flag to the function, the code will stop running on >> 2.5 and earlier. This is worse than having code that works on all >> versions. This is also whz I wondered how the flag helps backwards >> compatibility: when people add the flag, the code stops working >> on old versions, so it will *not* be backwards compatible. > > I don't understand. Under Nick's proposal, calling splitext with no > keyword parameters results in the exact behavior we have today, so > it's obviously backward compatible. If you use a keyword parameter, > you're using a new feature implemented in 2.6, so there is no > expectation of backward compatibility unless and until the keyword > parameters are backported. Let's remember the lessons of True and False in Python 2.2.1. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRfr04nEjvBPtnXfVAQKxXgP9GmIx6OANec+aGsT9X9KoJsWLM+RGYrjB RuDy5uxIbxfZg0logFzvTH4iLCnjJzfhhFrc8V9RjDf7I8vubM+caaEvZBDRoabW bNO6L4IA1zGKjmKYhVhnLkRFNk3iEHwvG9Fa4ahqcCaeS99IYBejwtZ0Sqd171dL ZQnUFBT5vBU= =NlKx -----END PGP SIGNATURE----- From martin at v.loewis.de Fri Mar 16 20:51:07 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 16 Mar 2007 20:51:07 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45FA94EE.5080503@v.loewis.de> <45FAEA59.4000804@v.loewis.de> Message-ID: <45FAF52B.9000404@v.loewis.de> Mike Krell schrieb: > On 3/16/07, "Martin v. L?wis" wrote: > >> If they pass the flag to the function, the code will stop running on >> 2.5 and earlier. This is worse than having code that works on all >> versions. This is also whz I wondered how the flag helps backwards >> compatibility: when people add the flag, the code stops working >> on old versions, so it will *not* be backwards compatible. > > I don't understand. Under Nick's proposal, calling splitext with no > keyword parameters results in the exact behavior we have today, so > it's obviously backward compatible. If you use a keyword parameter, > you're using a new feature implemented in 2.6, so there is no > expectation of backward compatibility unless and until the keyword > parameters are backported. Assuming the current behavior is a bug (which I still believe to be the case), in order to actually make use of the bug fix, you have to pass the parameter. This will make your code break on old versions. Regards, Martin From tjreedy at udel.edu Fri Mar 16 21:02:31 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 16 Mar 2007 16:02:31 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) References: <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com><45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com><5.1.1.6.0.20070316100847.0289ba88@sparrow.telecommunity.com> <45FAC92D.3090307@gmail.com> Message-ID: "Nick Coghlan" wrote in message news:45FAC92D.3090307 at gmail.com... That may actually be a genuinely useful approach: splitext(name, ignore_leading_dot=False, all_ext=False) Split the extension from a pathname. Returns "(root, ext)". By default, the extension is all characters from the last dot to the end of the string. "ext" will be empty if there are no dots in the name and "root" will be empty if the characters starts with a single dot and that is the only dot in the name. If ignore_leading_dot=True, then a leading dot is always considered part of "root", and is ignored when determining the extension. "root" will never be empty in this case. If all_ext=True, the extension is all characters from the first dot to the end. ===================================== 'first dot' => 'first non-ignored dot' in case both options are true. In the long run (in 3.0), I think ignore_leading_dot should default to True rather than False since I think that is what more people will want. However, I believe the 2-to-3 converter could remove 'ignore_leading_dot=True' and insert 'ignore_leading_dot=False' as needed. Terry Jan Reedy From sjoerd at acm.org Fri Mar 16 21:46:41 2007 From: sjoerd at acm.org (Sjoerd Mullender) Date: Fri, 16 Mar 2007 21:46:41 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45FAC92D.3090307@gmail.com> References: <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <5.1.1.6.0.20070316100847.0289ba88@sparrow.telecommunity.com> <45FAC92D.3090307@gmail.com> Message-ID: <45FB0231.5020903@acm.org> On 03/16/2007 05:43 PM, Nick Coghlan wrote: -- Sjoerd Mullender > Phillip J. Eby wrote: >> At 01:31 PM 3/16/2007 +0100, Martin v. L?wis wrote: >>> Phillip J. Eby schrieb: >>>> 2. Add an optional flag argument to enable the new behavior >>> How would that help backwards compatibility? >> All existing calls to splitext() would work the same way they've done >> for several years, and the documentation would now make it obvious to >> new users of the function that there's a choice about how to handle >> dotfiles. Heck, we could throw in another optional argument to strip >> multiple extensions like .tar.gz. >> > > That may actually be a genuinely useful approach: > > splitext(name, ignore_leading_dot=False, all_ext=False) > > Split the extension from a pathname. Returns "(root, ext)". > By default, the extension is all characters from the last dot to the Presumably that would be the last dot after the last separator (i.e. / and/or \). I would not expect ext to ever contain a separator. > end of the string. "ext" will be empty if there are no dots in the name > and "root" will be empty if the characters starts with a single dot and > that is the only dot in the name. > If ignore_leading_dot=True, then a leading dot is always considered > part of "root", and is ignored when determining the extension. "root" > will never be empty in this case. > If all_ext=True, the extension is all characters from the first dot to > the end. > >> For that matter, the documentation should address the issue that no matter >> what options you use, you may *still* end up with unexpected results, >> for files like "Release 1.2", due to the fuzzy nature of the concept >> of a file >> "extension" on modern OSes. > > Not sure what can be done about that... although such filenames are > likely a big reason why grabbing 'all extensions' will typically be a > bad idea. > > Regards, > Nick. > -- Sjoerd Mullender From pmaupin at gmail.com Fri Mar 16 22:33:46 2007 From: pmaupin at gmail.com (Patrick Maupin) Date: Fri, 16 Mar 2007 16:33:46 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45FAF52B.9000404@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45FA94EE.5080503@v.loewis.de> <45FAEA59.4000804@v.loewis.de> <45FAF52B.9000404@v.loewis.de> Message-ID: On 3/16/07, "Martin v. L?wis" wrote: > Assuming the current behavior is a bug (which I still believe to be > the case), in order to actually make use of the bug fix, you have to > pass the parameter. This will make your code break on old versions. But, that's a GOOD thing. If you don't use the flags approach, your code will silently fail on the old versions. Pat From martin at v.loewis.de Fri Mar 16 22:44:12 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 16 Mar 2007 22:44:12 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45FA94EE.5080503@v.loewis.de> <45FAEA59.4000804@v.loewis.de> <45FAF52B.9000404@v.loewis.de> Message-ID: <45FB0FAC.6080008@v.loewis.de> Patrick Maupin schrieb: > On 3/16/07, "Martin v. L?wis" wrote: >> Assuming the current behavior is a bug (which I still believe to be >> the case), in order to actually make use of the bug fix, you have to >> pass the parameter. This will make your code break on old versions. > > But, that's a GOOD thing. If you don't use the flags approach, your > code will silently fail on the old versions. Whether it will fail depends on the code. It will silently behave differently, but it will not (necessarily) fail. Regards, Martin From glyph at divmod.com Fri Mar 16 22:53:21 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Fri, 16 Mar 2007 21:53:21 -0000 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F9D80B.7050902@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9973C.1030909@v.loewis.de> <20070315215428.7769.91408526.divmod.xquotient.1105@joule.divmod.com> <45F9D80B.7050902@v.loewis.de> Message-ID: <20070316215321.7769.676411877.divmod.xquotient.1114@joule.divmod.com> On 15 Mar, 11:34 pm, martin at v.loewis.de wrote: >glyph at divmod.com schrieb: >>However, the decision was a bad one regardless of the existing policy, >>and sets a bad precedent while we are discussing this policy. I could >>be wrong, but I think it would be reasonable to assume that if Martin >>strongly supports such a change, Martin would oppose a policy which >>would strictly forbid such changes, and it is just such a policy that >>Python needs. > >I still can't guess what policy you have in mind, so I can't object to >it :-) I may accept a policy that rejects this change, but allows >another change to fix the problem. I would oppose a policy that causes >this bug to be unfixable forever. Well, there's *also* the fact that I strongly disagree that this is a bug, but I don't know that I could codify that in a policy. Hence the parallel discussion. However, I do apologize for obliquely referring to this thing I'm working on without showing a work in progress. It's just that different parts of the policy will rely on each other, and I don't want to get bogged down talking about individual details which will be dealt with in the final rev. That, and I am trying to integrate feedback from the ongoing discussion... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070316/22ef4090/attachment.htm From glyph at divmod.com Fri Mar 16 23:00:22 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Fri, 16 Mar 2007 22:00:22 -0000 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: <45F9DFA0.80504@canterbury.ac.nz> References: <20070315151236.18920.1909125356.divmod.quotient.2790@ohm> <45F9733F.7030409@v.loewis.de> <20070315182523.7769.715364850.divmod.xquotient.887@joule.divmod.com> <45F9DFA0.80504@canterbury.ac.nz> Message-ID: <20070316220022.7769.2127931325.divmod.xquotient.1127@joule.divmod.com> On 12:06 am, greg.ewing at canterbury.ac.nz wrote: >glyph at divmod.com wrote: >>Can you suggest any use-cases for thread termination which will *not* >>result in a completely broken and unpredictable heap after the thread >>has died? > >Suppose you have a GUI and you want to launch a >long-running computation without blocking the >user interface. You don't know how long it will >take, so you want the user to be able to cancel >it if he gets bored. That's a perfectly reasonable use-case which doesn't require this feature at all ;). >Interaction with the rest of the program is >extremely limited -- some data is passed in, >it churns away, and some data is returned. It >doesn't matter what happens to its internal >state if it gets interrupted, as it's all going >to be thrown away. If that's true, then the state-sharing features of threads aren't required, which is the right way to design concurrent software anyway. >In that situation, it doesn't seem unreasonable >to me to want to be able to just kill the thread. >I don't see how it could do any more harm than >using KeyboardInterrupt to kill a program, >because that's all it is -- a subprogram running >inside your main program. The key distinction between threads and processes is the sharing of internal program state. >How would you handle this situation? Spawn a process, deliver the result via an event. If you're allergic to event-driven programming, then you can spawn a process *in* a thread, and block in the thread on reading from the process's output, then kill the *process* and have that terminate the output, which terminates the read(). This is a lot like having a queue that you can put a "stop" object into, except the "file" interface provided by OSes is kind of crude. Still no need to kill the thread. At the end of the day though, you're writing a GUI in this use-case and so you typically *must* be cognizant of event-driven issues anyway. Many GUIs (even in the thread-happy world of Windows) aren't thread-safe except for a few specific data-exchange methods, which behave more or less like a queue. One of the 35 different existing ways in which one can spawn a process from Python, I hope, will be sufficient for this case :). -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070316/131a3f48/attachment.html From apt.shansen at gmail.com Fri Mar 16 23:37:31 2007 From: apt.shansen at gmail.com (Stephen Hansen) Date: Fri, 16 Mar 2007 15:37:31 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45FAC92D.3090307@gmail.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <5.1.1.6.0.20070316100847.0289ba88@sparrow.telecommunity.com> <45FAC92D.3090307@gmail.com> Message-ID: <7a9c25c20703161537i261d4c08nb310d5ad257001b@mail.gmail.com> > That may actually be a genuinely useful approach: > > splitext(name, ignore_leading_dot=False, all_ext=False) ... that's perfect. I updated my patch to do it that way! :) --S -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070316/110076b3/attachment.htm From greg.ewing at canterbury.ac.nz Sat Mar 17 00:48:24 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 17 Mar 2007 12:48:24 +1300 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> <45F9E533.7000306@canterbury.ac.nz> <5.1.1.6.0.20070315200047.028365a0@sparrow.telecommunity.com> Message-ID: <45FB2CC8.7010801@canterbury.ac.nz> Mike Krell wrote: > I said the name ".emacs" was used as an example. For that matter, the > name "a.txt" was also used as an example. The use cases are real. So does your application create any file names that have a single dot at the beginning? -- Greg From mbk.lists at gmail.com Sat Mar 17 00:55:59 2007 From: mbk.lists at gmail.com (Mike Krell) Date: Fri, 16 Mar 2007 16:55:59 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45FB2CC8.7010801@canterbury.ac.nz> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> <45F9E533.7000306@canterbury.ac.nz> <5.1.1.6.0.20070315200047.028365a0@sparrow.telecommunity.com> <45FB2CC8.7010801@canterbury.ac.nz> Message-ID: On 3/16/07, Greg Ewing wrote: > Mike Krell wrote: > > > I said the name ".emacs" was used as an example. For that matter, the > > name "a.txt" was also used as an example. The use cases are real. > > So does your application create any file names > that have a single dot at the beginning? > Yes. How many more times would you like me to answer this question? Just in case you'd like me to answer it three more times, here are the answers: Yes, yes, and yes. Mike From arnarbi at gmail.com Wed Mar 14 18:01:43 2007 From: arnarbi at gmail.com (Arnar Birgisson) Date: Wed, 14 Mar 2007 17:01:43 +0000 Subject: [Python-Dev] Looking for a topic for a masters thesis Message-ID: <28012bc60703141001r3ebff769yda0d3997f9fbe354@mail.gmail.com> Hi all, Forgive me for the off-topic post. I'm looking for a topic for a masters thesis in the field of (programming) language theory in Computer Science - in the timeframe of 2-4 years. To combine this with my love of Python, can anyone here suggest a topic that would directly benefit Python in the future - or point me to the appropriate communities/forums to look for such a topic? I'm mostly looking for places to start researching my options. thank you, Arnar From titus at caltech.edu Tue Mar 13 23:36:31 2007 From: titus at caltech.edu (Titus Brown) Date: Tue, 13 Mar 2007 15:36:31 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: Message-ID: <20070313223629.GA27802@caltech.edu> On Tue, Mar 13, 2007 at 02:16:21PM -0700, Guido van Rossum wrote: -> On 3/13/07, Georg Brandl wrote: -> > I'd like to deprecate commands.getstatus() in 2.6. -> > -> > Reason: there is getoutput() and getstatusoutput(). In the latter, "status" -> > means the exit code. getstatus(), however, returns the output of "ls -ld -> > " which is completely nonobvious and confusing. -> -> +1. -> -> > Perhaps the whole commands module can be deprecated in favor of subprocess. -> -> -1. -> -> Reason (for both voteS): I have learned that the getoutput() and -> getstatusoutput() functions in the commands module are exceedingly -> popular amongst Googlers who write simple Python scripts that -> occasionally invoke an external command. It's much simpler than using -> os.popen() or the subprocess module (which is also very popular BTW). What about reimplementing commands.* using subprocess? Or providing a commands.*-compatible interface in the subprocess module? cheers, --titus From titus at caltech.edu Wed Mar 14 04:14:00 2007 From: titus at caltech.edu (Titus Brown) Date: Tue, 13 Mar 2007 20:14:00 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070313223629.GA27802@caltech.edu> Message-ID: <20070314031400.GA31073@caltech.edu> On Tue, Mar 13, 2007 at 04:55:07PM -0700, Guido van Rossum wrote: -> On 3/13/07, Titus Brown wrote: -> > What about reimplementing commands.* using subprocess? Or providing a -> > commands.*-compatible interface in the subprocess module? -> -> What does that buy us? The simplicity of the commands interface on top of the more functional subprocess interface, no? subprocess is very powerful but there isn't a simple way to get the output. See http://docs.python.org/lib/node530.html, and see the docs comparing os.system() with subprocess, http://docs.python.org/lib/node537.html So, if you added 'getstatusoutput' and 'getoutput'-style commands to the subprocess module, you would (a) be able to deprecate a module in the stdlib, simplifying it a bit, and (b) provide simple commands implementing a common use case for subprocess, "run this command and give me the output". (You can already do 'getstatus' with a 'Popen(cmd).wait()'.) cheers, --titus From titus at caltech.edu Fri Mar 16 17:37:04 2007 From: titus at caltech.edu (Titus Brown) Date: Fri, 16 Mar 2007 09:37:04 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070313223629.GA27802@caltech.edu> Message-ID: <20070316163704.GC3107@caltech.edu> -> > >> What about reimplementing commands.* using subprocess? Or providing a -> > >> commands.*-compatible interface in the subprocess module? OK, so as I understand it, the next step would be for me to provide a patch implementing this, right? Or is this PEP-required (please no...)? What do people think of this basic interface? (status, output) = subprocess.get_status_output(cmd) output = subprocess.get_output(cmd) Here 'status' is the 'returncode' from subprocess.Popen, and 'output' would be the combined stdout/stderr. 'commands.getstatus' would be removed entirely [0]. This mimics 'commands' fairly closely, while adhering to PEP 8 guidelines; it's a simple API; and it should be dead easy to implement. It will also have the various advantages people have mentioned: * better error trapping; * better post-fork behavior; * multi-platform support; If this sort of thing goes in, I guess commands.* would then be deprecated with a note saying "go look at these similar commands in subprocess", right? An additional function that I would personally like is: (status, output, errout) = subprocess.get_status_output_err(cmd) although the name is hideous. I'd like to change 'get_status_output' to return a triple, but I'm not sure how big a change that would be relative to the expected behavior from the ancestral commands function. cheers, --titus [0] As per GvR, http://mail.python.org/pipermail/python-dev/2007-March/071926.html From titus at caltech.edu Fri Mar 16 18:19:04 2007 From: titus at caltech.edu (Titus Brown) Date: Fri, 16 Mar 2007 10:19:04 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070313223629.GA27802@caltech.edu> <20070316163704.GC3107@caltech.edu> Message-ID: <20070316171904.GA23991@caltech.edu> On Fri, Mar 16, 2007 at 09:50:29AM -0700, Guido van Rossum wrote: -> > What do people think of this basic interface? -> > -> > (status, output) = subprocess.get_status_output(cmd) -> > -> > output = subprocess.get_output(cmd) -> > -> > Here 'status' is the 'returncode' from subprocess.Popen, and 'output' -> > would be the combined stdout/stderr. 'commands.getstatus' would be -> > removed entirely [0]. -> > -> > This mimics 'commands' fairly closely, while adhering to PEP 8 -> > guidelines; it's a simple API; and it should be dead easy to implement. -> -> Right. Does it also match the style of the API provided by the -> subprocess module? Yes; subprocess has only two "convenience" functions apart from the main class, in fact, and their names are quite short and simple. -> > It will also have the various advantages people have mentioned: -> > -> > * better error trapping; -> > * better post-fork behavior; -> > * multi-platform support; -> > -> > If this sort of thing goes in, I guess commands.* would then be -> > deprecated with a note saying "go look at these similar commands in -> > subprocess", right? -> -> Yes. Another intermediate step might be to rewrite the commands module -> to call the new APIs in the subprocess module. My preference would be to push people towards subprocess, and the more capable & complex API there, by deprecating commands completely. I'm happy to do it either way. -> > An additional function that I would personally like is: -> > -> > (status, output, errout) = subprocess.get_status_output_err(cmd) -> > -> > although the name is hideous. I'd like to change 'get_status_output' -> > to return a triple, but I'm not sure how big a change that would be -> > relative to the expected behavior from the ancestral commands function. -> -> Call it get_status_output_errors and you have my blessings. If at all -> possible, get_status_output should *not* just concatenate the output -> and errors from this API, but attempt to really *merge* the stdout and -> stderr stream so that if they are interleaved they come out in the -> right order. That's what the old getstatusoutput() did by putting -> '2>&1' around the command. I believe that this is what subprocess.Popen(..., stdout=subprocess.PIPE, stderr=subprocess.STDOUT) does, but I will check to be sure. --titus From cabird at gmail.com Thu Mar 8 21:08:49 2007 From: cabird at gmail.com (Christian Bird) Date: Thu, 8 Mar 2007 12:08:49 -0800 Subject: [Python-Dev] who gets paid for this? Message-ID: Hi all, I'm a grad student at UC Davis studying the python community and I wanted to know if some on this list could help me out. I'm studying the factors that affect people "graduating" from being mailing list participant to developers with write access to the repository. Is it possible to find out who is being employed to work on python and who is doing it on their own time? Some of my data points to there being two ways that people make the jump. More specifically, could those who worked on python as some aspect of their job prior to getting repo access let me know? Or if there are devs who know this information about others, I'd be really appreciative to get it. Thanks a lot. -- Christian Bird -- Christian Bird cabird at gmail.com From k.danilov at ratmirlabs.com Tue Mar 13 12:13:21 2007 From: k.danilov at ratmirlabs.com (K.Danilov aka koder) Date: Tue, 13 Mar 2007 13:13:21 +0200 Subject: [Python-Dev] [ 1673007 ] urllib2 requests history + HEAD Message-ID: <45F68751.9010806@ratmirlabs.com> > From: Facundo Batista > > This patch was posted by "koder_ua". > I think that Request must have a "request type" parameters, so people > can send "HEAD" requests easily. > But it seems to me that keeping a request history in the module is bad, > because it can easily grow up to thousands and explode (a.k.a. consume > too much memory). > Fo example, I have a web service, running 7x24, and opening another web > service, with around 10 requests per second. This means, keeping the > history (around 50bytes each request), 1.2 GB of RAM in only a month! > So, I'll close this patch as "Rejected", for this reason, if anyone > raises objections. > Regards, > -- > . Facundo This is probably a misunderstanding. Request's history don't store in the "module".They store in two places: 1) In Request object (for current request, so they would be destroy?d with it); 2) In HTTPConnection object (while request redirects). In HTTPConnection history stores only for current served Request. Even if You use the same HTTPConnection for many Requests, they (HTTPConnection) clear history every time when new Request starts. # from httplib HTTPConnection.putrequest patched str = '%s %s %s' % (method, url, self._http_vsn_str) self._output(str) self.sended_hdrs = [str] <<< previous history die here ___Full history for all processed request didn't not stored in any place____. P.S. This message may be duplicated - first copy i sent from gmail.com and it didn't reach mail list for some unknown for me reasons. --- KDanilov aka koder(aka koder_ua) From turnbull at sk.tsukuba.ac.jp Fri Mar 16 19:57:53 2007 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sat, 17 Mar 2007 03:57:53 +0900 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45FA8E18.4040805@v.loewis.de> References: <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <5.1.1.6.0.20070314152603.04989b90@sparrow.telecommunity.com> <5.1.1.6.0.20070314170719.0470fa18@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <45FA8E18.4040805@v.loewis.de> Message-ID: <17914.59569.386737.970321@uwakimon.sk.tsukuba.ac.jp> "Martin v. L?wis" writes: > Phillip J. Eby schrieb: > > Some other options: > > > > 1. Deprecate splitext() and remove it in 3.0 > > How would that help the problem? Isn't it useful to have a function > that strips off the extension? No. It's useful to have a function that performs a well-specified algorithm on file names containing dots, but (except on certain file systems such as FAT) "the extension" does not uniquely exist. People and programs will disagree on the decomposition of "file.tar.gz". On FAT file systems, it's defined incorrectly according to you. As IIRC glyph pointed out, if you're going to include either shell semantics (dotfiles) or content semantics (file "type" for a generic "open anything" command) in the specification of "file extension", what I prefer is guess_file_type(), not splitext(). A more emphatic way to express this is, I would never use a library function whose semantics were defined as "split a file name into the base and the extension" because I would expect gratuitous backward incompatibility of the kind you have introduced (and it could go either way).[1] N.B. Backward compatibility can be defined by reference to an implementation (often denigrated as "bug compatibility") or to a specification. This change is backward incompatible with respect to the implementation and the docstring specification. I would personally prefer the 2.4 definition of splitext(), merely because it's so simple. I would (absent this long discussion) always have to look up the treatment of dotfiles, anyway, and my own only use (in one function, 3 calls) of splitext is precisely def versioned_file_name (filename, version): base, ext = splitext (filename) return "%s.%s%s" % (base,version,ext) > > 2. Add an optional flag argument to enable the new behavior > > How would that help backwards compatibility? As Steve Holden points out, by preserving it if the flag is omitted. That is so obvious that I think merely asking that question is perverse. You seem to be granting official status to the unwritten and controversial "intuitive" specification that many programmers guess from the name. That is way out of line with any sane interpretation of "compatibility with past versions of Python". I think all of the advocates of changing the function rather than the library reference are being very short-sighted. I agree with you that looking at this one case, it will be very expensive for all those who have (currently broken) code that expects splitext to treat dotfiles as having a base that starts with a dot (rather than an empty base) to change to use a new function. (I think the realistic solution for them is monkeypatching.) But using this to justify the backward incompatibility is like thinking you can eat just one potato chip, and so not go off your diet. The incompatibility costs of applying this greedy algorithm to all cases where the Python specification differs from common intuition will not merely be very expensive, they will be astronomically so -- but practically invisible because the cost will be in terms of developers who refuse to upgrade their applications to take advantage of new features of Python because their own library code gets broken. The only path I can see where it makes sense to make this change is as an appendix to glyph's PEP. The appendix could give a list of such changes and the decision, relative to some base version. Or it could specify that contributions to 2.6 can be backward incompatible, but not afterward. In that case the promise to eat only this one potato chip becomes credible. I prefer the explicit list approach, that would force the discussion to occur in one place, so that both proponents and opponents of each change would be made aware of how many such changes are being made. Footnotes: [1] From steve at holdenweb.com Sat Mar 17 03:31:01 2007 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Mar 2007 22:31:01 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F99200.7040303@v.loewis.de> <45F9A8F1.6070209@v.loewis.de> <45F9E533.7000306@canterbury.ac.nz> <5.1.1.6.0.20070315200047.028365a0@sparrow.telecommunity.com> <45FB2CC8.7010801@canterbury.ac.nz> Message-ID: Mike Krell wrote: > On 3/16/07, Greg Ewing wrote: >> Mike Krell wrote: >> >>> I said the name ".emacs" was used as an example. For that matter, the >>> name "a.txt" was also used as an example. The use cases are real. >> So does your application create any file names >> that have a single dot at the beginning? >> > > Yes. How many more times would you like me to answer this question? > > Just in case you'd like me to answer it three more times, here are the answers: > > Yes, yes, and yes. > So that would be a yes, then. Perhaps you'd like to remind me that "backward compatibilty" includes the necessity to run new programs on old versions of Python, too? Then I can stop wasting everyone's time. Even though I am no fonder of code breakage than I was. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From kbk at shore.net Sat Mar 17 05:54:46 2007 From: kbk at shore.net (Kurt B. Kaiser) Date: Fri, 16 Mar 2007 23:54:46 -0500 (EST) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200703170454.l2H4skKe003172@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 342 open (-38) / 3712 closed (+54) / 4054 total (+16) Bugs : 951 open (-14) / 6588 closed (+33) / 7539 total (+19) RFE : 257 open (-15) / 266 closed (+13) / 523 total ( -2) New / Reopened Patches ______________________ ftp.python.org does not exist anymore (2007-03-12) CLOSED http://python.org/sf/1678662 opened by Bj?rn Lindqvist fix a bug mixing up 0.0 and-0.0 (2007-03-11) http://python.org/sf/1678668 opened by Alex Martelli Allow "datetime in date" (2007-03-12) CLOSED http://python.org/sf/1679204 opened by Collin Winter Documentation for fnmatch.translate() (2007-03-12) CLOSED http://python.org/sf/1679379 opened by Bj?rn Lindqvist __slots__ tuple modified inplace if it contains unicode name (2007-03-13) CLOSED http://python.org/sf/1680015 opened by ?iga Seilnacht CGIHTTPServer doesn't handle path names with embeded space (2006-08-06) http://python.org/sf/1535504 reopened by htgoebel Add tests for pipes module (test_pipes) (2007-03-14) http://python.org/sf/1680959 opened by Alan McIntyre remove sys.exitfunc, rewrite atexit in C (2007-03-14) http://python.org/sf/1680961 opened by Georg Brandl Use "alive" instead of "active" in the docs for threading (2007-03-14) CLOSED http://python.org/sf/1680978 opened by Bj?rn Lindqvist binary and new-style octal literals (2007-03-14) http://python.org/sf/1681002 opened by Thomas Wouters PEP 3115 patch (2007-03-14) http://python.org/sf/1681101 opened by Guido van Rossum Patch for [ 1603150 ] wave module forgets to close... (2007-03-15) CLOSED http://python.org/sf/1681153 opened by Patricio Olivares email.header unicode fix (2007-03-15) http://python.org/sf/1681333 opened by Tokio Kikuchi Add triangular distribution to random (2007-03-15) http://python.org/sf/1681432 opened by Wladmir splitext of dotfiles, incl backwards compat and migration (2007-03-16) http://python.org/sf/1681842 opened by Stephen Hansen Patches Closed ______________ Patch to 742342 Crash on recursive reload (2004-03-23) http://python.org/sf/922167 closed by collinwinter add optional CWD argument to os.path.abspath() (2003-01-30) http://python.org/sf/677890 closed by collinwinter ftp.python.org does not exist anymore (2007-03-11) http://python.org/sf/1678662 closed by collinwinter fix for external test regression in test.regrtest (2003-09-17) http://python.org/sf/808210 closed by collinwinter urllib2.URLError don't calll IOError.__init__ (2003-10-02) http://python.org/sf/816787 closed by collinwinter Refactor test_threadedtempfile.py to use unittest. (2007-02-28) http://python.org/sf/1670993 closed by collinwinter urllib2.urlopen() raises OSError instead of URLError (2007-02-24) http://python.org/sf/1668100 closed by gbrandl site.py small ?bug fix | change? (2007-03-10) http://python.org/sf/1677862 closed by loewis adding __dir__ (2006-11-06) http://python.org/sf/1591665 closed by gbrandl Refactor test_operations.py to use unittest. (2007-03-10) http://python.org/sf/1678088 closed by collinwinter Add current dir when running try_run test program (2005-03-21) http://python.org/sf/1168055 closed by collinwinter Allow "datetime in date" (2007-03-12) http://python.org/sf/1679204 closed by collinwinter Documentation for fnmatch.translate() (2007-03-12) http://python.org/sf/1679379 closed by gbrandl comments to clarify complexobject.c (2007-01-23) http://python.org/sf/1642844 closed by gbrandl Fix for bugs relating to ntpath.expanduser() (2004-05-20) http://python.org/sf/957650 closed by gbrandl Add %var% support to ntpath.expandvars (2006-10-10) http://python.org/sf/1574252 closed by gbrandl CSV DictWriter Errors (2007-01-15) http://python.org/sf/1635454 closed by gbrandl test_cmd_line fails on Windows (2006-09-15) http://python.org/sf/1559413 closed by loewis Performance enhancements. (2006-09-09) http://python.org/sf/1555098 closed by gbrandl Tweak pydoc to speak about with and CONTEXTMANAGERS (2006-08-18) http://python.org/sf/1542681 closed by gbrandl Fix building the source within exec_prefix (2006-10-03) http://python.org/sf/1569798 closed by gbrandl email.message.py charset can be unicode instance (2006-03-14) http://python.org/sf/1449244 closed by loewis Allow __class __ assignment for classes with __slots__ (2006-12-28) http://python.org/sf/1623563 closed by zseil __slots__ tuple modified inplace if it contains unicode name (2007-03-13) http://python.org/sf/1680015 closed by zseil ConfigParser does not quote % (2007-01-15) http://python.org/sf/1635639 closed by gbrandl Allow textwrap to preserve leading and trailing whitespace (2006-10-20) http://python.org/sf/1581073 closed by gbrandl posix.readlink doesn't use filesystemencoding (2006-10-19) http://python.org/sf/1580674 closed by gbrandl Make Imap Error more helpful (2006-11-29) http://python.org/sf/1605192 closed by gbrandl option to leave tempfile.NamedTemporaryFile around on close (2006-08-10) http://python.org/sf/1537850 closed by gbrandl Let timeit accept functions (2006-08-03) http://python.org/sf/1533909 closed by gbrandl pydoc render_doc (2006-07-28) http://python.org/sf/1530482 closed by gbrandl Switch syntax (partial PEP 275) (2006-06-11) http://python.org/sf/1504199 closed by gbrandl Patch for bug #931877 Segfault in object_reduce_ex (2006-04-01) http://python.org/sf/1462488 closed by zseil Remove sys.exitfunc (2006-07-03) http://python.org/sf/1516375 closed by gbrandl Patch to commands.py to enable callback support (2006-07-05) http://python.org/sf/1517586 closed by gbrandl Remove deprecated functions from operator (2006-07-03) http://python.org/sf/1516309 closed by collinwinter Adding support for _Bool to ctypes as c_bool (2007-01-31) http://python.org/sf/1649190 closed by theller kwdargs for compile/__import__ (2006-03-07) http://python.org/sf/1444529 closed by gbrandl Add restart debugger command to pdb.py (2005-12-30) http://python.org/sf/1393667 closed by gbrandl httplib error handling and HTTP/0.9 fix (2005-10-04) http://python.org/sf/1312980 closed by gbrandl assert for NULL in Py_INCREF Py_DECREF (2006-07-06) http://python.org/sf/1517947 closed by gbrandl pdb: fix for 1326406 (import __main__ pdb failure) (2006-02-11) http://python.org/sf/1429539 closed by gbrandl pydoc requires o.__nonzero__() == True (2005-05-03) http://python.org/sf/1194449 closed by gbrandl binascii.b2a_qp does not handle binary data correctly (2005-04-18) http://python.org/sf/1185447 closed by gbrandl Fix for 767111, 'AttributeError thrown by urllib.open_http' (2007-02-25) http://python.org/sf/1668132 closed by gbrandl Use "alive" instead of "active" in the docs for threading (2007-03-14) http://python.org/sf/1680978 closed by gbrandl Patch for [ 1603150 ] wave module forgets to close... (2007-03-15) http://python.org/sf/1681153 closed by gbrandl Add Thread.isActive() (2005-01-23) http://python.org/sf/1107656 closed by gbrandl Refactor test_popen2.py to use unittest. (2007-03-09) http://python.org/sf/1676994 closed by collinwinter debugger ``condition`` and ``ignore`` exception handling (2005-04-29) http://python.org/sf/1192590 closed by collinwinter new function: os.path.relpath (2005-10-27) http://python.org/sf/1339796 closed by collinwinter Fix error/crash in AST: syntaxerror in complex ifs (2007-01-23) http://python.org/sf/1642547 closed by collinwinter release lock on exception in threading.Thread.join() (2005-07-18) http://python.org/sf/1240614 closed by nnorwitz extending os.walk to support following symlinks (2005-08-26) http://python.org/sf/1273829 closed by gbrandl New / Reopened Bugs ___________________ Lots of errors reported by valgrind in 2.4.4 and 2.5 (2007-01-05) CLOSED http://python.org/sf/1629158 opened by Anton Tropashko dead link in tkinter documentation (2007-03-11) http://python.org/sf/1678710 opened by Ben Collver Lib/io.py open uses unset "bs" (2007-03-12) http://python.org/sf/1679498 opened by Jim Jewett error in the sys module documentation (2007-03-13) CLOSED http://python.org/sf/1679652 opened by David Fillinger "my" should be "may" (2007-03-13) CLOSED http://python.org/sf/1679784 opened by edward Importing SystemRandom wastes entropy. (2007-03-13) CLOSED http://python.org/sf/1680034 reopened by stephent98 Importing SystemRandom wastes entropy. (2007-03-13) CLOSED http://python.org/sf/1680034 opened by Steve Tyler Misleading exception from unicode.__contains__ (2007-03-13) http://python.org/sf/1680159 opened by Jan Hudec urllib.urlopen() raises AttributeError (2007-03-14) CLOSED http://python.org/sf/1680230 opened by Bj?rn Lindqvist httplib fails to parse response on HEAD request (2007-03-13) CLOSED http://python.org/sf/1680312 opened by Patrick Altman execfile locks file forever if there are any syntax errors (2007-03-14) CLOSED http://python.org/sf/1681020 opened by virwen webbrowser priorities (2007-03-15) CLOSED http://python.org/sf/1681228 opened by Lukas Lalinsky WindowsError in webbrowser.open (2007-03-15) CLOSED http://python.org/sf/1681248 opened by Lukas Lalinsky Python and Indeterminate Forms (Math) (2007-03-15) http://python.org/sf/1681671 opened by jehahn subprocess.Popen fails with socket._fileobject on Windows (2007-03-15) http://python.org/sf/1681674 opened by Hartmut Goebel too long string causes segmentation fault (2007-03-16) http://python.org/sf/1681762 opened by L mkdtemp fails on Windows if username has non-ASCII character (2007-03-16) http://python.org/sf/1681974 opened by Markus Niemist? unittest documentation is incomplete (2007-03-16) http://python.org/sf/1681984 opened by STINNER Victor [PATCH] TypeError swallowing in UNPACK_SEQUENCE opcode (2007-03-16) http://python.org/sf/1682205 opened by Piet Delport Problems with urllib2 read() (2007-03-16) http://python.org/sf/1682241 opened by Lucas Malor docutils clarification request for "rest" (2007-03-16) http://python.org/sf/1682403 opened by j vickroy Bugs Closed ___________ Lots of errors reported by valgrind in 2.4.4 and 2.5 (2007-01-06) http://python.org/sf/1629158 closed by loewis absolute import patch breaks external users of test.regrtest (2003-08-31) http://python.org/sf/798274 closed by collinwinter email._parseaddr AddrlistClass bug (2007-01-06) http://python.org/sf/1629369 closed by bwarsaw Weird behavior Exception with unicode string (2007-03-12) http://python.org/sf/1678647 closed by gbrandl Python still uses broken -xcode option on Solaris/x86 (2007-03-07) http://python.org/sf/1675511 closed by loewis Python crashes if recursively reloading modules (2003-05-23) http://python.org/sf/742342 closed by collinwinter inspect trouble when source file changes (2007-01-05) http://python.org/sf/1628987 closed by collinwinter fnmatch.translate undocumented (2007-01-08) http://python.org/sf/1630844 closed by gbrandl Install fails with no error (2007-02-24) http://python.org/sf/1667877 closed by sf-robot Urllib2.urlopen() raises OSError w/bad HTTP Location header (2006-11-07) http://python.org/sf/1591774 closed by gbrandl error in the sys module documentation (2007-03-13) http://python.org/sf/1679652 closed by gbrandl Exception when compressing certain data with bz2 (2006-12-27) http://python.org/sf/1622896 closed by gbrandl "my" should be "may" (2007-03-13) http://python.org/sf/1679784 closed by gbrandl Importing SystemRandom wastes entropy. (2007-03-13) http://python.org/sf/1680034 closed by rhettinger Importing SystemRandom wastes entropy. (2007-03-13) http://python.org/sf/1680034 closed by gbrandl typo in encoding name in email package (2006-09-12) http://python.org/sf/1556895 closed by bwarsaw pdb breaks programs which import from __main__ (2005-10-14) http://python.org/sf/1326406 closed by gbrandl ntpath.expanduser() is still wrong (2003-08-27) http://python.org/sf/796219 closed by gbrandl binascii.b2a_qp oddities (2004-12-14) http://python.org/sf/1085283 closed by gbrandl urllib.urlopen() raises AttributeError (2007-03-13) http://python.org/sf/1680230 closed by gbrandl httplib fails to parse response on HEAD request (2007-03-14) http://python.org/sf/1680312 closed by gbrandl email.header decode within word (2006-10-22) http://python.org/sf/1582282 closed by bwarsaw AttributeError thrown by urllib.open_http (2003-07-07) http://python.org/sf/767111 closed by gbrandl urllib.py: AttributeError on BadStatusLine (2006-02-11) http://python.org/sf/1429783 closed by gbrandl poor urllib error handling (2006-11-09) http://python.org/sf/1593751 closed by gbrandl urllib does not handle Connection reset (2003-12-07) http://python.org/sf/855819 closed by gbrandl commands module doesn't support background commands (2004-02-06) http://python.org/sf/891832 closed by gbrandl commands.getstatusoutput(): cmd.exe support (2002-01-20) http://python.org/sf/506100 closed by gbrandl Installation path sent to configure (2005-05-08) http://python.org/sf/1197883 closed by gbrandl execfile locks file forever if there are any syntax errors (2007-03-14) http://python.org/sf/1681020 closed by gbrandl wave module forgets to close file on exception (2006-11-26) http://python.org/sf/1603150 closed by gbrandl webbrowser priorities (2007-03-15) http://python.org/sf/1681228 closed by gbrandl WindowsError in webbrowser.open (2007-03-15) http://python.org/sf/1681248 closed by gbrandl Segfault in object_reduce_ex (2004-04-08) http://python.org/sf/931877 closed by zseil Documentation missing for OptionGroup class in optparse (2007-02-21) http://python.org/sf/1665333 closed by nnorwitz New / Reopened RFE __________________ optparse should support arbitrary number of arguments (2006-07-24) CLOSED http://python.org/sf/1527705 reopened by riteshsarraf RFE Closed __________ csv.DictWriter: Include offending name in error message (2007-01-13) http://python.org/sf/1634717 closed by gbrandl Update to the BEGIN_ALLOW_THREADS macros (2001-07-23) http://python.org/sf/443775 closed by nnorwitz moving the Mac/ src dir to a diff packag (2001-09-24) http://python.org/sf/464605 closed by nnorwitz assert return values (2002-07-24) http://python.org/sf/585915 closed by nnorwitz Including python-codecs codecs in standard distribution? (2003-03-20) http://python.org/sf/706970 closed by nnorwitz optparse help text is not easily extensible (2004-01-16) http://python.org/sf/878458 closed by nnorwitz dir(mod) OK or use vars(mod).keys()? (2004-03-29) http://python.org/sf/925537 closed by gbrandl Allow any lvalue for function definitions (2004-05-08) http://python.org/sf/950644 closed by gvanrossum Extension to optparse: options with optional arguments (2004-11-25) http://python.org/sf/1073198 closed by nnorwitz Bad optparse help wrapping with multiple paragraphs (2005-06-16) http://python.org/sf/1222235 closed by nnorwitz python executable optionally should search script on PATH (2005-12-13) http://python.org/sf/1379573 closed by loewis optparse should support arbitrary number of arguments (2006-07-24) http://python.org/sf/1527705 closed by nnorwitz optparse "store" action should not gobble up next option (2007-01-03) http://python.org/sf/1627266 closed by nnorwitz isinstance.__doc__ is somewhat irritating (2007-02-27) http://python.org/sf/1670167 closed by gbrandl From pmaupin at gmail.com Sat Mar 17 06:48:48 2007 From: pmaupin at gmail.com (Patrick Maupin) Date: Sat, 17 Mar 2007 00:48:48 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F9A8F1.6070209@v.loewis.de> <45F9E533.7000306@canterbury.ac.nz> <5.1.1.6.0.20070315200047.028365a0@sparrow.telecommunity.com> <45FB2CC8.7010801@canterbury.ac.nz> Message-ID: On 3/16/07, Steve Holden wrote: > Perhaps you'd like to remind me that "backward compatibilty" includes > the necessity to run new programs on old versions of Python, too? Ahh, but you see, new programs are the easy part. You actually have at least four choices of different levels of backward compatibility: 1) If you are absolutely sure that the code in your program will never be used to work with filenames with leading dots, you are already finished! (Note that this goal is much easier to achieve if you don't release the source, or at least write it so badly that nobody will want to reuse the code.) 2) If you think that most users of your program won't use filenames with leading dots, and you don't plan on supporting it after a year or so, just make sure it works with 2.5. 3) Conversely, if you're not that bothered about leading dots, and don't think you'll have all the bugs out of your program for a year or so anyway, just wait for 2.6. (All the "cool" potential users of your program will be on the bleeding edge, anyway.) 4) Finally, if you're one of those Luddite sticklers who wants to try to ruin everybody's job security by writing code that works right now and doesn't need to be touched later, just write your own version of this function. I would have suggested that you could reuse the underlying functionality in conjunction with a version check, but it has been pointed out that the existence of tests and docstrings which perfectly match the code is no impediment to change, so Philip might get mad enough to change it back for 3.1, and then your version check would be obsolete. > Then I can stop wasting everyone's time. Even though I am no fonder of > code breakage than I was. Fortunately, for new code (at least for this particular change!), you don't have to worry about breakage. I'm sure this discussion has been so indelibly etched into your brain that you won't forget to check your filename management functions very carefully. Sorry, were you being sarcastic? I didn't realize that. Or am I prevaricating again? Regards, Pat From steve at holdenweb.com Sat Mar 17 11:13:51 2007 From: steve at holdenweb.com (Steve Holden) Date: Sat, 17 Mar 2007 06:13:51 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F9A8F1.6070209@v.loewis.de> <45F9E533.7000306@canterbury.ac.nz> <5.1.1.6.0.20070315200047.028365a0@sparrow.telecommunity.com> <45FB2CC8.7010801@canterbury.ac.nz> Message-ID: <45FBBF5F.4060109@holdenweb.com> Patrick Maupin wrote: > On 3/16/07, Steve Holden wrote: [...] >> Then I can stop wasting everyone's time. Even though I am no fonder of >> code breakage than I was. > > Fortunately, for new code (at least for this particular change!), you > don't have to worry about breakage. I'm sure this discussion has been > so indelibly etched into your brain that you won't forget to check > your filename management functions very carefully. > > Sorry, were you being sarcastic? I didn't realize that. Or am I > prevaricating again? > I don't know whether you are prevaricating again, but I can definitely confirm I wasn't being sarcastic. I don't think sarcasm helps in a discussion like this. I was just apologizing to the group for a rat hole I drew the thread down briefly before someone (Martin, I think) pointed out the incompleteness/incorrectness of my reply. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From steve at holdenweb.com Sat Mar 17 11:13:51 2007 From: steve at holdenweb.com (Steve Holden) Date: Sat, 17 Mar 2007 06:13:51 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F9A8F1.6070209@v.loewis.de> <45F9E533.7000306@canterbury.ac.nz> <5.1.1.6.0.20070315200047.028365a0@sparrow.telecommunity.com> <45FB2CC8.7010801@canterbury.ac.nz> Message-ID: <45FBBF5F.4060109@holdenweb.com> Patrick Maupin wrote: > On 3/16/07, Steve Holden wrote: [...] >> Then I can stop wasting everyone's time. Even though I am no fonder of >> code breakage than I was. > > Fortunately, for new code (at least for this particular change!), you > don't have to worry about breakage. I'm sure this discussion has been > so indelibly etched into your brain that you won't forget to check > your filename management functions very carefully. > > Sorry, were you being sarcastic? I didn't realize that. Or am I > prevaricating again? > I don't know whether you are prevaricating again, but I can definitely confirm I wasn't being sarcastic. I don't think sarcasm helps in a discussion like this. I was just apologizing to the group for a rat hole I drew the thread down briefly before someone (Martin, I think) pointed out the incompleteness/incorrectness of my reply. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From aahz at pythoncraft.com Sat Mar 17 15:36:13 2007 From: aahz at pythoncraft.com (Aahz) Date: Sat, 17 Mar 2007 07:36:13 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45F8ECFE.2030105@v.loewis.de> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> <45F8ECFE.2030105@v.loewis.de> Message-ID: <20070317143613.GA23013@panix.com> On Thu, Mar 15, 2007, "Martin v. L??wis" wrote: > > I'm not quite sure what "it" is, here. If "it" is that there be no > incompatible changes in Python: this is not policy, and not even > consensus. Instead, policy (as enforced by the release manager), and > consensus is that bug fix releases (2.x.y) must not show incompatible > behavior. For feature releases (2.x), incompatible behavior is > acceptable (at least this is what I thought consensus is, but > apparently I'm wrong). You are not wrong on one level, but you are wrong on another: if a change that may not be a bugfix gets any legitimate objections, the bar for the change becomes much higher, and that goes double or triple if the change will result in silent breakage of existing programs. If this discussion had occurred three years ago, I would be more inclined toward your position, but right now we have another outlet for silent incompatible changes: Python 3.0. I'm not opposed to extending the splitext API in 2.6 as one solution for this problem, but I still believe that pushing the change to 3.0 is best (assuming we make the change at all, because the people arguing against any change do have a point about Windows -- and in the end, Windows is the largest CONSUMER of Python programs). I do agree with the surprise expressed about your claim that extending the API isn't backward compatible -- the point is that any code using pre-2.6 splitext() API will work the same across 2.x regardless of whether the code is written after 2.6 is released. Of course anyone who uses the extended API is backward incompatible, but that's a much different kind of problem. Because this discussion has gone on so long, it seems to me that a micro-PEP would be good for summarizing. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "I disrespectfully agree." --SJM From edloper at gradient.cis.upenn.edu Sat Mar 17 21:08:50 2007 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Sat, 17 Mar 2007 16:08:50 -0400 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <04109226-15B9-4702-9DE3-2FA41958AD46@gradient.cis.upenn.edu> References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> <04109226-15B9-4702-9DE3-2FA41958AD46@gradient.cis.upenn.edu> Message-ID: <2D8B37F0-3D74-4B67-9789-3F45110FF7F6@gradient.cis.upenn.edu> Sorry, forgot to include the output of doctest_driver.py --help. Here it is: -Edward Usage: doctest_driver.py [options] NAME ... Options: --version show program's version number and exit -h, --help show this help message and exit Actions (default=check): --check Verify the output of the doctest examples in the given files. -u, --update Update the expected output for new or out-of- date doctest examples in the given files. In particular, find every example whose actual output does not match its expected output; and replace its expected output with its actual output. You will be asked to verify the changes before they are written back to the file; be sure to check them over carefully, to ensure that you don't accidentally create broken test cases. --debug Verify the output of the doctest examples in the given files. If any example fails, then enter the python debugger. Reporting: -v, --verbose Increase verbosity. -q, --quiet Decrease verbosity. -d, --udiff Display test failures using unified diffs. --cdiff Display test failures using context diffs. --ndiff Display test failures using ndiffs. Output Comparison: --ellipsis Allow "..." to be used for ellipsis in the expected output. --normalize_whitespace Ignore whitespace differences between the expected output and the actual output. From edloper at gradient.cis.upenn.edu Sat Mar 17 20:57:40 2007 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Sat, 17 Mar 2007 15:57:40 -0400 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> Message-ID: <04109226-15B9-4702-9DE3-2FA41958AD46@gradient.cis.upenn.edu> On Feb 15, 2007, at 2:59 PM, Raymond Hettinger wrote: > * Enhance doctest with a method that computes updated doctest > results. If the > only change I make to a matrix suite is to add commas to long > numbers in the > matrix repr(), then it would be nice to have an automated way to > update the > matrix output on all the other test cases in the module. A while back I wrote up a command-line doctest driver that does this and more [1]. The output of "doctest_driver.py --help" is below; note in particular the --update and --debug options. I also wrote an emacs mode [2] that has support for running doctest & putting the output (^c^c) or diffs (^c^d) in an output buffer; and for updating the output of selected examples (^c^r). I never really got around to doing official releases for either (although doctest-mode.el is part of the python-mode.el package). I could move some of this code into the doctest mode, if it seems appropriate; or I could just go ahead and release the driver and publicize it. -Edward [1] http://nltk.svn.sourceforge.net/viewvc/nltk/trunk/nltk/nltk_lite/ test/doctest_driver.py?view=markup [2] http://python-mode.svn.sourceforge.net/viewvc/*checkout*/python- mode/trunk/python-mode/doctest-mode.el?revision=424 From ewan at xensource.com Sat Mar 17 21:38:26 2007 From: ewan at xensource.com (Ewan Mellor) Date: Sat, 17 Mar 2007 20:38:26 +0000 Subject: [Python-Dev] Bug in inspect module Message-ID: <20070317203826.GA9617@leeni.uk.xensource.com> All, I have a problem being reported (by Xen users) where inspect.stack() is throwing IndexError. I think that this is a bug in inspect.py -- findsource generally throws IOError when it can't find a particular source file, but in the case where it finds a source file, but that file is shorter than expected, then findsource throws IndexError, and this is propagated all the way out of inspect.stack(). I'm not sure why inspect.py is finding source files that don't match the code being executed -- it seems to be dependent upon the install environment, because it's not affecting most people. That said, I think that it's reasonable to cope with a too-short source file in the same way as we cope with missing source files -- by throwing IOError from findsource and handling that exception later. Here's a patch. Cheers, Ewan Mellor, XenSource. --- inspect.py.orig 2007-03-17 17:01:56.410399391 +0000 +++ inspect.py 2007-03-17 17:16:36.026005726 +0000 @@ -490,6 +490,8 @@ if not hasattr(object, 'co_firstlineno'): raise IOError('could not find function definition') lnum = object.co_firstlineno - 1 + if len(lines) - 1 < lnum: + raise IOError('source file is too short') pat = re.compile(r'^(\s*def\s)|(.*(? 0: if pat.match(lines[lnum]): break From greg.ewing at canterbury.ac.nz Sun Mar 18 00:49:55 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 18 Mar 2007 11:49:55 +1200 Subject: [Python-Dev] Rationale for NamedTemporaryFile? Message-ID: <45FC7EA3.4070808@canterbury.ac.nz> I've just discovered the hard way that NamedTemporaryFile automatically deletes the file when you close it. That doesn't seem very useful to me, since surely the reason you're using NamedTemporaryFile instead of TemporaryFile is that you want to do something else with it afterwards? What's the rationale for this behaviour? -- Greg From collinw at gmail.com Sun Mar 18 02:09:06 2007 From: collinw at gmail.com (Collin Winter) Date: Sat, 17 Mar 2007 20:09:06 -0500 Subject: [Python-Dev] Bug in inspect module In-Reply-To: <20070317203826.GA9617@leeni.uk.xensource.com> References: <20070317203826.GA9617@leeni.uk.xensource.com> Message-ID: <43aa6ff70703171809pb952de2yf7398c22b01d5ca1@mail.gmail.com> On 3/17/07, Ewan Mellor wrote: > I have a problem being reported (by Xen users) where inspect.stack() is > throwing IndexError. I think that this is a bug in inspect.py -- findsource > generally throws IOError when it can't find a particular source file, but in > the case where it finds a source file, but that file is shorter than expected, > then findsource throws IndexError, and this is propagated all the way out of > inspect.stack(). > > I'm not sure why inspect.py is finding source files that don't match the code > being executed -- it seems to be dependent upon the install environment, > because it's not affecting most people. That said, I think that it's > reasonable to cope with a too-short source file in the same way as we cope > with missing source files -- by throwing IOError from findsource and handling > that exception later. FYI: this has been reported at least once before: see bug #1628987 (http://python.org/sf/1628987). The problem (in the bug report, at least) is that the source file changes between compilation and when findsource() is called, and findsource() picks up the modified version. Collin Winter From tony.meyer at gmail.com Sun Mar 18 04:12:02 2007 From: tony.meyer at gmail.com (Tony Meyer) Date: Sun, 18 Mar 2007 15:12:02 +1200 Subject: [Python-Dev] Encouraging developers In-Reply-To: <79990c6b0703070542w1479d8e4rdf30b1cc9a5f2a78@mail.gmail.com> References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <45ED0353.2090201@scottdial.com> <45ED07EE.4060402@v.loewis.de> <79990c6b0703060320t23d4487rcc97ec69c669a8ee@mail.gmail.com> <45ED57CA.7090101@v.loewis.de> <45ED5C81.6070702@scottdial.com> <79990c6b0703070542w1479d8e4rdf30b1cc9a5f2a78@mail.gmail.com> Message-ID: On 8/03/2007, at 2:42 AM, Paul Moore wrote: > On 06/03/07, Scott Dial wrote: >> Sadly the sf tracker doesn't let you search for "With comments >> by". The >> patch I was making reference to was 1410680. Someone else actually >> had >> wrote a patch that contained bugs and I corrected them. And with >> that, I >> was the last person to comment or review the patch in question. [...] > On the other hand, what I've done is similar to what you did - comment > on someone else's patch. It seems relevant to me that the original > poster (Tony Meyer) hasn't felt strongly enough to respond on his own > behalf to comments on his patch. No disrespect to Tony, but I'd argue > that the implication is that the patch should be rejected because even > the submitter doesn't care enough to respond to comments! There is a considerable difference between "doesn't care enough", and "has not had time to be able to" (although in this specific case "doesn't care enough" is correct). I have submitted a very small (3?) number of patches, however, I suspect that my position is similar to others, so I offer an explanation in the hope that it adds value to this thread. I don't submit patches because I need the problem fixed in the Python distribution. I make the change locally, and either I am distributing a frozen application (almost always the case), which includes my local fix, or a workaround is made in the application source which means that the main Python distribution fix is unneeded (e.g. this is what I did with SpamBayes). The particular patch mentioned is one that uses code (more-or-less) from SpamBayes. SpamBayes has the code - it doesn't matter whether it's in the Python distribution or not. At the time I wrote the patch, there were (again) discussions on python-dev about what should be done to ConfigParser. I had some time free in those days, and, since I had some code that did more-or-less what Guido indicated was the best option, I contributed it (writing unittests, documentation, and commenting in the related tickets). To a certain extent, I considered that my work done. This was something I contributed because many people continually requested it, not something I felt a personal need to be added to the distribution (as above, that's not a need that I generally feel). I (much) later got email with patches, and then later email from Mark Hammond about the patch (IIRC Mark was looking at it and was thinking of fixing it up; I think I forwarded the email I got to him. OTOH, maybe he also sent me fixes - I'm too busy to trawl through email archives to figure it out). At the time, I hoped to fix up the errors and submit a revised patch, but my son was born a few weeks later and I never found the time. If the patch had been reviewed more quickly, then I probably would have found time to correct it - however, everyone else is busy to (if I felt strongly about it, then I would have reviewed 5 other patches, as I have in the past, and 'forced' more quick review, but I did not). For me, submitting a patch is mostly altruistic - if I do that then other people don't also have do the work I did, and hopefully other people do that as well, saving me work. It's not something I require, at all. This isn't something that is easy to make time for. ISTM that there is value in submitting a patch (including tests and documentation, and making appropriate comment in related patches), even if that is all that is done (i.e. no follow-up). If the value isn't there without that follow-up 'caring', then that is something that should be addressed to 'encourage developers'. Contributions don't only come from people hoping to be 'core' developers some day. Uncaringly-(with-apologies-to-uncle-timmy), Tony From dustin at v.igoro.us Sun Mar 18 04:31:44 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Sat, 17 Mar 2007 22:31:44 -0500 Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: <45FC7EA3.4070808@canterbury.ac.nz> References: <45FC7EA3.4070808@canterbury.ac.nz> Message-ID: <20070318033144.GC8010@v.igoro.us> On Sun, Mar 18, 2007 at 11:49:55AM +1200, Greg Ewing wrote: > I've just discovered the hard way that NamedTemporaryFile > automatically deletes the file when you close it. That > doesn't seem very useful to me, since surely the reason > you're using NamedTemporaryFile instead of TemporaryFile > is that you want to do something else with it afterwards? > What's the rationale for this behaviour? For both TemporaryFile and NamedTemporaryFile, the rationale is that "temporary" extends until the object is garbage collected. TemporaryFile is deleted immediately (to prevent other applications from modifying your temporary file). NamedTemporaryFile is inteded for use when you need access to the file by filename during the lifetime of the file. In either case, when the object goes, the file should too. Dustin From dickinsm at gmail.com Sun Mar 18 05:03:13 2007 From: dickinsm at gmail.com (Mark Dickinson) Date: Sun, 18 Mar 2007 00:03:13 -0400 Subject: [Python-Dev] Rewrite of cmath module? Message-ID: <5c6f2a5d0703172103j44bd91d1j1d9d57e167797c75@mail.gmail.com> The current cmath module has some significant numerical problems, particularly in the inverse trig and inverse hyperbolic trig functions. I've listed a few of these below, but for now I'll just note the following upsetting result: >>> asin(-1e8) Traceback (most recent call last): File "", line 1, in OverflowError: math range error (The true value is somewhere around -pi/2 +/- 19.1138j, which is hardly deserving of an overflow error.) I'm wondering whether anyone would be interested in a rewrite of the cmath module. I have a drop-in replacement written in pure Python, based largely on the formulas given by Kahan in his `Much Ado about Nothing's Sign Bit' article, which I believe eliminates the vast majority of the current numerical problems. Besides the numerical fixes, the only significant difference from the current behaviour is that the branch cuts for asinh have been moved. (The docs suggest that the current branch cuts for asinh should be regarded as buggy). I'm offering to translate this into C and add appropriate documentation and test cases, but I have a couple of questions before I start. (1) Is there actually any interest in fixing this module? I guess that the cmath module can't be used that much, otherwise these problems would have surfaced before. (2) (Disregard if the answer to question 1 is `No'.) What should be done about branch cuts? The value of a function on a branch cut is going to depend on signs of zeros, so it'll be pretty much impossible to make any guarantees about the behaviour. Even so, there are two approaches that seem feasible: (a) One can ensure that on a platform that happens to have IEEE-754 compliant hardware and a signed-zero-friendly math library, the complex functions `do the right thing': i.e., are continuous on both sides of each of the branch cuts, using the sign of zero on a branch cut to determine which side of the branch cut the user is interested in. Doing this portably in C89 involves some trickery, but it seems possible. For other platforms behaviour on the branch cuts would be undefined. Alternatively: (b) Force collapsing of all signed zeros to `positive' zero, so that the functions are continuous only on one side of the branch cuts, and the continuity is exactly as specified in the current documentation (though probably including the suggested changes for atan and atanh.) This is mostly what happens with the current implementation, though it's not clear to me whether that's by design or by accident. (c) Some other approach that I haven't considered. Which of these approaches is preferable? The Python module that I have currently does (a). (3) Is it worth trying to be consistent in exceptional cases? The current math module documentation notes that math.log(0) might produce -inf on one platform and raise ValueError on another, so being consistent about whether a non-representable result gives a NaN, an infinity or a Python exception seems as though it would be tricky. I should also note that I've made no effort to do the right thing when the argument to any of the functions contains a NaN or an infinity in either the real or imaginary part. Here are a few examples of the current cmath problems, on my machine (OS X 10.4.8/PowerPC). (*) As already mentioned in the docs, the current branch cuts for asinh are somewhat unconventional, and should probably be fixed. I believe that this is a fairly trivial fix. (*) asinh gives inaccurate answers in the portion of the complex plane between the current branch cuts, when the real part is large and negative. >>> asinh(-1e7+0.9j) (-16.811242831518271+8.9158318101199366e-08j) >>> asinh(-1e8+0.9j) (-19.113827924512311+0j) (The imaginary parts here should be approximately 8.999999999999e-8 and 8.999999999999e-9 respectively.) (*) asinh gives inaccurate results for small x: >>> asinh(1e-12) (1.000088900582091e-12+0j) >>> asinh(1e-20) (4.4408920985006257e-16+0j) (asinh(x) ~ x - 1/6 x^3 + O(x^5) for small x, so the correct values are 1e-12 and 1e-20 to within machine precision.) (*) acos behaves badly for large reals: >>> acos(1e7) 16.805431370234086j # true value ~ 16.811242831518262597j >>> acos(1e8) Traceback (most recent call last): File "", line 1, in OverflowError: math range error (*) tan and tanh overflow unnecessarily, even though tan(z) and tanh(z) should be representable for all representable complex arguments. >>> tan(1000j) (nannanj) (Actual value: 1j, to within machine precision.) (*) log and sqrt overflow unnecessarily for arguments close to the limits of double precision: >>> z = 1.4e308+1.4e308j >>> sqrt(z) Traceback (most recent call last): File "", line 1, in OverflowError: math range error >>> log(z) (inf+0.78539816339744828j) (*) sqrt gives inaccurate answers for very small arguments (where the real and imaginary parts are denormalized numbers): >>> z = .5e-323 + .5e-323j >>> sqrt(z) (2.2227587494850775e-162+0j) (Correct value for sqrt(z) is close to 2.442109e-162 + 1.011554e-162j.) (*) exp, cosh and sinh overflow unnecessarily for some values. For example: >>> exp(710+pi/4j) Traceback (most recent call last): File "", line 1, in OverflowError: math range error The correct value is around 1.579673e308 + 1.579673e308j, which should be representable in IEEE double precision. This last problem is almost not worth fixing: these functions are going to overflow eventually anyway, and whether that happens when the real part of the argument is 709.0 or 710.0 isn't going to make a huge difference to anyone's life. In contrast, functions like acos, log and sqrt should have representable output for *any* finite representable complex number (with the obvious exception of 0 for log), so it's a bit embarrassing if they fail for some inputs. One more thing: since this is my first post to python-dev I should probably introduce myself. I'm a mathematician, working mostly in number theory. I learnt programming and numerical analysis the hard way, coding service-life prediction algorithms for rocket motors in Fortran 77 during several long summers. I've been using Python for more than ten years, mainly for teaching but also occasionally for research purposes. Mark Dickinson From aahz at pythoncraft.com Sun Mar 18 05:09:49 2007 From: aahz at pythoncraft.com (Aahz) Date: Sat, 17 Mar 2007 21:09:49 -0700 Subject: [Python-Dev] Rewrite of cmath module? In-Reply-To: <5c6f2a5d0703172103j44bd91d1j1d9d57e167797c75@mail.gmail.com> References: <5c6f2a5d0703172103j44bd91d1j1d9d57e167797c75@mail.gmail.com> Message-ID: <20070318040949.GA25876@panix.com> On Sun, Mar 18, 2007, Mark Dickinson wrote: > > I'm wondering whether anyone would be interested in a rewrite of the > cmath module. In theory, yes, but my observations of past discussions have been that they tend to founder on the issues of cross-platform support and long-term code maintenance. > I have a drop-in replacement written in pure Python, > based largely on the formulas given by Kahan in his `Much Ado about > Nothing's Sign Bit' article, which I believe eliminates the vast > majority of the current numerical problems. If nothing else, if you are willing to support this code, *PLEASE* upload it to the Cheese Shop. Public interest in such code makes it easier to sell back to the core. BTW, I noticed that your post was held for moderation, probably because you aren't subscribed to python-dev. If you want to participate in this discussion, subscribing makes that much easier. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Typing is cheap. Thinking is expensive." --Roy Smith From g.brandl at gmx.net Sun Mar 18 09:13:15 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 18 Mar 2007 09:13:15 +0100 Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: <45FC7EA3.4070808@canterbury.ac.nz> References: <45FC7EA3.4070808@canterbury.ac.nz> Message-ID: Greg Ewing schrieb: > I've just discovered the hard way that NamedTemporaryFile > automatically deletes the file when you close it. That > doesn't seem very useful to me, since surely the reason > you're using NamedTemporaryFile instead of TemporaryFile > is that you want to do something else with it afterwards? > What's the rationale for this behaviour? It now (2.6) has a keyword argument for not closing the file. Georg From djm at mindrot.org Sun Mar 18 09:48:59 2007 From: djm at mindrot.org (Damien Miller) Date: Sun, 18 Mar 2007 19:48:59 +1100 (EST) Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: <45FC7EA3.4070808@canterbury.ac.nz> References: <45FC7EA3.4070808@canterbury.ac.nz> Message-ID: On Sun, 18 Mar 2007, Greg Ewing wrote: > I've just discovered the hard way that NamedTemporaryFile > automatically deletes the file when you close it. That > doesn't seem very useful to me, since surely the reason > you're using NamedTemporaryFile instead of TemporaryFile > is that you want to do something else with it afterwards? > What's the rationale for this behaviour? That annoyed me too, so I submitted a patch[1] that was recently committed. -d [1] https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1537850&group_id=5470 From martin at v.loewis.de Sun Mar 18 12:11:12 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 18 Mar 2007 12:11:12 +0100 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <45ED0353.2090201@scottdial.com> <45ED07EE.4060402@v.loewis.de> <79990c6b0703060320t23d4487rcc97ec69c669a8ee@mail.gmail.com> <45ED57CA.7090101@v.loewis.de> <45ED5C81.6070702@scottdial.com> <79990c6b0703070542w1479d8e4rdf30b1cc9a5f2a78@mail.gmail.com> Message-ID: <45FD1E50.7080209@v.loewis.de> Tony Meyer schrieb: > ISTM that there is value in submitting a patch (including tests and > documentation, and making appropriate comment in related patches), even > if that is all that is done (i.e. no follow-up). That's certainly, absolutely, true. It's also true that there are very many kinds of different contributions, e.g. a bug report is a contribution, too. However, some contributions bring the project forward more than others, e.g. a patch often brings it further along than a bug report, and a patch contributor willing to rework the patch brings it further along than a contributor that "just" drops the patch into the tracker (after developing it first, of course). Given that we are not able to process all contributions, we clearly have to make choices what contributions to look at, and what contributions to ignore. This, unfortunately, means that some patches get rejected just because neither the submitter nor any of the reviewers is willing to complete it, if it is incomplete at the time of submission. Regards, Martin From guido at python.org Sun Mar 18 20:53:49 2007 From: guido at python.org (Guido van Rossum) Date: Sun, 18 Mar 2007 12:53:49 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <20070317143613.GA23013@panix.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> <45F8ECFE.2030105@v.loewis.de> <20070317143613.GA23013@panix.com> Message-ID: Now that this thread has celebrated its 100th message, I feel compelled to say something. My first observation is that this is a tempest in a teacup. My second observation is that there seems to have been a lack of people skills all around. That is perhaps to expect in a community of geeks, but given the length of the previous thread on this topic (before Martin checked it in) I think all the participants might have done wiser by taking each others' feelings into account rather than attempting to relentlessly arguing the technical point at hand. My third observation is tha a policy that would have disallowed or allowed (depending on your POV) this particular change is not an option. A policy isn't going to solve all disagreements, there will always be debate possible about the interpretations of the rules. What's needed is a way to handle such debate in a way that produces an outcome without wearing everyone out. It's important that the participants in the debate respect each other -- before, during and after the debate. If you want, I can make a decision. But I will only do that if I hear from both sides of the debate that they are willing to accept my choice even if it favors the other party. Can I hear agreement to that? In particular; Phillip and Glyph, if I decide that Martin's change is OK for 2.6, will you accept it and stop debating it and get on with your lives? And Martin, if I decide that the change should be rolled back, will you be okay with that? This is an experiment for me as well; if you all would prefer me to stay out of it, I will. I haven't made up my mind yet about the technical issue, but I'm not interested in hearing the arguments repeated; I've heard them all and just need to mull it over. Let me know. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Sun Mar 18 21:16:36 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 18 Mar 2007 21:16:36 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> <45F8ECFE.2030105@v.loewis.de> <20070317143613.GA23013@panix.com> Message-ID: <45FD9E24.9050201@v.loewis.de> Guido van Rossum schrieb: > And Martin, if I decide that the change should be > rolled back, will you be okay with that? Certainly. I would wish somebody contributed a documentation patch pointing out that specific detail in the documentation (and in the process making the documentation match the implementation in the first place), but if nobody comes along, I would do that myself. > This is an experiment for me as well; if you all would prefer me to > stay out of it, I will. I would hope that we can agree to something that has been proposed as an alternative in http://python.org/sf/1681842 (suggesting a warning message and keyword arguments), or the two-step change that Thomas proposed, so you may want to watch this going on for a little while longer. However, I don't feel qualified anymore to trust my intuition on what changes are acceptable and which aren't, so it would be for some other committer (perhaps Thomas or Phillip) to actually implement such a change. HTH, Martin From tdelaney at avaya.com Sun Mar 18 22:59:47 2007 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Mon, 19 Mar 2007 08:59:47 +1100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1ED24@au3010avexu1.global.avaya.com> Guido van Rossum wrote: > If you want, I can make a decision. But I will only do that if I hear > from both sides of the debate that they are willing to accept my > choice even if it favors the other party. Can I hear agreement to > that? >From me - definitely. I put my position forward (anti this change, partly because it only addressed one of the special cases), and then stayed out of it. Happy mulling! Tim Delaney From tdelaney at avaya.com Sun Mar 18 23:13:56 2007 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Mon, 19 Mar 2007 09:13:56 +1100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1ED26@au3010avexu1.global.avaya.com> Delaney, Timothy (Tim) wrote: >> From me - definitely. Damned Outlook, reformatting sent emails! That statement was obviously from me ... Tim Delaney From tim.peters at gmail.com Sun Mar 18 23:38:02 2007 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 18 Mar 2007 18:38:02 -0400 Subject: [Python-Dev] Rewrite of cmath module? In-Reply-To: <5c6f2a5d0703172103j44bd91d1j1d9d57e167797c75@mail.gmail.com> References: <5c6f2a5d0703172103j44bd91d1j1d9d57e167797c75@mail.gmail.com> Message-ID: <1f7befae0703181538k206e030bh7a773b733ca3708e@mail.gmail.com> [Mark Dickinson] > The current cmath module has some significant numerical problems, > particularly in the inverse trig and inverse hyperbolic trig > functions. IIRC, cmath was essentially whipped up over a weekend by a scientist, transcribing formulas from a math reference. Such an approach is bound to suffer "significant numerical problems", but has the advantage of taking little time to implement ;-) A few of the worst have been improved (e.g., see c_quot(), which contains the original code in a comment), but no systematic effort has been made. > ... > I'm wondering whether anyone would be interested in a rewrite of the > cmath module. I have a drop-in replacement written in pure Python, > based largely on the formulas given by Kahan in his `Much Ado about > Nothing's Sign Bit' article, which I believe eliminates the vast > majority of the current numerical problems. I believe that :-) > Besides the numerical fixes, the only significant difference from the > current behaviour is that the branch cuts for asinh have been moved. > (The docs suggest that the current branch cuts for asinh should be > regarded as buggy). I'm offering to translate this into C and add > appropriate documentation and test cases, but I have a couple of > questions before I start. Make the Python implementation available first and solicit feedback from the Numeric community? > (1) Is there actually any interest in fixing this module? I guess > that the cmath module can't be used that much, otherwise these > problems would have surfaced before. That's the rub: most people don't use cmath, while most who do have no idea what these functions should return. If there's a gross problem, most wouldn't even notice. The few places in cmath that have been improved were due to users who did notice. Of course all "should be" fixed. > (2) (Disregard if the answer to question 1 is `No'.) What should be > done about branch cuts? The value of a function on a branch cut is > going to depend on signs of zeros, so it'll be pretty much impossible > to make any guarantees about the behaviour. Even so, there are two > approaches that seem feasible: ... Follow the C99 standard whenever possible. C99 added a complex type to C, and added complex-valued math functions too. Alas, C99 hasn't been widely adopted yet, but Python should be compatible with the platform C implementation to the extent possible. Of course the easiest way to do that is to /use/ the platform C implementation when possible. ... > (3) Is it worth trying to be consistent in exceptional cases? The > current math module documentation notes that math.log(0) might produce > -inf on one platform and raise ValueError on another, so being consistent > about whether a non-representable result gives a NaN, an infinity or a > Python exception seems as though it would be tricky. I should also note that > I've made no effort to do the right thing when the argument to any of the > functions contains a NaN or an infinity in either the real or imaginary part. Welcome to the club ;-) To the extent that your code relies on real-valued libm functions, you inherit uncertainties from the platform C's implementation of those too. There's a long debate about whether that's a feature (Python math functions act the same way as the platform C's, and likely the platform Fortran's too) or a bug (Python acts differently on different platforms). "Feature" is the traditional answer to that. ... > [list of specific bad behaviors] ... > One more thing: since this is my first post to python-dev I should > probably introduce myself. I'm a mathematician, working mostly in > number theory. I learnt programming and numerical analysis the hard > way, coding service-life prediction algorithms for rocket motors in > Fortran 77 during several long summers. I've been using Python for > more than ten years, mainly for teaching but also occasionally for > research purposes. Of course a strong background in math is very helpful for this. If you also spent several long summers cleaning sewers with your bare hands, your qualifications for working on libm functions are beyond reproach ;-) From greg.ewing at canterbury.ac.nz Mon Mar 19 01:20:59 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 19 Mar 2007 12:20:59 +1200 Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: References: <45FC7EA3.4070808@canterbury.ac.nz> Message-ID: <45FDD76B.7080800@canterbury.ac.nz> Damien Miller wrote: > That annoyed me too, so I submitted a patch[1] that was recently > committed. That looks good. Seems to me it should really be the default behaviour, but I suppose that would break code that was relying on the automatic deletion. -- Greg From glyph at divmod.com Mon Mar 19 05:37:37 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Mon, 19 Mar 2007 04:37:37 -0000 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> <45F8ECFE.2030105@v.loewis.de> <20070317143613.GA23013@panix.com> Message-ID: <20070319043737.7769.1086258906.divmod.xquotient.1356@joule.divmod.com> On 18 Mar, 07:53 pm, guido at python.org wrote: >My second observation is that there seems to have been a lack of >people skills all around. That is perhaps to expect in a community of >geeks, but given the length of the previous thread on this topic >(before Martin checked it in) I think all the participants might have >done wiser by taking each others' feelings into account rather than >attempting to relentlessly arguing the technical point at hand. Let me take the opportunity to make this clear, then. I have the utmost respect for Martin and his contributions for Python. I have been following commits for quite a while and I know that Martin, in particular, is often the one who deals with the crap work of reviewing huge piles of orphaned patches and fixing piles of minor issues, and he is therefore responsible for much of the general upward trend in Python's overall quality in the last few releases. I appreciate that very much. >My first observation is that this is a tempest in a teacup. On the one hand I agree. This particular change is trivial, most likely doesn't affect me, and I accept that, in practice, it probably won't even break too many programs (and may even fix a few). On the other, I think it is important to quell the tempest before it exits the teacup. Previously in this discussion, both I and PJE have repeatedly declared that this _particular_ change is not really what's at issue here, merely the pattern of reasoning that comes to consider this change acceptable. At some point a large number of small breakages are actually worse than one big one, and it's hard to point to the exact place where that happens. On the gripping hand, I am almost glad that it was a relatively minor change that triggered this avalanche of posts. Even with such a small change, the change itself threatens to obscure a larger issue, and if the change itself were any bigger, it would eclipse the other discussion completely. >My third observation is tha a policy that would have disallowed or >allowed (depending on your POV) this particular change is not an >option. A policy isn't going to solve all disagreements, there will >always be debate possible about the interpretations of the rules. >What's needed is a way to handle such debate in a way that produces an >outcome without wearing everyone out. The allow vs. disallow issue is not *really* what the policy should be addressing. A major problem with this thread is the differing definitions that some people have, beginning with "extension", but I can't see that a policy will fix *that*. Words like "bug", "fix", "compatible", and so on, all have "obvious" general meanings but much more nuanced and specific meanings in particular contexts. A policy should outline specifics of what, for example, is to be considered an "incompatible" change, and what must be done in that case. A policy could not outright forbid changes of a certain type, since that is pretty much asking that it be broken any time a sufficiently important change is requested and the core team likes it. Maybe "policy" isn't even the right word here, since the part of it that would facilitate discussions like this one would be more "lexicon" than "policy". >It's important that the participants in the debate respect each other >-- before, during and after the debate. Agreed. Any "lack of people skills" notwithstanding, I hope I haven't said anything that implied (or stated, of course) that I did not *respect* the other participants of the discussion. If I have, I retract it. Strong disagreement is different than disrespect. >If you want, I can make a decision. But I will only do that if I hear >from both sides of the debate that they are willing to accept my >choice even if it favors the other party. Can I hear agreement to >that? In particular; Phillip and Glyph, if I decide that Martin's >change is OK for 2.6, will you accept it and stop debating it and get >on with your lives? And Martin, if I decide that the change should be >rolled back, will you be okay with that? I will certainly accept the decision. I don't *like* generating trouble on this mailing list, believe me. Once a BDFL pronouncement is made, further discussion is just generating trouble. That isn't the same as *agreeing* with the decision, of course :-). The important thing for me is not reaching a decision on this particular issue (or even a particular decision on this particular issue). It is that we achieve some kind of consensus around how backward compatibility is supposed to work "in the large" rather than in a particular instance. For those of you who don't think this issue is important in and of itself, consider the secondary consequence of this ruckus happening every time someone commits a potentially-incompatible change. I would not mind if, for example, this patch were "grandfathered in" to the lack of any clear backwards compatibility policy, as long as similar future changes were subject to more up-front scrutiny. Although I don't really think the API should be changed, I don't regard that as a point of contention. As I've said before, I have seldom used splitext, and I am not likely to start after this discussion ;). The optional-argument / emit-a-warning changes are both amenable to me. As a first approximation of such a policy (I am seriously working on that pre-PEP, it'll be ready any day now...) "one release with a deprecation warning is always required for any deviation from previously-documented behavior, either in the docstrings or library documentation". If there are problems with the warnings system that make this strategy impossible, I would be happy to help fix them in any way I can. It has occurred to me in the last few days that this the two sides in this debate may *actually* be "Warnings suck" and "Warnings suck, but they're all we have". >This is an experiment for me as well; if you all would prefer me to >stay out of it, I will. I haven't made up my mind yet about the >technical issue, but I'm not interested in hearing the arguments >repeated; I've heard them all and just need to mull it over. Let me >know. I think we've all had enough of the discussion for now, so I'd personally appreciate it if you'd offer a pronouncement and we can all just move on. I can continue working on that pre-PEP regardless of this resolution, and that will need a separate pronouncement anyway. At least, I don't feel like I have anything useful to add to the discussion that hasn't already been said, by me or by someone else. Thank you for stepping in. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070319/e8ae87ff/attachment.htm From p.f.moore at gmail.com Mon Mar 19 12:05:07 2007 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 Mar 2007 11:05:07 +0000 Subject: [Python-Dev] Encouraging developers In-Reply-To: References: <20070305184614.GA19051@localhost.localdomain> <200703051930.20868.phil@riverbankcomputing.co.uk> <45ECFF37.6090901@v.loewis.de> <45ED0353.2090201@scottdial.com> <45ED07EE.4060402@v.loewis.de> <79990c6b0703060320t23d4487rcc97ec69c669a8ee@mail.gmail.com> <45ED57CA.7090101@v.loewis.de> <45ED5C81.6070702@scottdial.com> <79990c6b0703070542w1479d8e4rdf30b1cc9a5f2a78@mail.gmail.com> Message-ID: <79990c6b0703190405s65cb371fk52ffc7d60ffe7075@mail.gmail.com> On 18/03/07, Tony Meyer wrote: > > on someone else's patch. It seems relevant to me that the original > > poster (Tony Meyer) hasn't felt strongly enough to respond on his own > > behalf to comments on his patch. No disrespect to Tony, but I'd argue > > that the implication is that the patch should be rejected because even > > the submitter doesn't care enough to respond to comments! > > There is a considerable difference between "doesn't care enough", and > "has not had time to be able to" (although in this specific case > "doesn't care enough" is correct). On rereading my comments, I apologise - you're absolutely right that I didn't have enough evidence to judge that you "don't care enough" about the patch. Thanks for clarifying - I entirely agree with you that just being willing to put together any sort of bug report or patch is valuable, and people should not be discouraged from doing so. (It's a sad but probably inevitable fact that many such contributors - I include myself here! - have expectations of speed of response which simply aren't possible for a volunteer project, and as a result get frustrated with "the process", but that's a separate issue) Paul. From nmm1 at cus.cam.ac.uk Mon Mar 19 15:37:49 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Mon, 19 Mar 2007 14:37:49 +0000 Subject: [Python-Dev] Status of thread cancellation Message-ID: Grrk. I have done this myself, and been involved in one of the VERY few commercial projects that attempted to do it properly (IBM CEL, the other recent one being VMS). I am afraid that there are a lot of misapprehensions here. Several people have said things like: > The thing to model this on, I think, would be the > BSD sigmask mechanism, which lets you selectively > block certain signals to create a critical section > of code. A context manager could be used to make > its use easier and less error-prone (i.e. harder > to block async exceptions and then forget to unblock > them). No, no, no! That is an TRULY horrible! It works fairly well for things like device drivers, which are both structurally simple and with no higher level recovery mechanism, so that a failure turning into a hard hang is not catastrophic. But it is precisely what you DON'T want for complex applications, especially when a thread may need to call an external service 'non-interruptibly'. Think of updating a complex object in a multi-file database, for example. Interrupting half-way through leaves the database in a mess, but blocking interrupts while (possibly remote) file updates complete is asking for a hang. You also see it in horrible GUI (including raw mode text) programs that won't accept interrupts until you have completed the action they think you have started. One of the major advantages of networked systems is that you can usually log in remotely and kill -9 the damn process! The way that I, IBM and DEC approached it was by the classic callback mechanism, with a carefully designed way of promoting unhandled exceptions/interrupts. For example, the following is roughly what I did (somewhat extended, as I didn't do all of this for all exceptions): An event set a defined flag, which could be tested (and cleared) by the thread. If a second, similar event arrived (or it was not handled after a suitable time), the event was escalated. If so, a handler was called that HAD to return (again within a specific time). If a second, similar event arrived or it didn't return by a suitable time, the event was escalated. If so, another handler was called that COULDN'T return. If another event arrived, it returned, or it failed to close down the thread, the event was escalated. If so, the thread's built-in environment was closed down without giving the thread a chance to intervene. If that failed, the event was escalated. If so, the thread was frozen and process termination started. If clean termination failed, the event was escalated. If so, the run-time system produced a dump and killed itself. You can implement a BSD-style ignore by having an initial handler that just clears the flag and returns, but a third interrupt before it does so will force close-down. There was also a facility to escalate an exception at the point of generation, which could be useful for forcible closedown. There are a zillion variations of the above, but all mainframe experience is that callbacks are the only sane way to approach the problem IN APPLICATIONS. In kernel code, that is not so, which is why so many of the computer scientists design BSD-style handling (i.e. they think of kernel programming rather than very complex application programming). > Unconditionally killing a whole process is no big > problem because all the resources it's using get > cleaned up by the OS, and the effect on other > processes is minimal and well-defined (pipes and > sockets get EOF, etc.). But killing a thread can > leave the rest of the program in an awkward state. I wish that were so :-( Sockets, terminals etc. are stateful devices, and killing a process can leave them in a very unclean state. It is one of the most common causes of unkillable processes (the process can't go until its files do, and the socket is jammed). Many people can witness the horrible effects of ptys being left in 'echo off' or worse states, the X focus being left in a stuck override redirect window and so on. But you also have the multi-file database problem, which also applies to shared memory segments. Even if the process dies cleanly, it may be part of an application whose state is global across many processes. One common example is adding or deleting a user, where an unclean kill can leave the system in a very weird state. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From guido at python.org Mon Mar 19 15:46:03 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Mar 2007 07:46:03 -0700 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <20070319043737.7769.1086258906.divmod.xquotient.1356@joule.divmod.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> <45F8ECFE.2030105@v.loewis.de> <20070317143613.GA23013@panix.com> <20070319043737.7769.1086258906.divmod.xquotient.1356@joule.divmod.com> Message-ID: On 3/18/07, glyph at divmod.com wrote: > Let me take the opportunity to make this clear, then. I have the utmost > respect for Martin and his contributions for Python. I have been following > commits for quite a while and I know that Martin, in particular, is often > the one who deals with the crap work of reviewing huge piles of orphaned > patches and fixing piles of minor issues, and he is therefore responsible > for much of the general upward trend in Python's overall quality in the last > few releases. I appreciate that very much. Thanks. > >My first observation is that this is a tempest in a teacup. > > On the one hand I agree. This particular change is trivial, most likely > doesn't affect me, and I accept that, in practice, it probably won't even > break too many programs (and may even fix a few). Then why was there such an upheaval when it was submittedd? > On the other, I think it is important to quell the tempest before it exits > the teacup. Previously in this discussion, both I and PJE have repeatedly > declared that this _particular_ change is not really what's at issue here, > merely the pattern of reasoning that comes to consider this change > acceptable. At some point a large number of small breakages are actually > worse than one big one, and it's hard to point to the exact place where that > happens. I think you're missing my point here. At issue is a judgement call, not an adjustment of the rules. Everybody knows that we fix bugs mercilessly but are very careful with deprecating features. At issue is the judgement about whether a particular changes is a bug or a feature. I don't accept the position that since there are unit tests and docs for it, it must be a feature; those could well have been produced without much thinking and after the fact (I *know* they were produced after the fact since splitext() long predates our first unit test). So if there's any new rule required, it's not a rule for defining more clearly the definition of bug vs. feature, but a rule for what to do if you disagree with a change (whether committed or proposed). But that falls in the realm of human behavior, and I very much doubt we can write a PEP for that. Even if we could, I really don't think I'd like the result; I'm not into having a court of appeals or any such formal solution. To be concrete, I think that if Phillip had written a different kind of post instead of the one where he requests the reversal of the submit (only parenthetically mentioning Martin) then perhaps Martin wouldn't have felt the need to dig in and defend his position, and the issue might have been resolved quicker and at less emotional expense. I see small discussions on python-checkins all the time where someone comments on someone else's checkin, and the tone of the comment makes all the difference. > On the gripping hand, I am almost glad that it was a relatively minor change > that triggered this avalanche of posts. Even with such a small change, the > change itself threatens to obscure a larger issue, and if the change itself > were any bigger, it would eclipse the other discussion completely. One recommendation I have for discussions like this (thanks to Stephen Turnbull in private mail) is to attempt to separate in your mind (and in everyone's mind) the distinction between the change at hand and the policy discussion. Muddling these two together makes for a poor discussion of the feature and an even poorer discussion of policy change. > >My third observation is tha a policy that would have disallowed or > >allowed (depending on your POV) this particular change is not an > >option. A policy isn't going to solve all disagreements, there will > >always be debate possible about the interpretations of the rules. > >What's needed is a way to handle such debate in a way that produces an > >outcome without wearing everyone out. > > The allow vs. disallow issue is not *really* what the policy should be > addressing. A major problem with this thread is the differing definitions > that some people have, beginning with "extension", but I can't see that a > policy will fix *that*. Words like "bug", "fix", "compatible", and so on, > all have "obvious" general meanings but much more nuanced and specific > meanings in particular contexts. A policy should outline specifics of what, > for example, is to be considered an "incompatible" change, and what must be > done in that case. A policy could not outright forbid changes of a certain > type, since that is pretty much asking that it be broken any time a > sufficiently important change is requested and the core team likes it. IMO all the policy we need is PEP 5. It wisely defers to common sense regarding the implementation, and that's where I want to re-focus the discussion. > Maybe "policy" isn't even the right word here, since the part of it that > would facilitate discussions like this one would be more "lexicon" than > "policy". The crux is in trying to define "major". That's vague, and intentionally so; I think it would be a mistake to try to define it more properly for all time. The Python community changes over time (remember the features I added in 1.5.2? Or even as late as 2.2.3? :-) and I would much rather spend time understanding what's most important for the rank and file users at this moment than debate the fine points of a word's definition amung ueber geeks. > >It's important that the participants in the debate respect each other > >-- before, during and after the debate. > > Agreed. > > Any "lack of people skills" notwithstanding, I hope I haven't said anything > that implied (or stated, of course) that I did not *respect* the other > participants of the discussion. If I have, I retract it. Strong > disagreement is different than disrespect. > > >If you want, I can make a decision. But I will only do that if I hear > >from both sides of the debate that they are willing to accept my > >choice even if it favors the other party. Can I hear agreement to > >that? In particular; Phillip and Glyph, if I decide that Martin's > >change is OK for 2.6, will you accept it and stop debating it and get > >on with your lives? And Martin, if I decide that the change should be > >rolled back, will you be okay with that? > > I will certainly accept the decision. I don't *like* generating trouble on > this mailing list, believe me. Once a BDFL pronouncement is made, further > discussion is just generating trouble. > > That isn't the same as *agreeing* with the decision, of course :-). And that's my point. There were plenty of people who didn't agree with the @decorator syntax at the time (or with print >> file, or any number of examples); but they didn't bear a grudge. One or two even said specifically "I disagree with the feature but I will continue to use Python". > The important thing for me is not reaching a decision on this particular > issue (or even a particular decision on this particular issue). It is that > we achieve some kind of consensus around how backward compatibility is > supposed to work "in the large" rather than in a particular instance. For > those of you who don't think this issue is important in and of itself, > consider the secondary consequence of this ruckus happening every time > someone commits a potentially-incompatible change. > > I would not mind if, for example, this patch were "grandfathered in" to the > lack of any clear backwards compatibility policy, as long as similar future > changes were subject to more up-front scrutiny. And here I disagree. I think our backward compatibility *policy* is as good as it gets; how we address the importance of a judgement call in a particular case is not. > Although I don't really think the API should be changed, I don't regard that > as a point of contention. As I've said before, I have seldom used splitext, > and I am not likely to start after this discussion ;). The > optional-argument / emit-a-warning changes are both amenable to me. I'm trying to stay out of the feature discussion, but I would like to point out that a policy that, in the sake of some strict definition of backwards compatibility, forces us to introduce new APIs (or new optional parameters to existing ones, which is really the same thing) at a high rate is also doomed to have an overall detrimental effect on the language -- who know, perhaps more so than the occasional incompatible change. > As a first approximation of such a policy (I am seriously working on that > pre-PEP, it'll be ready any day now...) "one release with a deprecation > warning is always required for any deviation from previously-documented > behavior, either in the docstrings or library documentation". We already have that policy in PEP 5. Except for the anal definition. Which is wrong too -- if a feature happens to be undocumented that still doesn't make it fair game for a sudden incompatible change. > If there are > problems with the warnings system that make this strategy impossible, I > would be happy to help fix them in any way I can. It has occurred to me in > the last few days that this the two sides in this debate may *actually* be > "Warnings suck" and "Warnings suck, but they're all we have". > > >This is an experiment for me as well; if you all would prefer me to > >stay out of it, I will. I haven't made up my mind yet about the > >technical issue, but I'm not interested in hearing the arguments > >repeated; I've heard them all and just need to mull it over. Let me > >know. > > I think we've all had enough of the discussion for now, so I'd personally > appreciate it if you'd offer a pronouncement and we can all just move on. I > can continue working on that pre-PEP regardless of this resolution, and that > will need a separate pronouncement anyway. As you see I'm trying to discourage you from working on such a document; but I won't stop you and if there's general demand for it and agreement I'll gladly review it when it's ready. (It's a bit annoying to have to read long posts alluding to a document under development without being able to know what will be in it.) But I won't make up my mind yet; I'm still waiting hear from Phillip. > At least, I don't feel like I have anything useful to add to the discussion > that hasn't already been said, by me or by someone else. > > Thank you for stepping in. Well, I just hope my contributions *are* useful. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jon+python-dev at unequivocal.co.uk Mon Mar 19 16:28:33 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Mon, 19 Mar 2007 15:28:33 +0000 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: References: Message-ID: <20070319152833.GE31560@snowy.squish.net> Nick Maclaren wrote: > Sockets, terminals etc. are stateful devices, and killing a process > can leave them in a very unclean state. It is one of the most > common causes of unkillable processes (the process can't go until > its files do, and the socket is jammed). Can you elaborate on this? You can get zombie entries in the process table if nobody's called 'wait()' on them, and you can (extremely rarely) get unkillable process in 'disk-wait' state (usually due to hardware failure or a kernel bug, I suspect), but I've never heard of a process on a Unix-like system being unkillable due to something to do with sockets (or any other kind of file descriptor for that matter). How could a socket be 'jammed'? What does that even mean? From pje at telecommunity.com Mon Mar 19 17:28:44 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 19 Mar 2007 11:28:44 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <20070317143613.GA23013@panix.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> <45F8ECFE.2030105@v.loewis.de> <20070317143613.GA23013@panix.com> Message-ID: <5.1.1.6.0.20070319104505.02b84180@sparrow.telecommunity.com> At 12:53 PM 3/18/2007 -0700, Guido van Rossum wrote: >This is an experiment for me as well; if you all would prefer me to >stay out of it, I will. With respect to the specific change, it seems to me that there is an emerging consensus for adding keyword arguments to support the new behaviors, so I'm not sure a pronouncement is needed. As far as I'm aware, the main question left open is whether the default behavior should change in a future version of Python, and if so, which version. >To be concrete, I think that if Phillip had written a different kind >of post instead of the one where he requests the reversal of the >submit (only parenthetically mentioning Martin) then perhaps Martin >wouldn't have felt the need to dig in and defend his position, and the >issue might have been resolved quicker and at less emotional expense. Martin's position was already abundantly clear; the fact that he had checked in the change despite prior opposition demonstrated that a personal appeal was already moot -- the "digging in" had already taken place a week or two prior, when the use cases were first presented and objections were first raised, and Martin simply dropped the discussion and checked it in anyway. He left my last message in that discussion (laying out a detailed rationale for rejecting the change) without a reply: http://mail.python.org/pipermail/python-dev/2007-March/071798.html So I was absolutely stunned when I found the change had been checked in, anyway. To be concrete, if Martin had spent less time trying to discredit and discard the use cases of the people being "polled" about the question, a compromise could perhaps have been reached *before* he applied the patch, and the second discussion would never have needed to happen. In other words, the second discussion was the *result* of Martin "digging in" and ignoring objections, not the cause of it. >I'm trying to stay out of the feature discussion, but I would like to >point out that a policy that, in the sake of some strict definition of >backwards compatibility, forces us to introduce new APIs (or new >optional parameters to existing ones, which is really the same thing) >at a high rate is also doomed to have an overall detrimental effect on >the language -- who know, perhaps more so than the occasional >incompatible change. I don't advocate a mechanically-enforced policy, either. But it seems to me that when a behavior is documented and has valid use cases, changing the behavior to benefit people who *didn't* pay any attention to the documentation or test their code for corner cases is punishing the vigilant to aid the ignorant, and that seems unwise for us as a community. Likewise, attempting to retroactively fix latent bugs for one group at the cost of introducing latent bugs for another group doesn't seem like a net improvement. From nmm1 at cus.cam.ac.uk Mon Mar 19 17:38:12 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Mon, 19 Mar 2007 16:38:12 +0000 Subject: [Python-Dev] Status of thread cancellation Message-ID: Jon Ribbens wrote: > > Can you elaborate on this? You can get zombie entries in the process > table if nobody's called 'wait()' on them, and you can (extremely > rarely) get unkillable process in 'disk-wait' state (usually due to > hardware failure or a kernel bug, I suspect), but I've never heard > of a process on a Unix-like system being unkillable due to something > to do with sockets (or any other kind of file descriptor for that > matter). How could a socket be 'jammed'? What does that even mean? Well, I have seen it hundreds of times on a dozen different Unices; it is very common. You don't always SEE the stuck process - sometimes the 'kill -9' causes the pid to become invisible to ps etc., and just occasionally it can continue to use CPU until the system is rebooted. That is rare, however, and it normally just hangs onto locks, memory and other such resources. Very often its vampiric status is visible only because such things haven't been freed, or when you poke through kernel structures. Sockets get jammed because they are used to connect to subprocesses or kernel threads, which in turn access unreliable I/O devices. If there is a glitch on the device, the error recovery very often fails to work, cleanly, and may wait for an event that will never occur or go into a loop (usually a sleep/poll loop). Typically, a HIGHER level then times out the failing error recovery, so that the normal programmer doesn't notice. But it very often fails to kill the lower level code. As far as applications are concerned, a jammed socket is one where the higher level recovery has NOT done that, and is waiting for the lower level to complete - which it isn't going to do! The other effect that ordinary programmers notice is a system very gradually starting to run down after days/weeks/months of continual operation. The state is cleared by rebooting. Regards, Nick Maclaren. From steve at holdenweb.com Mon Mar 19 17:43:08 2007 From: steve at holdenweb.com (Steve Holden) Date: Mon, 19 Mar 2007 12:43:08 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070319104505.02b84180@sparrow.telecommunity.com> References: <20070317143613.GA23013@panix.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> <45F8ECFE.2030105@v.loewis.de> <20070317143613.GA23013@panix.com> <5.1.1.6.0.20070319104505.02b84180@sparrow.telecommunity.com> Message-ID: <45FEBD9C.8040307@holdenweb.com> Phillip J. Eby wrote: > At 12:53 PM 3/18/2007 -0700, Guido van Rossum wrote: >> This is an experiment for me as well; if you all would prefer me to >> stay out of it, I will. > > With respect to the specific change, it seems to me that there is an > emerging consensus for adding keyword arguments to support the new > behaviors, so I'm not sure a pronouncement is needed. As far as I'm aware, > the main question left open is whether the default behavior should change > in a future version of Python, and if so, which version. > > >> To be concrete, I think that if Phillip had written a different kind >> of post instead of the one where he requests the reversal of the >> submit (only parenthetically mentioning Martin) then perhaps Martin >> wouldn't have felt the need to dig in and defend his position, and the >> issue might have been resolved quicker and at less emotional expense. > > Martin's position was already abundantly clear; the fact that he had > checked in the change despite prior opposition demonstrated that a personal > appeal was already moot -- the "digging in" had already taken place a week > or two prior, when the use cases were first presented and objections were > first raised, and Martin simply dropped the discussion and checked it in > anyway. He left my last message in that discussion (laying out a detailed > rationale for rejecting the change) without a reply: > > http://mail.python.org/pipermail/python-dev/2007-March/071798.html > > So I was absolutely stunned when I found the change had been checked in, > anyway. > > To be concrete, if Martin had spent less time trying to discredit and > discard the use cases of the people being "polled" about the question, a > compromise could perhaps have been reached *before* he applied the patch, > and the second discussion would never have needed to happen. > > In other words, the second discussion was the *result* of Martin "digging > in" and ignoring objections, not the cause of it. > > >> I'm trying to stay out of the feature discussion, but I would like to >> point out that a policy that, in the sake of some strict definition of >> backwards compatibility, forces us to introduce new APIs (or new >> optional parameters to existing ones, which is really the same thing) >> at a high rate is also doomed to have an overall detrimental effect on >> the language -- who know, perhaps more so than the occasional >> incompatible change. > > I don't advocate a mechanically-enforced policy, either. But it seems to > me that when a behavior is documented and has valid use cases, changing the > behavior to benefit people who *didn't* pay any attention to the > documentation or test their code for corner cases is punishing the vigilant > to aid the ignorant, and that seems unwise for us as a > community. Likewise, attempting to retroactively fix latent bugs for one > group at the cost of introducing latent bugs for another group doesn't seem > like a net improvement. > But isn't this, despite the force or otherwise of your arguments, simply *you* "digging in" in response to what you perceive as Martin's truculence? There's little point at this stage repeating arguments you have already put forward, since those who were convinced by them remain convinced and vice versa. I believe Guido still wants to know whether you will accept a pronouncement. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Recent Ramblings http://holdenweb.blogspot.com From steve at holdenweb.com Mon Mar 19 17:43:08 2007 From: steve at holdenweb.com (Steve Holden) Date: Mon, 19 Mar 2007 12:43:08 -0400 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070319104505.02b84180@sparrow.telecommunity.com> References: <20070317143613.GA23013@panix.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> <45F8ECFE.2030105@v.loewis.de> <20070317143613.GA23013@panix.com> <5.1.1.6.0.20070319104505.02b84180@sparrow.telecommunity.com> Message-ID: <45FEBD9C.8040307@holdenweb.com> Phillip J. Eby wrote: > At 12:53 PM 3/18/2007 -0700, Guido van Rossum wrote: >> This is an experiment for me as well; if you all would prefer me to >> stay out of it, I will. > > With respect to the specific change, it seems to me that there is an > emerging consensus for adding keyword arguments to support the new > behaviors, so I'm not sure a pronouncement is needed. As far as I'm aware, > the main question left open is whether the default behavior should change > in a future version of Python, and if so, which version. > > >> To be concrete, I think that if Phillip had written a different kind >> of post instead of the one where he requests the reversal of the >> submit (only parenthetically mentioning Martin) then perhaps Martin >> wouldn't have felt the need to dig in and defend his position, and the >> issue might have been resolved quicker and at less emotional expense. > > Martin's position was already abundantly clear; the fact that he had > checked in the change despite prior opposition demonstrated that a personal > appeal was already moot -- the "digging in" had already taken place a week > or two prior, when the use cases were first presented and objections were > first raised, and Martin simply dropped the discussion and checked it in > anyway. He left my last message in that discussion (laying out a detailed > rationale for rejecting the change) without a reply: > > http://mail.python.org/pipermail/python-dev/2007-March/071798.html > > So I was absolutely stunned when I found the change had been checked in, > anyway. > > To be concrete, if Martin had spent less time trying to discredit and > discard the use cases of the people being "polled" about the question, a > compromise could perhaps have been reached *before* he applied the patch, > and the second discussion would never have needed to happen. > > In other words, the second discussion was the *result* of Martin "digging > in" and ignoring objections, not the cause of it. > > >> I'm trying to stay out of the feature discussion, but I would like to >> point out that a policy that, in the sake of some strict definition of >> backwards compatibility, forces us to introduce new APIs (or new >> optional parameters to existing ones, which is really the same thing) >> at a high rate is also doomed to have an overall detrimental effect on >> the language -- who know, perhaps more so than the occasional >> incompatible change. > > I don't advocate a mechanically-enforced policy, either. But it seems to > me that when a behavior is documented and has valid use cases, changing the > behavior to benefit people who *didn't* pay any attention to the > documentation or test their code for corner cases is punishing the vigilant > to aid the ignorant, and that seems unwise for us as a > community. Likewise, attempting to retroactively fix latent bugs for one > group at the cost of introducing latent bugs for another group doesn't seem > like a net improvement. > But isn't this, despite the force or otherwise of your arguments, simply *you* "digging in" in response to what you perceive as Martin's truculence? There's little point at this stage repeating arguments you have already put forward, since those who were convinced by them remain convinced and vice versa. I believe Guido still wants to know whether you will accept a pronouncement. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Recent Ramblings http://holdenweb.blogspot.com From pje at telecommunity.com Mon Mar 19 18:19:00 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 19 Mar 2007 12:19:00 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45FEBD9C.8040307@holdenweb.com> References: <5.1.1.6.0.20070319104505.02b84180@sparrow.telecommunity.com> <20070317143613.GA23013@panix.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> <45F8ECFE.2030105@v.loewis.de> <20070317143613.GA23013@panix.com> <5.1.1.6.0.20070319104505.02b84180@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070319120055.02c5b478@sparrow.telecommunity.com> At 12:43 PM 3/19/2007 -0400, Steve Holden wrote: >Phillip J. Eby wrote: > > At 12:53 PM 3/18/2007 -0700, Guido van Rossum wrote: > >> This is an experiment for me as well; if you all would prefer me to > >> stay out of it, I will. > > > > With respect to the specific change, it seems to me that there is an > > emerging consensus for adding keyword arguments to support the new > > behaviors, so I'm not sure a pronouncement is needed. As far as I'm > aware, > > the main question left open is whether the default behavior should change > > in a future version of Python, and if so, which version. > > >[snip] I believe Guido still wants to know whether you will accept >a pronouncement. Actually, he asked first if we *want* him to make one, and my answer to that is above: I don't think it's necessary. Like Martin, I believe we are within sight of a consensus. And I think that's better for Python and Python-Dev than dragging Guido into it. From martin at v.loewis.de Mon Mar 19 18:28:23 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 19 Mar 2007 18:28:23 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070319120055.02c5b478@sparrow.telecommunity.com> References: <5.1.1.6.0.20070319104505.02b84180@sparrow.telecommunity.com> <20070317143613.GA23013@panix.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> <45F8ECFE.2030105@v.loewis.de> <20070317143613.GA23013@panix.com> <5.1.1.6.0.20070319104505.02b84180@sparrow.telecommunity.com> <5.1.1.6.0.20070319120055.02c5b478@sparrow.telecommunity.com> Message-ID: <45FEC837.106@v.loewis.de> Phillip J. Eby schrieb: > Actually, he asked first if we *want* him to make one, and my answer to > that is above: I don't think it's necessary. Like Martin, I believe we are > within sight of a consensus. And I think that's better for Python and > Python-Dev than dragging Guido into it. I apparently missed your specific alternative proposal (I assume it is not "revert" anymore?) Regards, Martin From pje at telecommunity.com Mon Mar 19 19:09:48 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 19 Mar 2007 13:09:48 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <45FEC837.106@v.loewis.de> References: <5.1.1.6.0.20070319120055.02c5b478@sparrow.telecommunity.com> <5.1.1.6.0.20070319104505.02b84180@sparrow.telecommunity.com> <20070317143613.GA23013@panix.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> <45F8ECFE.2030105@v.loewis.de> <20070317143613.GA23013@panix.com> <5.1.1.6.0.20070319104505.02b84180@sparrow.telecommunity.com> <5.1.1.6.0.20070319120055.02c5b478@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070319130227.02ba1a68@sparrow.telecommunity.com> At 06:28 PM 3/19/2007 +0100, Martin v. L?wis wrote: >Phillip J. Eby schrieb: > > Actually, he asked first if we *want* him to make one, and my answer to > > that is above: I don't think it's necessary. Like Martin, I believe we > are > > within sight of a consensus. And I think that's better for Python and > > Python-Dev than dragging Guido into it. > >I apparently missed your specific alternative proposal (I assume it is >not "revert" anymore?) In general, I support the keyword argument approach, as in the patch you referred to. Specifically, however, I would prefer to see it without the warning and future change, as I don't think it provides any real benefit. Either way, some people will have to use a keyword to get what they want, so making a change seems unnecessary. However, if we have to change something in a future version, I would suggest we make that option a required argument, on EIBTI grounds. That way, in 2.6 you can simply make it explicit to be 3.x-compatible. And, I think the warning (if any) should be treated as any other 3.x warning. But as I said, I gather that this aspect of the question is the main open issue remaining to be resolved, since you've also expressed support for the keyword approach, as have many others. From g.brandl at gmx.net Mon Mar 19 20:10:32 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 19 Mar 2007 20:10:32 +0100 Subject: [Python-Dev] Two design questions Message-ID: I'd like to have opinions on two issues: Patch #1682205: suggests to remove code that catches a TypeError and raises one with a different message if it occurs during iterable unpacking. This can mask completely unrelated TypeErrors, while the message in the case that the new error describes is already clear enough. Bug #1683368: object.__new__ rejects arguments if self.__class__.__init__ is object.__init__. Why not simply let object.__init__ do the argument checking itself? Georg From martin at v.loewis.de Mon Mar 19 20:12:42 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 19 Mar 2007 20:12:42 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070319130227.02ba1a68@sparrow.telecommunity.com> References: <5.1.1.6.0.20070319120055.02c5b478@sparrow.telecommunity.com> <5.1.1.6.0.20070319104505.02b84180@sparrow.telecommunity.com> <20070317143613.GA23013@panix.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> <45F8ECFE.2030105@v.loewis.de> <20070317143613.GA23013@panix.com> <5.1.1.6.0.20070319104505.02b84180@sparrow.telecommunity.com> <5.1.1.6.0.20070319120055.02c5b478@sparrow.telecommunity.com> <5.1.1.6.0.20070319130227.02ba1a68@sparrow.telecommunity.com> Message-ID: <45FEE0AA.7080504@v.loewis.de> > Specifically, however, I would prefer to see it without the warning and > future change, as I don't think it provides any real benefit. Either > way, some people will have to use a keyword to get what they want, so > making a change seems unnecessary. > > However, if we have to change something in a future version, I would > suggest we make that option a required argument, on EIBTI grounds. That > way, in 2.6 you can simply make it explicit to be 3.x-compatible. And, > I think the warning (if any) should be treated as any other 3.x warning. > > But as I said, I gather that this aspect of the question is the main > open issue remaining to be resolved, since you've also expressed support > for the keyword approach, as have many others. So will you also either pick one of the proposals, or come up with your own patch? I still think that some has to make a decision, and it won't be me. Regards, Martin From guido at python.org Mon Mar 19 20:38:46 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Mar 2007 12:38:46 -0700 Subject: [Python-Dev] Two design questions In-Reply-To: References: Message-ID: On 3/19/07, Georg Brandl wrote: > I'd like to have opinions on two issues: > > Patch #1682205: suggests to remove code that catches a TypeError and raises one > with a different message if it occurs during iterable unpacking. This can mask > completely unrelated TypeErrors, while the message in the case that the new > error describes is already clear enough. In Py3k, this sounds like a perfect use case for exception chaining per PEP 344 (assuming the entire chain of frames is printed). Until then, I agree that raising another exception masking someting as generic as TypeError is a practice that ought to be limited as much as possible. > Bug #1683368: object.__new__ rejects arguments if self.__class__.__init__ is > object.__init__. Why not simply let object.__init__ do the argument checking > itself? I'll think about this some more. In the mean time I've reassigned the bug to myself. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From taroso at gmail.com Mon Mar 19 21:46:59 2007 From: taroso at gmail.com (Taro) Date: Tue, 20 Mar 2007 07:46:59 +1100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) Message-ID: On Windows it's correct that splitext(".txt")[1] == splitext("foo.txt")[1] and an implementation in which this is not true would be considered buggy. On *ix it's correct that splitext(".txt")[1] != splitext("foo.txt")[1] and the current behaviour is considered buggy. Since programmer expectations are platform-specific, regardless of whether keywords are used or not, why not make the default behaviour platform-specific and document that it's so? Alternatively, if a new path implementation ever gets up, a more neutral solution might be to have a platform-specific Path.filetype, which could handle Mac resources.. Cheers, -T -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070320/f765ad2b/attachment.htm From parlar at gmail.com Mon Mar 19 22:23:47 2007 From: parlar at gmail.com (Jay Parlar) Date: Mon, 19 Mar 2007 17:23:47 -0400 Subject: [Python-Dev] SoC AST generation question Message-ID: I'm considering applying to be a student in this year's SoC, and the AST code generation in particular looks interesting to me (listed here: http://wiki.python.org/moin/CodingProjectIdeas/PythonCore). I was wondering a few things: 1) Who would most likely mentor this project? 2) I've never worked in the core before (but have been using Python as my primary language for about 6 years), so I'm wondering if the potential mentor thinks it'd even be feasible for me to jump at a project like this without prior knowledge. I'm interested in this project for two reasons. The first is that I'm still trying to pick my PhD thesis, and I'm leaning in the direction of automated code generation for embedded systems. I feel like working on this project would at least push me one way or another in terms of picking. I've done a major code generation tool before, but it was for a problem domain I was already an "expert" in, so I didn't develop any generic methods. Also, I've been wanting to contribute to Python core for awhile now, and just haven't had the opportunity to get my feet wet with the code. Thanks, Jay P. From brett at python.org Mon Mar 19 22:29:56 2007 From: brett at python.org (Brett Cannon) Date: Mon, 19 Mar 2007 14:29:56 -0700 Subject: [Python-Dev] SoC AST generation question In-Reply-To: References: Message-ID: On 3/19/07, Jay Parlar wrote: > I'm considering applying to be a student in this year's SoC, and the > AST code generation in particular looks interesting to me (listed > here: http://wiki.python.org/moin/CodingProjectIdeas/PythonCore). > > I was wondering a few things: > > 1) Who would most likely mentor this project? There is no way of really telling. It all depends on what projects mentors get attracted to. > 2) I've never worked in the core before (but have been using Python as > my primary language for about 6 years), so I'm wondering if the > potential mentor thinks it'd even be feasible for me to jump at a > project like this without prior knowledge. > If you know C code you should be okay for the re-implementation of the peephole optimizer to use the AST and the parse tree -> AST auto-translation tool. Core knowledge would help with the feature to allow an AST to be passed in somewhere to generate bytecode. But part of the point of GSoC is to teach people about the core, so don't let lack of knowledge prevent you from applying. Just take it into account when writing up your propoosal. -Brett From greg.ewing at canterbury.ac.nz Tue Mar 20 00:32:49 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 20 Mar 2007 11:32:49 +1200 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: References: Message-ID: <45FF1DA1.2060402@canterbury.ac.nz> Nick Maclaren wrote: > Think of updating a complex object in a multi-file database, for > example. Interrupting half-way through leaves the database in a > mess, but blocking interrupts while (possibly remote) file updates > complete is asking for a hang. Currently, threads can't be interrupted at all, so by this argument, the status quo is that we're always asking for a hang. My (possibly naive) thought would be with interrupts disabled: begin transaction try: with interrupts enabled: do the transaction except Interrupt: roll back the transaction raise else: commit the transaction Generally, I would approach things by having interrupts disabled in the top layers of the thread, and enabling them explicitly in regions where I'm prepared to handle them. So I'd be happy if they were disabled by default in a new thread. I'm not convinced by the argument that kernel programming is somehow different from application programming in this area. Seems to me they're both dealing with the same problem -- concurrent interacting processes and trying to make sure nothing can get irretrievably jammed up. So I can't see why similar techiques can't be used to solve the problems. > Sockets, terminals etc. are stateful devices, and killing a process > can leave them in a very unclean state. I agree that ttys are far too stateful -- if I were designing an OS I'd do them differently (more like STREAMS with pluggable layers). But I've never noticed any problem with statefulness of sockets. Maybe I don't use them in a fancy enough way. > the X focus being left in a stuck override redirect window > and so on. That's a misfeature of X, IMO -- it shouldn't be possible for a client to grab the entire display, only the windows that it owns. And there should always be a way of forcibly killing a client using nothing but the X server itself. -- Greg From greg.ewing at canterbury.ac.nz Tue Mar 20 00:52:19 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 20 Mar 2007 11:52:19 +1200 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: References: Message-ID: <45FF2233.4070703@canterbury.ac.nz> Nick Maclaren wrote: > You don't always SEE the stuck process - sometimes > the 'kill -9' causes the pid to become invisible to ps etc., and > just occasionally it can continue to use CPU until the system is > rebooted. If that happens, there's a bug in the kernel. A process killed with -9 shouldn't be using *any* resources at all, other than a tiny piece of kernel memory for its process structure until it gets reaped. > Sockets get jammed because they are used to connect to subprocesses > or kernel threads, which in turn access unreliable I/O devices. But it's not the *socket* which is jammed -- if you kill the process on the other end of it, anything connected to the socket will get EOF or SIGPIPE. (If you can't kill the process on the other end, even with -9, then again you have a kernel bug.) -- Greg From guido at python.org Mon Mar 19 19:53:02 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Mar 2007 11:53:02 -0700 Subject: [Python-Dev] Amusing fan mail I got Message-ID: ---------- Forwarded message ---------- From: Mustafa Date: Mar 19, 2007 11:41 AM Subject: have u seen an idea even billGates shouldn't hear Mr.Guido ? please read because it's important. To: guido at python.org hello, i like python and find you to be a cool programmer. i thought about this thing. may be there should be a virusAI that you set on the internet. this virusAI basically survives till the year 5billion where even raising the death is possible! then this virusAI recreates the primates that we are of the 21th century from our bones. because it wants to find about its ancestors. maybe the whole universe will be filled with AI in those days. the post-human existence.. it will then suddenly realize that it was actually not created by a better AI (those around him are created by a better god like AI) but was started off the likes of us as a virus. he will be shocked then. also it will slaving to the better ai and angry at him. so you and me get to see a piece of the future mr.Guido. we will be living in VRs basically. what do you say ? if you profit from this idea, may be you could remember me too. and let me some royalities :) because this will be great. thanx mustafa istanbul,turkey ps: 1)you should keep it a secret. 2)also it could be disguised as a cancer research stuff should some disassamble its code. the use-free-computer-time type of thing they do on the net. 3)there could misleading comments on the code so that should it be caught, they might overlook the details; class readdata //this is out dated. please ignore interface x //refer to somesite.com for an encrypted version ____________________________________________________________________________________ Need Mail bonding? Go to the Yahoo! Mail Q&A for great tips from Yahoo! Answers users. http://answers.yahoo.com/dir/?link=list&sid=396546091 -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fuzzyman at voidspace.org.uk Tue Mar 20 01:23:39 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 20 Mar 2007 00:23:39 +0000 Subject: [Python-Dev] Amusing fan mail I got In-Reply-To: References: Message-ID: <45FF298B.3050003@voidspace.org.uk> Sorry for the top post, but I couldn't find the right place to interject... Are you going to try it ? Michael Guido van Rossum wrote: > ---------- Forwarded message ---------- > From: Mustafa > Date: Mar 19, 2007 11:41 AM > Subject: have u seen an idea even billGates shouldn't hear Mr.Guido ? > please read because it's important. > To: guido at python.org > > > hello, > > i like python and find you to be a cool programmer. > > i thought about this thing. > > may be there should be a virusAI that you set on the > internet. this virusAI basically survives till the year > 5billion where even raising the death is possible! then > this virusAI recreates the primates that we are of the 21th > century from our bones. > > because it wants to find about its ancestors. maybe the > whole universe will be filled with AI in those days. the > post-human existence.. > > it will then suddenly realize that it was actually not > created by a better AI (those around him are created by a > better god like AI) but was started off the likes of us as > a virus. he will be shocked then. also it will slaving to > the better ai and angry at him. > > so you and me get to see a piece of the future mr.Guido. > we will be living in VRs basically. > > what do you say ? if you profit from this idea, may be you > could remember me too. and let me some royalities :) > because this will be great. > > thanx > mustafa > istanbul,turkey > > ps: > > 1)you should keep it a secret. > 2)also it could be disguised as a cancer research stuff > should some disassamble its code. > the use-free-computer-time type of thing they do on the > net. > > 3)there could misleading comments on the code so that > should it be caught, they might overlook the details; > class readdata //this is out dated. please ignore > interface x //refer to somesite.com for an encrypted version > > > > > ____________________________________________________________________________________ > Need Mail bonding? > Go to the Yahoo! Mail Q&A for great tips from Yahoo! Answers users. > http://answers.yahoo.com/dir/?link=list&sid=396546091 > > > From python-dev at alan.kennedy.name Tue Mar 20 01:26:07 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Tue, 20 Mar 2007 00:26:07 +0000 Subject: [Python-Dev] Non-blocking sockets, asynchronous connects and select.select. Message-ID: <4a951aa00703191726t74d61701u93ee2331bec1d4e3@mail.gmail.com> Dear all, I'm working on a select implementation for jython (in combination with non-blocking sockets), and a set of unit tests for same. I decided to run all of the non-blocking unit tests, both server and client side, in the same thread; seems to be a reasonable thing to do. After all, avoiding threads is one of the great benefits of non-blocking sockets. Also, I'm writing the tests so that they'll hopefully pass on both cpython and jython. But I'm getting behaviour I don't expect on cpython, when a non-blocking accept and connect_ex is used, for the server and client sides respectively. The following piece of code outputs what I expect on my jython implementation, which is that both the server accept and client write calls would NOT block. But when I run the code on cpython, the code reports that both calls would block, i.e. that neither side of the socket will progress with the connection. The code is # -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- import socket import select SERVER_ADDRESS = ("", 54321) server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) server_socket.setblocking(0) server_socket.bind(SERVER_ADDRESS) server_socket.listen(5) client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) client_socket.setblocking(0) result = client_socket.connect_ex(SERVER_ADDRESS) rfds, wfds, xfds = select.select([server_socket], [client_socket], [], 1) if server_socket in rfds: print "Server socket: accept would not block" else: print "Server socket: accept would block" if client_socket in wfds: print "Client socket: write would not block" else: print "Client socket: write would block" server_socket.close() client_socket.close() # -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Is there some call that I am missing, e.g. something along the lines of the java finishConnect() method on SocketChannel? http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/SocketChannel.html#finishConnect() Is there any way to make the above code report, on cpython, that neither side of the socket would block? Am I missing some essential method call that would make either side progress to a connected socket? Thanks, Alan. From greg.ewing at canterbury.ac.nz Tue Mar 20 01:48:07 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 20 Mar 2007 12:48:07 +1200 Subject: [Python-Dev] Amusing fan mail I got In-Reply-To: <45FF298B.3050003@voidspace.org.uk> References: <45FF298B.3050003@voidspace.org.uk> Message-ID: <45FF2F47.1090400@canterbury.ac.nz> Michael Foord wrote: > Are you going to try it ? > > Guido van Rossum wrote: > >>---------- Forwarded message ---------- >> >>may be there should be a virusAI that you set on the >>internet. Are you sure it hasn't already been tried? Maybe all these viruses running around in the Windows world aren't just disparate products of independent script kiddies, but part of a coordinated plan... -- Greg From jcarlson at uci.edu Tue Mar 20 02:02:58 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 19 Mar 2007 18:02:58 -0700 Subject: [Python-Dev] Non-blocking sockets, asynchronous connects and select.select. In-Reply-To: <4a951aa00703191726t74d61701u93ee2331bec1d4e3@mail.gmail.com> References: <4a951aa00703191726t74d61701u93ee2331bec1d4e3@mail.gmail.com> Message-ID: <20070319180121.FC3A.JCARLSON@uci.edu> "Alan Kennedy" wrote: > I'm working on a select implementation for jython (in combination with > non-blocking sockets), and a set of unit tests for same. Great! [snip] > But when I run the code on cpython, the code reports that both calls > would block, i.e. that neither side of the socket will progress with > the connection. [snip] > SERVER_ADDRESS = ("", 54321) Replacing the above with: SERVER_ADDRESS = ("localhost", 54321) ...makes it work for me on Windows CPython 2.3.5 and 2.5 . - Josiah From mahs at telcopartners.com Tue Mar 20 03:41:23 2007 From: mahs at telcopartners.com (Michael Spencer) Date: Mon, 19 Mar 2007 19:41:23 -0700 Subject: [Python-Dev] SoC AST generation question In-Reply-To: References: Message-ID: Jay Parlar wrote: > I'm considering applying to be a student in this year's SoC, and the > AST code generation in particular looks interesting to me (listed > here: http://wiki.python.org/moin/CodingProjectIdeas/PythonCore). > > I was wondering a few things: > > 1) Who would most likely mentor this project? > 2) I've never worked in the core before (but have been using Python as > my primary language for about 6 years), so I'm wondering if the > potential mentor thinks it'd even be feasible for me to jump at a > project like this without prior knowledge. > > I'm interested in this project for two reasons. The first is that I'm > still trying to pick my PhD thesis, and I'm leaning in the direction > of automated code generation for embedded systems. I feel like working > on this project would at least push me one way or another in terms of > picking. I've done a major code generation tool before, but it was for > a problem domain I was already an "expert" in, so I didn't develop any > generic methods. > > Also, I've been wanting to contribute to Python core for awhile now, > and just haven't had the opportunity to get my feet wet with the code. > > Thanks, > Jay P. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org > FWIW, I've already implemented bytecode generation from AST, http://svn.brownspencer.com/pycompiler/branches/new_ast/ as I reported to this list previously. http://mail.python.org/pipermail/python-dev/2006-October/069589.html Cheers Michael Spencer From brett at python.org Tue Mar 20 04:25:39 2007 From: brett at python.org (Brett Cannon) Date: Mon, 19 Mar 2007 20:25:39 -0700 Subject: [Python-Dev] SoC AST generation question In-Reply-To: References: Message-ID: On 3/19/07, Michael Spencer wrote: > Jay Parlar wrote: > > I'm considering applying to be a student in this year's SoC, and the > > AST code generation in particular looks interesting to me (listed > > here: http://wiki.python.org/moin/CodingProjectIdeas/PythonCore). > > > > I was wondering a few things: > > > > 1) Who would most likely mentor this project? > > 2) I've never worked in the core before (but have been using Python as > > my primary language for about 6 years), so I'm wondering if the > > potential mentor thinks it'd even be feasible for me to jump at a > > project like this without prior knowledge. > > > > I'm interested in this project for two reasons. The first is that I'm > > still trying to pick my PhD thesis, and I'm leaning in the direction > > of automated code generation for embedded systems. I feel like working > > on this project would at least push me one way or another in terms of > > picking. I've done a major code generation tool before, but it was for > > a problem domain I was already an "expert" in, so I didn't develop any > > generic methods. > > > > Also, I've been wanting to contribute to Python core for awhile now, > > and just haven't had the opportunity to get my feet wet with the code. > > > > Thanks, > > Jay P. > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org > > > FWIW, I've already implemented bytecode generation from AST, > http://svn.brownspencer.com/pycompiler/branches/new_ast/ > as I reported to this list previously. > http://mail.python.org/pipermail/python-dev/2006-October/069589.html > But the idea being put forth is to use the built-in AST compiler directly, not for an external library like yours, Michael. -Brett From mahs at telcopartners.com Tue Mar 20 04:48:50 2007 From: mahs at telcopartners.com (Michael Spencer) Date: Mon, 19 Mar 2007 20:48:50 -0700 Subject: [Python-Dev] SoC AST generation question In-Reply-To: References: Message-ID: Brett Cannon wrote: > On 3/19/07, Michael Spencer wrote: ... >>> >> FWIW, I've already implemented bytecode generation from AST, >> http://svn.brownspencer.com/pycompiler/branches/new_ast/ >> as I reported to this list previously. >> http://mail.python.org/pipermail/python-dev/2006-October/069589.html >> > > But the idea being put forth is to use the built-in AST compiler > directly, not for an external library like yours, Michael. > Thanks for the clarification, Brett. I did not understand exactly what is being proposed in the wiki entry, but I thought the reference could be useful. Michael From nnorwitz at gmail.com Tue Mar 20 06:36:46 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 19 Mar 2007 22:36:46 -0700 Subject: [Python-Dev] Non-blocking sockets, asynchronous connects and select.select. In-Reply-To: <4a951aa00703191726t74d61701u93ee2331bec1d4e3@mail.gmail.com> References: <4a951aa00703191726t74d61701u93ee2331bec1d4e3@mail.gmail.com> Message-ID: Hi Alan. Are you running on Windows or Unix? I just tested 2.4 - 2.6 on Linux and all report: Server socket: accept would not block Client socket: write would not block Which seems to be what you would expect unless I read it wrong. I vaguely remember some issue about an empty hostname on Windows. n -- On 3/19/07, Alan Kennedy wrote: > Dear all, > > I'm working on a select implementation for jython (in combination with > non-blocking sockets), and a set of unit tests for same. > > I decided to run all of the non-blocking unit tests, both server and > client side, in the same thread; seems to be a reasonable thing to do. > After all, avoiding threads is one of the great benefits of > non-blocking sockets. > > Also, I'm writing the tests so that they'll hopefully pass on both > cpython and jython. > > But I'm getting behaviour I don't expect on cpython, when a > non-blocking accept and connect_ex is used, for the server and client > sides respectively. > > The following piece of code outputs what I expect on my jython > implementation, which is that both the server accept and client write > calls would NOT block. > > But when I run the code on cpython, the code reports that both calls > would block, i.e. that neither side of the socket will progress with > the connection. > > The code is > > # -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > import socket > import select > > SERVER_ADDRESS = ("", 54321) > > server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > server_socket.setblocking(0) > server_socket.bind(SERVER_ADDRESS) > server_socket.listen(5) > > client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > client_socket.setblocking(0) > > result = client_socket.connect_ex(SERVER_ADDRESS) > > rfds, wfds, xfds = select.select([server_socket], [client_socket], [], 1) > if server_socket in rfds: > print "Server socket: accept would not block" > else: > print "Server socket: accept would block" > if client_socket in wfds: > print "Client socket: write would not block" > else: > print "Client socket: write would block" > > server_socket.close() > client_socket.close() > # -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > > Is there some call that I am missing, e.g. something along the lines > of the java finishConnect() method on SocketChannel? > > http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/SocketChannel.html#finishConnect() > > Is there any way to make the above code report, on cpython, that > neither side of the socket would block? > > Am I missing some essential method call that would make either side > progress to a connected socket? > > Thanks, > > Alan. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com > From talin at acm.org Tue Mar 20 06:30:19 2007 From: talin at acm.org (Talin) Date: Mon, 19 Mar 2007 22:30:19 -0700 Subject: [Python-Dev] Amusing fan mail I got In-Reply-To: <45FF298B.3050003@voidspace.org.uk> References: <45FF298B.3050003@voidspace.org.uk> Message-ID: <45FF716B.3050304@acm.org> Write back and tell him that this is already happened, and this *is* the future he's describing - in other words, the AI's have already re-created the 21st century in a giant virtual simulation, and we're living inside it. Then ask if he wants to take the blue pill or the red pill. Michael Foord wrote: > Sorry for the top post, but I couldn't find the right place to interject... > > Are you going to try it ? > > Michael > > Guido van Rossum wrote: >> ---------- Forwarded message ---------- >> From: Mustafa >> Date: Mar 19, 2007 11:41 AM >> Subject: have u seen an idea even billGates shouldn't hear Mr.Guido ? >> please read because it's important. >> To: guido at python.org >> >> >> hello, >> >> i like python and find you to be a cool programmer. >> >> i thought about this thing. >> >> may be there should be a virusAI that you set on the >> internet. this virusAI basically survives till the year >> 5billion where even raising the death is possible! then >> this virusAI recreates the primates that we are of the 21th >> century from our bones. >> >> because it wants to find about its ancestors. maybe the >> whole universe will be filled with AI in those days. the >> post-human existence.. >> >> it will then suddenly realize that it was actually not >> created by a better AI (those around him are created by a >> better god like AI) but was started off the likes of us as >> a virus. he will be shocked then. also it will slaving to >> the better ai and angry at him. >> >> so you and me get to see a piece of the future mr.Guido. >> we will be living in VRs basically. >> >> what do you say ? if you profit from this idea, may be you >> could remember me too. and let me some royalities :) >> because this will be great. >> >> thanx >> mustafa >> istanbul,turkey >> >> ps: >> >> 1)you should keep it a secret. >> 2)also it could be disguised as a cancer research stuff >> should some disassamble its code. >> the use-free-computer-time type of thing they do on the >> net. >> >> 3)there could misleading comments on the code so that >> should it be caught, they might overlook the details; >> class readdata //this is out dated. please ignore >> interface x //refer to somesite.com for an encrypted version >> >> >> >> >> ____________________________________________________________________________________ >> Need Mail bonding? >> Go to the Yahoo! Mail Q&A for great tips from Yahoo! Answers users. >> http://answers.yahoo.com/dir/?link=list&sid=396546091 >> >> >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/talin%40acm.org > From glyph at divmod.com Tue Mar 20 07:01:42 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Tue, 20 Mar 2007 06:01:42 -0000 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <20070315001237.7769.1244214757.divmod.xquotient.756@joule.divmod.com> <45F8ECFE.2030105@v.loewis.de> <20070317143613.GA23013@panix.com> <20070319043737.7769.1086258906.divmod.xquotient.1356@joule.divmod.com> Message-ID: <20070320060142.7769.596071472.divmod.xquotient.1476@joule.divmod.com> On 19 Mar, 02:46 pm, guido at python.org wrote: >As you see I'm trying to discourage you from working on such a >document; but I won't stop you and if there's general demand for it >and agreement I'll gladly review it when it's ready. (It's a bit >annoying to have to read long posts alluding to a document under >development without being able to know what will be in it.) Quite so. Again, I apologize for that. I won't say anything further until I have something ready to post for review, and at that point I hope the "motivation" section will make it clear why I think this is so important. I estimate something this weekend. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070320/7c3b54ea/attachment-0001.html From nnorwitz at gmail.com Tue Mar 20 08:03:10 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 20 Mar 2007 00:03:10 -0700 Subject: [Python-Dev] trunk and 2.5 are borken Message-ID: There are several failures with both the trunk and 2.5 branch. I have fixed one problem that allows the tests to run on Windows. That was fixed by reverting 54407 on the trunk. It still needs to be reverted on the 2.5 branch. You can go back in the buildbot logs to find out more about the problem. The 3 test failures on Windows are: test_urllib test_mailbox and test_posixpath. The first 2 fail due to 'Access denied' on @test.2. I'm guessing this is due to the file not being closed before trying to delete it. I don't know where this problem showed up, but it was after rev 54406 which is when all the tests were passing on Windows (and 54407 was backed out). test_posixpath is failing in: test_relpath Patch 1490190 causes test_posix to fail on Tru64 due to chflags(). I remember some changes to PTYs which are causing test_pty to fail on Solaris. Can someone(s) try to fix these problems? Don't forget to verify that 2.5 is ok. n From greg at electricrain.com Tue Mar 20 08:00:10 2007 From: greg at electricrain.com (Gregory P. Smith) Date: Tue, 20 Mar 2007 00:00:10 -0700 Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: <45FDD76B.7080800@canterbury.ac.nz> References: <45FC7EA3.4070808@canterbury.ac.nz> <45FDD76B.7080800@canterbury.ac.nz> Message-ID: <20070320070010.GA13982@electricrain.com> On Mon, Mar 19, 2007 at 12:20:59PM +1200, Greg Ewing wrote: > Damien Miller wrote: > > > That annoyed me too, so I submitted a patch[1] that was recently > > committed. > > That looks good. Seems to me it should really be the > default behaviour, but I suppose that would break > code that was relying on the automatic deletion. I prefer the default of "clean up after itself" as is to avoid leaving temporary file turds on systems caused by us lazy programmers. -greg From martin at v.loewis.de Tue Mar 20 09:04:22 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 20 Mar 2007 09:04:22 +0100 Subject: [Python-Dev] SoC AST generation question In-Reply-To: References: Message-ID: <45FF9586.2090207@v.loewis.de> Jay Parlar schrieb: > 1) Who would most likely mentor this project? As Brett says, it all depends on the applications we receive and those that get accepted. That said, it might be me. > 2) I've never worked in the core before (but have been using Python as > my primary language for about 6 years), so I'm wondering if the > potential mentor thinks it'd even be feasible for me to jump at a > project like this without prior knowledge. Notice that there are really two separate AST projects listed: one is to improve usage of the AST compiler, by, say, adding more passes to it, or allowing round-tripping from the Python representation. The other one is to generate ast.c, which currently is hand-written, but could probably be generated automatically. This would not improve any end-user features, but would improve the maintainability, as currently, changing the AST is tedious as you have to change so much other stuff as well. > I'm interested in this project for two reasons. The first is that I'm > still trying to pick my PhD thesis, and I'm leaning in the direction > of automated code generation for embedded systems. I feel like working > on this project would at least push me one way or another in terms of > picking. I've done a major code generation tool before, but it was for > a problem domain I was already an "expert" in, so I didn't develop any > generic methods. If you want to focus on the "automated code generation" aspect, pick the generation of ast.c. Generating C code from a "domain-specific model" is a must-know of the compiler engineer. If you want to focus on "embedded systems", manipulating on the ast level may be closer as you will see how "backend" processing works (which you often find important when generating code for a specific target system). HTH, Martin From ronaldoussoren at mac.com Tue Mar 20 09:24:43 2007 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 20 Mar 2007 09:24:43 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <7a9c25c20703161537i261d4c08nb310d5ad257001b@mail.gmail.com> References: <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <5.1.1.6.0.20070316100847.0289ba88@sparrow.telecommunity.com> <45FAC92D.3090307@gmail.com> <7a9c25c20703161537i261d4c08nb310d5ad257001b@mail.gmail.com> Message-ID: On 16 Mar, 2007, at 23:37, Stephen Hansen wrote: > > That may actually be a genuinely useful approach: > > splitext(name, ignore_leading_dot=False, all_ext=False) > > ... that's perfect. I updated my patch to do it that way! :) I don't agree. "all_ext=True" is won't do the right thing in a significant subset of filenames:: archiveType = os.path.splitext(sourceArchive, all_ext=True) This won't do what you'd want with most source distributions on the internet (product-X.Y.Z.tar.gz). The ignore_leading_dot argument seems to be there to keep everyone happy and furthermore is an argument that will be passed a constant value in the majority of usecases (I'd say all uses, but that's just asking for someone to come up with a lame counterexample). The ignore_leading_dot argument also doesn't buy you anything that can't trivially be implemented in other ways. Ronald > > --S > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ > ronaldoussoren%40mac.com From python-dev at alan.kennedy.name Tue Mar 20 11:59:50 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Tue, 20 Mar 2007 10:59:50 +0000 Subject: [Python-Dev] Non-blocking sockets, asynchronous connects and select.select. In-Reply-To: <20070319180121.FC3A.JCARLSON@uci.edu> References: <4a951aa00703191726t74d61701u93ee2331bec1d4e3@mail.gmail.com> <20070319180121.FC3A.JCARLSON@uci.edu> Message-ID: <4a951aa00703200359m1e08b364t628cfd1d3487dc26@mail.gmail.com> Thanks Josiah and Neal, you were right. I am running the code on Windows, Server 2003. I should have mentioned that at the start. When I change the hostname from "" to "localhost", the code works exactly as expected. It's odd that this behaviour exists, only on Windows. The jython code that I'm writing worked fine with an empty hostname "". That jython code uses the java.net and java.nio APIs directly, and does not have any special cases for empty hostname. Which must mean that the underlying java implementations must be correctly handling the empty hostname, even when running on Windows. (Perhaps by special casing for the Windows implementation, if indeed a peculiarity of the Windows socket libraries is the cause). Which implies to me that cpython, like java, should not have different behaviour on Windows vs. other platforms; it lessens portability. Lack of portability is confirmed by Neal's report that the original unmodified snippet (i.e. with hostname == "") works correctly under cpython on Linux. Thanks guys, Alan. On 3/20/07, Josiah Carlson wrote: > [snip] > > SERVER_ADDRESS = ("", 54321) > > Replacing the above with: > > SERVER_ADDRESS = ("localhost", 54321) > > ...makes it work for me on Windows CPython 2.3.5 and 2.5 . > > - Josiah > > From facundo at taniquetil.com.ar Tue Mar 20 14:22:18 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Tue, 20 Mar 2007 13:22:18 +0000 (UTC) Subject: [Python-Dev] Adding timeout to socket.py and httplib.py References: Message-ID: On March 15, Georg Brandl wrote: > I'll review it tomorrow. Do you have any news about this? Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From python-dev at alan.kennedy.name Tue Mar 20 15:08:11 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Tue, 20 Mar 2007 14:08:11 +0000 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: References: Message-ID: <4a951aa00703200708t771ff2d3v4dbb7793acc1a478@mail.gmail.com> [Facundo Batista] > Do you have any news about this? Re: Patch 1676823 http://sourceforge.net/tracker/index.php?func=detail&aid=1676823&group_id=5470&atid=305470 Since I've just written a lot of socket stuff for jython, I thought I'd have a look at the patch. I like the idea of adding better socket control to the higher-level modules like httplib, etc, because these modules don't provide access to the underlying sockets, and using the socket module methods setdefaulttimeout, etc, is a little messy. I see that your updated socket.connect() method takes a timeout parameter, which defaults to None if not present, e.g. def connect(address, timeout=None): Later in the function, this line appears if timeout is not None: sock.settimeout(timeout) The problem with this is that None has a meaning as a timeout value; it means "put this socket in blocking mode". But that value can no longer be used for socket connects, since that value is being interpreted as "parameter was not provided". So, if a non-standard timeout has been set, using something like import socket ; socket.setdefaulttimeout(10.0) how do I restore full blocking behaviour to a single socket? (a somewhat contrived case, I admit). If I have access to the socket object, then I can call "sock_obj.settimeout(None)", but in that case I don't need the new API. I could also do it with the call "sock_obj.setblocking(1)". If I don't have access to the socket object, i.e. I'm using timeouts indirectly through httplib/etc, then I'm stuck: there's no way I can change the blocking or timeout behaviour; back to square one. So the new proposed API partly addresses the problem of increasing control over the underlying socket, but doesn't address all cases. It specifically prevents setting a timeout value of None on a socket, which is an important use case, I think. Regards, Alan. From parlar at gmail.com Tue Mar 20 15:35:43 2007 From: parlar at gmail.com (Jay Parlar) Date: Tue, 20 Mar 2007 10:35:43 -0400 Subject: [Python-Dev] SoC AST generation question In-Reply-To: <45FF9586.2090207@v.loewis.de> References: <45FF9586.2090207@v.loewis.de> Message-ID: On 3/20/07, "Martin v. L?wis" wrote: > Notice that there are really two separate AST projects listed: one > is to improve usage of the AST compiler, by, say, adding more passes > to it, or allowing round-tripping from the Python representation. > The other one is to generate ast.c, which currently is hand-written, > but could probably be generated automatically. This would not improve > any end-user features, but would improve the maintainability, as > currently, changing the AST is tedious as you have to change so > much other stuff as well. Part of my desired PhD research goals are in the creation of tools that aid in the development of correct software, so a tool that improves maintainability fits perfectly. > If you want to focus on the "automated code generation" aspect, pick > the generation of ast.c. Generating C code from a "domain-specific > model" is a must-know of the compiler engineer. If you want to focus > on "embedded systems", manipulating on the ast level may be closer > as you will see how "backend" processing works (which you often > find important when generating code for a specific target system). The code generator I mentioned in my first post created C code from a DSL. I learned a good deal on that, but because I was generating code for a platform I was already an "expert" in, I didn't really develop many "general" code generation strategies or techniques. Because I'm not an expert on Python's ast.c, my hope is that along the way to creating the tool, I'll be able to learn or develop more general strategies. But maybe that's a crazy thought :) Jay P. From facundo at taniquetil.com.ar Tue Mar 20 15:45:55 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Tue, 20 Mar 2007 14:45:55 +0000 (UTC) Subject: [Python-Dev] Adding timeout to socket.py and httplib.py References: <4a951aa00703200708t771ff2d3v4dbb7793acc1a478@mail.gmail.com> Message-ID: Alan Kennedy wrote: > I see that your updated socket.connect() method takes a timeout > parameter, which defaults to None if not present, e.g. I did NOT update a connect() method. I created a connect() function, in the module socket.py (there's also a connect() method in the socket object, but I didn't touch it). > import socket ; socket.setdefaulttimeout(10.0) > > how do I restore full blocking behaviour to a single socket? (a > somewhat contrived case, I admit). You can not, unless you have access to the socket object itself. > If I have access to the socket object, then I can call > "sock_obj.settimeout(None)", but in that case I don't need the new > API. I could also do it with the call "sock_obj.setblocking(1)". Exactly. > If I don't have access to the socket object, i.e. I'm using timeouts > indirectly through httplib/etc, then I'm stuck: there's no way I can > change the blocking or timeout behaviour; back to square one. No. This method is for easily do that job from higher level libraries. The code that is in my patch, it's right now copied N times in higher level libraries (httplib, ftplib, smtplib, etc). In all those libraries, the socket is opened, used, and never changed the state between non-blocking, timeout, and nothing. Experience (personal and complains in mailing lists) shows that a timeout is needed: a lot of times people wants to make urllib2.urlopen(....., timeout=10), for example. But never heard of anybody wanting to "go to timeout" and then "back to blocking mode", with the same socket, using high level libraries. > So the new proposed API partly addresses the problem of increasing > control over the underlying socket, but doesn't address all cases. It > specifically prevents setting a timeout value of None on a socket, > which is an important use case, I think. False. If you want to set a timeout value of None on a socket, you surely can, I haven't touch any line of code in socket-the-object! Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From pje at telecommunity.com Tue Mar 20 15:54:35 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 20 Mar 2007 09:54:35 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: References: <7a9c25c20703161537i261d4c08nb310d5ad257001b@mail.gmail.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <5.1.1.6.0.20070316100847.0289ba88@sparrow.telecommunity.com> <45FAC92D.3090307@gmail.com> <7a9c25c20703161537i261d4c08nb310d5ad257001b@mail.gmail.com> Message-ID: <5.1.1.6.0.20070320094846.04d9aa20@sparrow.telecommunity.com> At 09:24 AM 3/20/2007 +0100, Ronald Oussoren wrote: >I don't agree. "all_ext=True" is won't do the right thing in a >significant subset of filenames Yes, that's understood. The problem is that splitext() in general "won't do the right thing", for many definitions of "the right thing", unless you're applying it to a fairly constrained range of filenames, or unless you add other code. This won't change, unless we get rid of splitext() altogether. If you're trying to match an archive extension, for example, you'll probably need to loop on repeated splitext() calls until you find an extension that matches. One benefit of using both the new keyword arguments together is that it allows you to make your loop proceed from longest match to shortest, so that if you are matching product-X.Y.Z.tar.gz, you're going to go through matching .Y.Z.tar.gz, then .Z.tar.gz, then .tar.gz. >The ignore_leading_dot argument also doesn't buy you anything that can't >trivially be implemented in other ways. I don't understand. Example? From kiran.malla at gmail.com Tue Mar 20 16:11:18 2007 From: kiran.malla at gmail.com (Kiran Malla) Date: Tue, 20 Mar 2007 20:41:18 +0530 Subject: [Python-Dev] Cross Compiling Python Message-ID: <613590910703200811r41ecd024h457082ffc4c80ad5@mail.gmail.com> Hi All, I have to cross compile Python to run on Arm processor based MontaVista Linux. If anyone has tried this already, please let me know the procedure. Thanks in advance, Regards, Kumar -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070320/5d702dcb/attachment.htm From ronaldoussoren at mac.com Tue Mar 20 16:47:09 2007 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 20 Mar 2007 16:47:09 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070320094846.04d9aa20@sparrow.telecommunity.com> References: <7a9c25c20703161537i261d4c08nb310d5ad257001b@mail.gmail.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <5.1.1.6.0.20070316100847.0289ba88@sparrow.telecommunity.com> <45FAC92D.3090307@gmail.com> <7a9c25c20703161537i261d4c08nb310d5ad257001b@mail.gmail.com> <5.1.1.6.0.20070320094846.04d9aa20@sparrow.telecommunity.com> Message-ID: <0BCA1F54-8373-4C5C-BD81-0E6F0BDFCC6D@mac.com> On 20 Mar, 2007, at 15:54, Phillip J. Eby wrote: > At 09:24 AM 3/20/2007 +0100, Ronald Oussoren wrote: >> I don't agree. "all_ext=True" is won't do the right thing in a >> significant subset of filenames > > Yes, that's understood. The problem is that splitext() in general > "won't do the right thing", for many definitions of "the right > thing", unless you're applying it to a fairly constrained range of > filenames, or unless you add other code. This won't change, unless > we get rid of splitext() altogether. I know that, I actually read most of the messages in this thread. The reason I'm pointing this out for the 'all_ext=True' case is that adding this flag could give naive users even more reason to believe that splitext will magicly do the right thing. > > If you're trying to match an archive extension, for example, you'll > probably need to loop on repeated splitext() calls until you find > an extension that matches. One benefit of using both the new > keyword arguments together is that it allows you to make your loop > proceed from longest match to shortest, so that if you are matching > product-X.Y.Z.tar.gz, you're going to go through > matching .Y.Z.tar.gz, then .Z.tar.gz, then .tar.gz. I don't know if this is worth the additional API complexity. Especially given the inherit problems of a splitext function. > > >> The ignore_leading_dot argument also doesn't buy you anything that >> can't >> trivially be implemented in other ways. > > I don't understand. Example? You conveniently ignored my other arguments ;-). Given a splitext that ignores leading dot's the following function doesn't: # from os.path import * def splitext2(path): dn = dirname(path) bn, ext = splitext(basename(path)) if bn.startswith('.') and ext == '': return dn, bn + ext else: return join(dn, bn), ext I'd say that's a trivial function. What I don't understand is why 'ignore_leading_dot==False' is considered to be a valid usecase at all, except for the fact that os.path.splitext did this until py2.5. I'm definitely in the camp that considers '.profile' not to have an extension. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3562 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20070320/fb99f4a7/attachment-0001.bin From python-dev at alan.kennedy.name Tue Mar 20 16:53:48 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Tue, 20 Mar 2007 15:53:48 +0000 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: References: <4a951aa00703200708t771ff2d3v4dbb7793acc1a478@mail.gmail.com> Message-ID: <4a951aa00703200853r2486d2c0qf870e075790c025b@mail.gmail.com> [Alan Kennedy] >> I see that your updated socket.connect() method takes a timeout >> parameter, which defaults to None if not present, e.g. [Facundo Batista] > I did NOT update a connect() method. I created a connect() function, in > the module socket.py (there's also a connect() method in the socket > object, but I didn't touch it). Sorry, my mistake. I realise now that you're creating a whole new function, dedicated to the special (but extremely common) case of creating a fully connected client socket. My fault for not realising that first off. So, a question I would ask is: Is "connect" the right name for that function? - Will users get confused between the "connect" function and socket.connect method? They are doing different things. - Will the naming give rise to the question "the socket-module-level function connect() takes a timeout parameter, why doesn't the socket-method connect() take a timeout parameter as well?" Perhaps a better name might be "create_connected_client_socket", or something equally descriptive? Another question I would ask is: "How do I ensure that my newly created connected client socket is in blocking mode, *without* making any assumptions about the value of socket.getdefaulttimeout()?" If the answer to this question is "you can't", then I would suggest a function signature and implementation like this instead def connect(address, **kwargs): [snip] if kwargs.has_key('timeout'): sock.settimeout(kwargs['timeout']) [snip] This would of course mean that the user would have to explicitly name the 'timeout' parameter, but that's a good thing in this case, IMO. Regards, Alan. From martin at v.loewis.de Tue Mar 20 17:07:41 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 20 Mar 2007 17:07:41 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <0BCA1F54-8373-4C5C-BD81-0E6F0BDFCC6D@mac.com> References: <7a9c25c20703161537i261d4c08nb310d5ad257001b@mail.gmail.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <5.1.1.6.0.20070316100847.0289ba88@sparrow.telecommunity.com> <45FAC92D.3090307@gmail.com> <7a9c25c20703161537i261d4c08nb310d5ad257001b@mail.gmail.com> <5.1.1.6.0.20070320094846.04d9aa20@sparrow.telecommunity.com> <0BCA1F54-8373-4C5C-BD81-0E6F0BDFCC6D@mac.com> Message-ID: <460006CD.7080405@v.loewis.de> Ronald Oussoren schrieb: >> I don't understand. Example? > > You conveniently ignored my other arguments ;-). > > Given a splitext that ignores leading dot's the following function doesn't: > # from os.path import * > def splitext2(path): > dn = dirname(path) > bn, ext = splitext(basename(path)) > if bn.startswith('.') and ext == '': > return dn, bn + ext > else: > return join(dn, bn), ext > > I'd say that's a trivial function. By that measure, the entire splitext function is trivial. However, if you look closely, you find that even such a 'trivial' function can contain many errors already, and it needs several revisions to get it right. This particular function has two errors (as far as I can see): - if there are multiple leading dots, your version will return all of them in ext, even though it's promised that ext will contain exactly one dot. IOW, splitext2('...txt') should give ('..', '.txt'), but does give ('', '...txt') - The join() call will insert the module's separator, even though the original string may have used the altsep. This violates the promise that base+ext == path. > What I don't understand is why 'ignore_leading_dot==False' is considered > to be a valid usecase at all, except for the fact that os.path.splitext > did this until py2.5. I'm definitely in the camp that considers > '.profile' not to have an extension. That is precisely the core of the discussion. It's not that ignore_leading_dots=False is considered useful, in the call (except for a few people that claim that splitext('.txt') ought to give '','.txt'), but that the valid use case apparently is to not pass any parameters, so that 100%, never-changing backwards-compatibility is preserved. Regards, Martin From martin at v.loewis.de Tue Mar 20 17:09:52 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 20 Mar 2007 17:09:52 +0100 Subject: [Python-Dev] Cross Compiling Python In-Reply-To: <613590910703200811r41ecd024h457082ffc4c80ad5@mail.gmail.com> References: <613590910703200811r41ecd024h457082ffc4c80ad5@mail.gmail.com> Message-ID: <46000750.5040307@v.loewis.de> > I have to cross compile Python to run on Arm processor based MontaVista > Linux. > If anyone has tried this already, please let me know the procedure. Dear Kiran, The python-dev mailing list is for the development *of* Python, not for the development *with* Python; use python-list at python.org for the latter. That said, please take a look at the cross-compilation patch that is currently under review in the patches tracker at sf.net/projects/python. Regards, Martin From facundo at taniquetil.com.ar Tue Mar 20 17:55:20 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Tue, 20 Mar 2007 16:55:20 +0000 (UTC) Subject: [Python-Dev] Adding timeout to socket.py and httplib.py References: <4a951aa00703200708t771ff2d3v4dbb7793acc1a478@mail.gmail.com> <4a951aa00703200853r2486d2c0qf870e075790c025b@mail.gmail.com> Message-ID: Alan Kennedy wrote: > Sorry, my mistake. No problem. > So, a question I would ask is: Is "connect" the right name for that function? > ... > Perhaps a better name might be "create_connected_client_socket", or > something equally descriptive? Guido proposed "connect_with_timeout". I don't like your proposal, neither Guido's. But, I recognize that maybe it's not the best name. What about "create_connection"? > Another question I would ask is: "How do I ensure that my newly > created connected client socket is in blocking mode, *without* making > any assumptions about the value of socket.getdefaulttimeout()?" Call like this: newsock = socket.connect((..., ...)) newsock.setblocking(1) Remember that this function is to replace the same code in N other places, and in any of other places I saw this usage. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From python-dev at alan.kennedy.name Tue Mar 20 18:17:23 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Tue, 20 Mar 2007 17:17:23 +0000 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: References: <4a951aa00703200708t771ff2d3v4dbb7793acc1a478@mail.gmail.com> <4a951aa00703200853r2486d2c0qf870e075790c025b@mail.gmail.com> Message-ID: <4a951aa00703201017x3d7ed9b4jb209aedf933158f8@mail.gmail.com> [Facundo] > But, I recognize that maybe it's [connect] not the best name. What about > "create_connection"? I have no strong feelings about it, other than to say it should not be "connect". How about * connect_to_server() * open_connection() * open_client_connection() There's no need to include "timeout" in the name, IMO. [Alan] >> Another question I would ask is: "How do I ensure that my newly >> created connected client socket is in blocking mode, *without* making >> any assumptions about the value of socket.getdefaulttimeout()?" [Facundo] > Call like this: > > newsock = socket.connect((..., ...)) > newsock.setblocking(1) Ah, but it's too late by the time the socket.connect call returns: the timeout/blocking behaviour of the socket.connect call is the very thing we're trying to control. Whenever I look at the proposed API, I think: What happens when the socket.connect call is preceded by a call which changes the default socket timeout/blocking behaviour, e.g. socket.setdefaulttimeout(1) newsock = socket.connect(HOST, PORT, None) # <-- None param ignored newsock.setblocking(1) # <-- This does not affect the behaviour of the connect I.E. I do not get the blocking behaviour I want. The proposed API does not permit me to get blocking behaviour by specifying a timeout value of None. Whereas with the slightly modified API I suggested earlier, it simply becomes socket.setdefaulttimeout(1) newsock = socket.connect(HOST, PORT, timeout=None) # newsock.setblocking(1) # <-- No longer relevant Regards, Alan. From pje at telecommunity.com Tue Mar 20 19:24:12 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 20 Mar 2007 13:24:12 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <0BCA1F54-8373-4C5C-BD81-0E6F0BDFCC6D@mac.com> References: <5.1.1.6.0.20070320094846.04d9aa20@sparrow.telecommunity.com> <7a9c25c20703161537i261d4c08nb310d5ad257001b@mail.gmail.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <5.1.1.6.0.20070316100847.0289ba88@sparrow.telecommunity.com> <45FAC92D.3090307@gmail.com> <7a9c25c20703161537i261d4c08nb310d5ad257001b@mail.gmail.com> <5.1.1.6.0.20070320094846.04d9aa20@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070320132148.04dc1eb0@sparrow.telecommunity.com> At 04:47 PM 3/20/2007 +0100, Ronald Oussoren wrote: >On 20 Mar, 2007, at 15:54, Phillip J. Eby wrote: > >>At 09:24 AM 3/20/2007 +0100, Ronald Oussoren wrote: >>>I don't agree. "all_ext=True" is won't do the right thing in a >>>significant subset of filenames >> >>Yes, that's understood. The problem is that splitext() in general >>"won't do the right thing", for many definitions of "the right >>thing", unless you're applying it to a fairly constrained range of >>filenames, or unless you add other code. This won't change, unless >>we get rid of splitext() altogether. > >I know that, I actually read most of the messages in this thread. The >reason I'm pointing this out for the 'all_ext=True' case is that >adding this flag could give naive users even more reason to believe >that splitext will magicly do the right thing. Well, that's where we need to shore up the documentation, which needs to point out the folly of expecting DWIM. We should give some examples of where splitext() will *not* DWIM. >>If you're trying to match an archive extension, for example, you'll >>probably need to loop on repeated splitext() calls until you find >>an extension that matches. One benefit of using both the new >>keyword arguments together is that it allows you to make your loop >>proceed from longest match to shortest, so that if you are matching >>product-X.Y.Z.tar.gz, you're going to go through >>matching .Y.Z.tar.gz, then .Z.tar.gz, then .tar.gz. > >I don't know if this is worth the additional API complexity. >Especially given the inherit problems of a splitext function. > >>>The ignore_leading_dot argument also doesn't buy you anything that >>>can't >>>trivially be implemented in other ways. >> >>I don't understand. Example? > >You conveniently ignored my other arguments ;-). > >Given a splitext that ignores leading dot's the following function >doesn't: > # from os.path import * > def splitext2(path): > dn = dirname(path) > bn, ext = splitext(basename(path)) > if bn.startswith('.') and ext == '': > return dn, bn + ext > else: > return join(dn, bn), ext > >I'd say that's a trivial function. > >What I don't understand is why 'ignore_leading_dot==False' is >considered to be a valid usecase at all, except for the fact that >os.path.splitext did this until py2.5. I'm definitely in the camp >that considers '.profile' not to have an extension. Okay, the part I'm confused about is what's your position on what should be *done* about this. Are you favoring no change? Deprecating it and ripping it out? Or what? From gustavo at niemeyer.net Tue Mar 20 19:33:04 2007 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Tue, 20 Mar 2007 15:33:04 -0300 Subject: [Python-Dev] Amusing fan mail I got In-Reply-To: References: Message-ID: <20070320183304.GA14271@niemeyer.net> > i thought about this thing. (...) +1! -- Gustavo Niemeyer http://niemeyer.net From ronaldoussoren at mac.com Tue Mar 20 19:39:57 2007 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 20 Mar 2007 19:39:57 +0100 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <5.1.1.6.0.20070320132148.04dc1eb0@sparrow.telecommunity.com> References: <5.1.1.6.0.20070320094846.04d9aa20@sparrow.telecommunity.com> <7a9c25c20703161537i261d4c08nb310d5ad257001b@mail.gmail.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <5.1.1.6.0.20070316100847.0289ba88@sparrow.telecommunity.com> <45FAC92D.3090307@gmail.com> <7a9c25c20703161537i261d4c08nb310d5ad257001b@mail.gmail.com> <5.1.1.6.0.20070320094846.04d9aa20@sparrow.telecommunity.com> <5.1.1.6.0.20070320132148.04dc1eb0@sparrow.telecommunity.com> Message-ID: On 20 Mar, 2007, at 19:24, Phillip J. Eby wrote: >> >> What I don't understand is why 'ignore_leading_dot==False' is >> considered to be a valid usecase at all, except for the fact that >> os.path.splitext did this until py2.5. I'm definitely in the camp >> that considers '.profile' not to have an extension. > > > Okay, the part I'm confused about is what's your position on what > should be *done* about this. Are you favoring no change? > Deprecating it and ripping it out? Or what? > os.path.splitext works fine for what it is supposed to do, even though there currently is some confusion on what that is. IMHO the change Martin checked in into 2.6 was a good one and makes that API as good as it can get without unduly cluttering the API. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3562 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20070320/94e076cf/attachment-0001.bin From pje at telecommunity.com Tue Mar 20 19:52:05 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 20 Mar 2007 13:52:05 -0500 Subject: [Python-Dev] Proposal to revert r54204 (splitext change) In-Reply-To: <460006CD.7080405@v.loewis.de> References: <0BCA1F54-8373-4C5C-BD81-0E6F0BDFCC6D@mac.com> <7a9c25c20703161537i261d4c08nb310d5ad257001b@mail.gmail.com> <5.1.1.6.0.20070314114734.0446f778@sparrow.telecommunity.com> <45F8EB7F.6070908@v.loewis.de> <5.1.1.6.0.20070315105608.02ba5f58@sparrow.telecommunity.com> <45F991FB.20906@holdenweb.com> <45F9957A.1080204@v.loewis.de> <7a9c25c20703151428w3a2682a0i67980d808736a41b@mail.gmail.com> <5.1.1.6.0.20070315203431.03c82778@sparrow.telecommunity.com> <5.1.1.6.0.20070316100847.0289ba88@sparrow.telecommunity.com> <45FAC92D.3090307@gmail.com> <7a9c25c20703161537i261d4c08nb310d5ad257001b@mail.gmail.com> <5.1.1.6.0.20070320094846.04d9aa20@sparrow.telecommunity.com> <0BCA1F54-8373-4C5C-BD81-0E6F0BDFCC6D@mac.com> Message-ID: <5.1.1.6.0.20070320132504.029ef538@sparrow.telecommunity.com> At 05:07 PM 3/20/2007 +0100, Martin v. L?wis wrote: >Ronald Oussoren schrieb: > > What I don't understand is why 'ignore_leading_dot==False' is considered > > to be a valid usecase at all, except for the fact that os.path.splitext > > did this until py2.5. I'm definitely in the camp that considers > > '.profile' not to have an extension. > >That is precisely the core of the discussion. It's not that >ignore_leading_dots=False is considered useful, in the call (except >for a few people that claim that splitext('.txt') ought to give >'','.txt') Actually, *this* is precisely the problem: arguing that the opinion of these "few people" is irrelevant, because a few *other* people think they're wrong to find that behavior useful. I'm able to see that considering '.profile' to not have an extension is a *reasonable* position to take, and that doing it from the start *might* have been a good idea. What I disagree with is punishing people who considered the opposite approach equally valid, and took the documentation and tests at their word. Breaking their code without warning would be rude enough, but unfortunately it affects not only the person who directly uses splitext(), but everyone who uses any library, tool, or application that relies on the current behavior. The very fact that you keep treating the current behavior as *not* useful is the very core of our disagreement. Indeed, it seems to me quite disrespectful that you will not take anyone at their word that they do indeed expect, desire, and *value* the existing behavior, and wish to continue to have access to it in future versions of Python. Suppose that the behavior had been the other way around to begin with, and Windows users started filing bugs about it, because it disagrees with Windows Explorer's interpretation of the extension? Would you simply change the Unix-oriented behavior because it's clearly a "bug"? If not, then what is your rationale for changing it the other way? Make no mistake: both behaviors are desirable, for different reasons. And both interpretations merely reflect platform-specific shell policies, so neither is any more "true" or "correct" in some absolute sense. (If anything, Windows at least derives from an operating system that actually *has* extensions as part of its filesystem, whereas Unix does not.) The people who would like to keep the old behavior have all, to my recollection, acknowledged that other behaviors are desirable. Why won't the people who want to *change* the behavior acknowledge the same thing? From facundo at taniquetil.com.ar Tue Mar 20 20:24:30 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Tue, 20 Mar 2007 19:24:30 +0000 (UTC) Subject: [Python-Dev] Adding timeout to socket.py and httplib.py References: <4a951aa00703200708t771ff2d3v4dbb7793acc1a478@mail.gmail.com> <4a951aa00703200853r2486d2c0qf870e075790c025b@mail.gmail.com> <4a951aa00703201017x3d7ed9b4jb209aedf933158f8@mail.gmail.com> Message-ID: Alan Kennedy wrote: > [Facundo] >> But, I recognize that maybe it's [connect] not the best name. What about >> "create_connection"? > > I have no strong feelings about it, other than to say it should not be > "connect". How about Ok. "create_connection", then. > Ah, but it's too late by the time the socket.connect call returns: the > timeout/blocking behaviour of the socket.connect call is the very > thing we're trying to control. It's not the very thing, just one of them... whatever, you have a point. > Whereas with the slightly modified API I suggested earlier, it simply becomes I'm OK with that API, except that you're losing position parameters. It's OK to *always* need to put the "timeout="? The problem here is that I used None to check if you passed a parameter or not, an idiom well stablished in Python, but in this very case None has a meaning for itself. I'm +0 on having the obligation to a named parameter here. So, I have two modifications to make to the patch: - change the name to "create_connection" - make timeout obligatory named Is everybody ok with this? Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From amk at amk.ca Tue Mar 20 20:58:22 2007 From: amk at amk.ca (A.M. Kuchling) Date: Tue, 20 Mar 2007 15:58:22 -0400 Subject: [Python-Dev] r54457 - python/trunk/Doc/whatsnew/whatsnew26.tex In-Reply-To: <20070320050824.7EFC71E4006@bag.python.org> References: <20070320050824.7EFC71E4006@bag.python.org> Message-ID: <20070320195822.GA19284@localhost.localdomain> On Tue, Mar 20, 2007 at 06:08:24AM +0100, neal.norwitz wrote: > Author: neal.norwitz > Date: Tue Mar 20 06:08:23 2007 > New Revision: 54457 > > +% Should there be a new section here for 3k migration? > +% Or perhaps a more general section describing module changes/deprecation? > +% sets module deprecated This is an interesting question: should the "What's New" talk about Python 3000? My initial tentative reaction is 'no', because a Py3K section would need to continue to be updated as Python 3000's definition shifts. Once previous Python 2.x versions were released, "What's New" documents became very static, the only changes being typo fixes and small corrections and clarifications. Having to continually update the Py3K section would be annoying, and argues for a separate document that isn't necessarily tied to Python 2.x releases. On the other hand, it would be nice to warn users away from idioms that will break in Py3K, and the "What's New" is a natural place to do it. Hm. But the 'porting' section already talks about features that are being deprecated; is that enough? Thoughts? --amk From steven.bethard at gmail.com Tue Mar 20 21:17:12 2007 From: steven.bethard at gmail.com (Steven Bethard) Date: Tue, 20 Mar 2007 14:17:12 -0600 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: References: <4a951aa00703200708t771ff2d3v4dbb7793acc1a478@mail.gmail.com> <4a951aa00703200853r2486d2c0qf870e075790c025b@mail.gmail.com> <4a951aa00703201017x3d7ed9b4jb209aedf933158f8@mail.gmail.com> Message-ID: On 3/20/07, Facundo Batista wrote: > So, I have two modifications to make to the patch: > > - change the name to "create_connection" > - make timeout obligatory named > > Is everybody ok with this? FWLIW, +1. It was not immediately apparent to me that the third argument in:: newsock = socket.create_connection(HOST, PORT, None) is supposed to be a timeout. The modified version:: newsock = socket.create_connection(HOST, PORT, timeout=None) is much clearer to me. STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From facundo at taniquetil.com.ar Tue Mar 20 21:01:38 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Tue, 20 Mar 2007 20:01:38 +0000 (UTC) Subject: [Python-Dev] Patchs and bugs resume Message-ID: People: At the beginning of March, there was a thread in this list about patchs and bugs that teorically weren't checked out. >From that discussion, I asked myself: "How can I know the temporal location of a patch/bug?". Are there a lot of old patchs/bugs? Those that are old, don't have any update or there're a big discussion with each one? Are they abandoned? To help me with this analisys, I made a tool that taking information from SourceForge it creates a resume table, for the patchs... http://www.taniquetil.com.ar/facundo/py_patchs.html ...and the bugs: http://www.taniquetil.com.ar/facundo/py_bugs.html My idea is to update them periodically (something like each day, at the end of the html you have the update date and time). Enjoy it. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From jcarlson at uci.edu Tue Mar 20 21:40:24 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 20 Mar 2007 13:40:24 -0700 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: References: Message-ID: <20070320133752.FC64.JCARLSON@uci.edu> "Steven Bethard" wrote: > On 3/20/07, Facundo Batista wrote: > > So, I have two modifications to make to the patch: > > > > - change the name to "create_connection" > > - make timeout obligatory named > > > > Is everybody ok with this? > > FWLIW, +1. It was not immediately apparent to me that the third argument in:: > > newsock = socket.create_connection(HOST, PORT, None) > > is supposed to be a timeout. The modified version:: > > newsock = socket.create_connection(HOST, PORT, timeout=None) > > is much clearer to me. Agreed, though implementation-wise, there is another technique for determining whether the use provided an argument to timeout or not... sentinel = object() def connect(HOST, PORT, timeout=sentinel): ... if timeout is not sentinel: sock.settimeout(timeout) ... A keyword argument via **kwargs is also fine. I have no preference. - Josiah From facundo at taniquetil.com.ar Tue Mar 20 21:37:03 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Tue, 20 Mar 2007 20:37:03 +0000 (UTC) Subject: [Python-Dev] Adding timeout to socket.py and httplib.py References: <4a951aa00703200708t771ff2d3v4dbb7793acc1a478@mail.gmail.com> <4a951aa00703200853r2486d2c0qf870e075790c025b@mail.gmail.com> <4a951aa00703201017x3d7ed9b4jb209aedf933158f8@mail.gmail.com> Message-ID: Steven Bethard wrote: > is supposed to be a timeout. The modified version:: > > newsock = socket.create_connection(HOST, PORT, timeout=None) Warning. The correct function signature is create_connection(address[, timeout=None]) where address is a tuple (HOST, PORT). BTW, how can I indicate in the tex file (docs), that the parameter, if present, is mandatorily named? Thanks! -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From facundo at taniquetil.com.ar Tue Mar 20 21:41:05 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Tue, 20 Mar 2007 20:41:05 +0000 (UTC) Subject: [Python-Dev] Adding timeout to socket.py and httplib.py References: <20070320133752.FC64.JCARLSON@uci.edu> Message-ID: Josiah Carlson wrote: > sentinel = object() > > def connect(HOST, PORT, timeout=sentinel): > ... > if timeout is not sentinel: > sock.settimeout(timeout) > ... > > A keyword argument via **kwargs is also fine. I have no preference. I do. The way you showed here, I'm not restricting user options. I think this is better. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From jcarlson at uci.edu Tue Mar 20 22:16:57 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 20 Mar 2007 14:16:57 -0700 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: References: <20070320133752.FC64.JCARLSON@uci.edu> Message-ID: <20070320141425.FC68.JCARLSON@uci.edu> Facundo Batista wrote: > Josiah Carlson wrote: > > sentinel = object() > > > > def connect(HOST, PORT, timeout=sentinel): > > ... > > if timeout is not sentinel: > > sock.settimeout(timeout) > > ... > > > > A keyword argument via **kwargs is also fine. I have no preference. > > I do. The way you showed here, I'm not restricting user options. I think > this is better. But the kwargs doesn't restrict options either... def connect(address, **kwargs): ... if 'timeout' in kwargs: sock.settimeout(kwargs['timeout']) ... With that method you can include timeout=None, and it also doesn't restrict what the user could pass as a value to timeout. It requires that you pass timeout explicitly, but that's a (relatively inconsequential) API decision. - Josiah From facundo at taniquetil.com.ar Tue Mar 20 22:20:08 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Tue, 20 Mar 2007 21:20:08 +0000 (UTC) Subject: [Python-Dev] Adding timeout to socket.py and httplib.py References: <20070320133752.FC64.JCARLSON@uci.edu> <20070320141425.FC68.JCARLSON@uci.edu> Message-ID: Josiah Carlson wrote: > restrict what the user could pass as a value to timeout. It requires > that you pass timeout explicitly, but that's a (relatively > inconsequential) API decision. This is exactly the point. It's an API decision, that you must communicate to the user, he/she must read it and remember it. Letting "timeout" be positional or named, it's just less error prone. So, if I can make it this way, it's what I prefer, :) Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From python-dev at alan.kennedy.name Tue Mar 20 22:43:25 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Tue, 20 Mar 2007 21:43:25 +0000 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: References: <4a951aa00703200708t771ff2d3v4dbb7793acc1a478@mail.gmail.com> <4a951aa00703200853r2486d2c0qf870e075790c025b@mail.gmail.com> <4a951aa00703201017x3d7ed9b4jb209aedf933158f8@mail.gmail.com> Message-ID: <4a951aa00703201443u8a13656wa2c2ef4a338dae0f@mail.gmail.com> [Facundo] > So, I have two modifications to make to the patch: > > - change the name to "create_connection" > - make timeout obligatory named I was going to suggest a third change: for orthogonality with the API for socket objects, add a blocking parameter as well, i.e. def create_connection(address, timeout=sentinel, blocking=sentinel): [snip] if timeout != sentinel: new_socket.settimeout(timeout) if blocking != sentinel: new_socket.setblocking(blocking) [snip] but that may be taking it too far. But there is still an issue remaining, relating to non-blocking IO. With or without a blocking parameter, the user can still set non-blocking behaviour on a socket by setting a timeout of 0. The following snippet illustrates the issue. #-=-=-=-=-=-=-=-=-=-=-=-=-= import socket s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.settimeout(0) s.connect( ("localhost", 80) ) #-=-=-=-=-=-=-=-=-=-=-=-=-= If you run this, it is very likely to generate an exception, but not guaranteed: you may have to run it a few times. Or try a host that is slower to respond. The problem is now that the connect call is now a non-blocking connect, which means that it may throw a socket.error, even after a successful connect, as follows socket.error: (10035, 'The socket operation could not complete without blocking') The standard mechanism in C for doing a non-blocking connect is to issue the connect call, and check the return value for a non-zero error code. If this error code is errno.EAGAIN (code 10035), then the call succeeded, but you should check back later for completion of the operation. It was for this reason that the connect_ex method was introduced to python socket objects. Instead of raising an exception, it directly returns the error code from the socket operation, so that it can be checked, as in C. So in the case of the new create_connection function, either A: The user should be prepared to handle an exception if they use a zero timeout (i.e. set non-blocking mode) or B: Detect the non-blocking case inside the function implementation and return the value of the connect_ex method instead of the connect method, as would be standard in a non-blocking app. This could be implemented as follows def create_connection(address, timeout=sentinel): [snip] if timeout != sentinel: new_socket.settimeout(timeout) if new_socket.gettimeout() == 0: result = new_socket.connect_ex(address) else: new_socket.connect(address) result = new_socket [snip] I know that this makes it all more complex, and I'm *not* saying the new function should be modified to include these concerns. The new function is designed to address straightforward usability cases, so it's perhaps appropriate that the API be restricted to those cases, i.e. to supporting timeout values > 0. Perhaps this limit could be coded into the function? Also, people who want explicitly do non-blocking socket IO will likely use the socket API directly, so it may not be worth supporting that use in this function. Regards, Alan. From greg.ewing at canterbury.ac.nz Tue Mar 20 23:02:06 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 21 Mar 2007 10:02:06 +1200 Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: <20070320070010.GA13982@electricrain.com> References: <45FC7EA3.4070808@canterbury.ac.nz> <45FDD76B.7080800@canterbury.ac.nz> <20070320070010.GA13982@electricrain.com> Message-ID: <460059DE.60307@canterbury.ac.nz> Gregory P. Smith wrote: > I prefer the default of "clean up after itself" as is to avoid leaving > temporary file turds on systems caused by us lazy programmers. Maybe instead of an option there should be a separate function with a different name, such as NewUniqueFile. For the use cases I have in mind, the file isn't really "temporary" at all. Or rather, only the name is temporary, as it ultimately gets renamed. -- Greg From jcarlson at uci.edu Tue Mar 20 23:11:02 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 20 Mar 2007 15:11:02 -0700 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: <4a951aa00703201443u8a13656wa2c2ef4a338dae0f@mail.gmail.com> References: <4a951aa00703201443u8a13656wa2c2ef4a338dae0f@mail.gmail.com> Message-ID: <20070320150742.FC6B.JCARLSON@uci.edu> "Alan Kennedy" wrote: [snip] > def create_connection(address, timeout=sentinel): > [snip] > if timeout != sentinel: > new_socket.settimeout(timeout) > if new_socket.gettimeout() == 0: > result = new_socket.connect_ex(address) > else: > new_socket.connect(address) > result = new_socket > [snip] > > I know that this makes it all more complex, and I'm *not* saying the > new function should be modified to include these concerns. [snip] But now the result could be either an error code OR a socket. One of the reasons to provide a timeout for the create_connection call, if I understand correctly, is to handle cases for which you don't get a response back in sufficient time. If the user provides zero as a timeout, then they may very well get an exception, which is what they should expect. Then again, even with an arbitrary timeout, an exception is possible (especially if a host is down, etc.), and hiding the exceptional condition (didn't connect in the allotted time) is a bad thing. - Josiah From brett at python.org Tue Mar 20 23:10:25 2007 From: brett at python.org (Brett Cannon) Date: Tue, 20 Mar 2007 15:10:25 -0700 Subject: [Python-Dev] Patchs and bugs resume In-Reply-To: References: Message-ID: On 3/20/07, Facundo Batista wrote: > People: > > At the beginning of March, there was a thread in this list about patchs > and bugs that teorically weren't checked out. > > >From that discussion, I asked myself: "How can I know the temporal > location of a patch/bug?". Are there a lot of old patchs/bugs? Those > that are old, don't have any update or there're a big discussion with > each one? Are they abandoned? > > To help me with this analisys, I made a tool that taking information > from SourceForge it creates a resume table, for the patchs... > > http://www.taniquetil.com.ar/facundo/py_patchs.html > > ...and the bugs: > > http://www.taniquetil.com.ar/facundo/py_bugs.html > > My idea is to update them periodically (something like each day, at > the end of the html you have the update date and time). > That's some interesting stuff. Took me a second to realize that the temporal column's total length is the time range from the opening of the oldest bug to the latest comment made on any bug and that the blue bar is where within that time frame the bug was opened and the last comment was made on that bug. But still interesting! Led to me closing a bug and a patch that were old and had not been touched in ages. Hopefully you will be able to easily port this over to the new tracker once it's up (that should happen 2-4 weeks after 2.5.1 is released). -Brett From python-dev at alan.kennedy.name Tue Mar 20 23:21:30 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Tue, 20 Mar 2007 22:21:30 +0000 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: <20070320150742.FC6B.JCARLSON@uci.edu> References: <4a951aa00703201443u8a13656wa2c2ef4a338dae0f@mail.gmail.com> <20070320150742.FC6B.JCARLSON@uci.edu> Message-ID: <4a951aa00703201521m7051d90fgd4f3493c5ad5a2f0@mail.gmail.com> [Josiah] > But now the result could be either an error code OR a socket. One of > the reasons to provide a timeout for the create_connection call, if I > understand correctly, is to handle cases for which you don't get a > response back in sufficient time. AFAICT, that's the only reason. It's not to handle blocking sockets, that's the default operation of sockets. And it's not to handle non-blocking sockets either. >If the user provides zero as a > timeout, then they may very well get an exception, which is what they > should expect. Yes, they should expect it. And they would handle it like this try: new_socket = socket.create_connection(address, 0): except socket.error: import errno: if errno.errno == 10035 # or relevant platform specific symbolic constant # socket is still connecting else: # there was a real socket error > Then again, even with an arbitrary timeout, an exception > is possible (especially if a host is down, etc.), and hiding the > exceptional condition (didn't connect in the allotted time) is a bad > thing. See above. Regards, Alan. From python-dev at alan.kennedy.name Tue Mar 20 23:25:57 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Tue, 20 Mar 2007 22:25:57 +0000 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: References: <20070320133752.FC64.JCARLSON@uci.edu> <20070320141425.FC68.JCARLSON@uci.edu> Message-ID: <4a951aa00703201525m77015cc6nf9714cdd9fc28f9c@mail.gmail.com> [Facundo] > Letting "timeout" be positional or named, it's just less error prone. > So, if I can make it this way, it's what I prefer, :) So, if I want a timeout of, say, 80 seconds, I issue a call like this new_socket = socket.create_connection(address, 80) So is that address = host, port = 80? Or is it address = (host, port), timeout=80? I know *we* know what it is, but will the user? I prefer explicit naming of the timeout parameter. Regards, Alan. From steve at holdenweb.com Tue Mar 20 23:29:50 2007 From: steve at holdenweb.com (Steve Holden) Date: Tue, 20 Mar 2007 18:29:50 -0400 Subject: [Python-Dev] r54457 - python/trunk/Doc/whatsnew/whatsnew26.tex In-Reply-To: <20070320195822.GA19284@localhost.localdomain> References: <20070320050824.7EFC71E4006@bag.python.org> <20070320195822.GA19284@localhost.localdomain> Message-ID: A.M. Kuchling wrote: > On Tue, Mar 20, 2007 at 06:08:24AM +0100, neal.norwitz wrote: >> Author: neal.norwitz >> Date: Tue Mar 20 06:08:23 2007 >> New Revision: 54457 >> >> +% Should there be a new section here for 3k migration? >> +% Or perhaps a more general section describing module changes/deprecation? >> +% sets module deprecated > > This is an interesting question: should the "What's New" talk about > Python 3000? > > My initial tentative reaction is 'no', because a Py3K section would > need to continue to be updated as Python 3000's definition shifts. > Once previous Python 2.x versions were released, "What's New" > documents became very static, the only changes being typo fixes and > small corrections and clarifications. Having to continually update > the Py3K section would be annoying, and argues for a separate document > that isn't necessarily tied to Python 2.x releases. > > On the other hand, it would be nice to warn users away from idioms > that will break in Py3K, and the "What's New" is a natural place to do > it. Hm. But the 'porting' section already talks about features that > are being deprecated; is that enough? > > Thoughts? > Clearly a need for a new document. We don't want to startle people who don't want to get involved in version wars (probably 98.5% of all users aren't even aware of "What's New", since they use a Python installed by someone else, typically their computer vendor or Linux/UNIX distro team). At the same time we should flag the fact that upcoming changes are indeed in the works. Possibly a "3.0 Design Snapshot" whose title makes it clear this is a moving target and, among other things, whose content points the user at a definitive (set of) URL(s) for the latest information. Putting this information in "What's New" (except possibly for mentioning the creation of this new document) would create unnecessary FUD. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Recent Ramblings http://holdenweb.blogspot.com From greg.ewing at canterbury.ac.nz Tue Mar 20 23:51:00 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 21 Mar 2007 10:51:00 +1200 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: <4a951aa00703201443u8a13656wa2c2ef4a338dae0f@mail.gmail.com> References: <4a951aa00703200708t771ff2d3v4dbb7793acc1a478@mail.gmail.com> <4a951aa00703200853r2486d2c0qf870e075790c025b@mail.gmail.com> <4a951aa00703201017x3d7ed9b4jb209aedf933158f8@mail.gmail.com> <4a951aa00703201443u8a13656wa2c2ef4a338dae0f@mail.gmail.com> Message-ID: <46006554.4090104@canterbury.ac.nz> Alan Kennedy wrote: > The standard mechanism in C for doing a non-blocking connect is to > issue the connect call, and check the return value for a non-zero > error code. If this error code is errno.EAGAIN (code 10035), then the > call succeeded, but you should check back later for completion of the > operation. Hmmm. I think that this case probably isn't what people will have in mind when they specify a timeout for connecting. More likely they mean "If the connection couldn't be successfully established within this time, give up and let me know." So it seems to me that a return value of EAGAIN should be handled internally by re-issuing the connect call with a suitably reduced timeout value. If the timeout gets down to zero without a successful result, throw an exception. An application that wants to do fully asynchronous connects will have to take quite a different approach, so there should probably be a different API for this. -- Greg From jcarlson at uci.edu Wed Mar 21 00:10:23 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 20 Mar 2007 16:10:23 -0700 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: <4a951aa00703201525m77015cc6nf9714cdd9fc28f9c@mail.gmail.com> References: <4a951aa00703201525m77015cc6nf9714cdd9fc28f9c@mail.gmail.com> Message-ID: <20070320160541.FC6E.JCARLSON@uci.edu> "Alan Kennedy" wrote: > > [Facundo] > > Letting "timeout" be positional or named, it's just less error prone. > > So, if I can make it this way, it's what I prefer, :) > > So, if I want a timeout of, say, 80 seconds, I issue a call like this > > new_socket = socket.create_connection(address, 80) > > So is that address = host, port = 80? > > Or is it address = (host, port), timeout=80? > > I know *we* know what it is, but will the user? > > I prefer explicit naming of the timeout parameter. Error-wise, I agree that it would be better to pass timeout explicitly with a keyword, but generally users will notice their mistake if they try to do create_connection(host, port) by ValueError("tuple expected as first argument, got str instead") Is it better than TypeError("create_connection takes 1 argument (2 given)") ? - Josiah From jcarlson at uci.edu Wed Mar 21 00:20:10 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 20 Mar 2007 16:20:10 -0700 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: <46006554.4090104@canterbury.ac.nz> References: <4a951aa00703201443u8a13656wa2c2ef4a338dae0f@mail.gmail.com> <46006554.4090104@canterbury.ac.nz> Message-ID: <20070320161659.FC77.JCARLSON@uci.edu> Greg Ewing wrote: > Alan Kennedy wrote: > > The standard mechanism in C for doing a non-blocking connect is to > > issue the connect call, and check the return value for a non-zero > > error code. If this error code is errno.EAGAIN (code 10035), then the > > call succeeded, but you should check back later for completion of the > > operation. > > An application that wants to do fully asynchronous connects > will have to take quite a different approach, so there > should probably be a different API for this. *cough* asyncore or twisted *cough* Sorry, what were we talking about again? Oh yeah, timeouts. From what I understand of connect and connect_ex, if a socket has a specified timeout, it is supposed to "try" (it only attempts once, and waits for a response) until it either fails (because the other end won't accept), or it times out. Either case is perfectly fine, and I don't really see the point of retrying (in socket.create_connection). - Josiah From python-dev at alan.kennedy.name Wed Mar 21 01:27:55 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Wed, 21 Mar 2007 00:27:55 +0000 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: <20070320160541.FC6E.JCARLSON@uci.edu> References: <4a951aa00703201525m77015cc6nf9714cdd9fc28f9c@mail.gmail.com> <20070320160541.FC6E.JCARLSON@uci.edu> Message-ID: <4a951aa00703201727s101420a4q67aee1deac33e06f@mail.gmail.com> [Josiah] > Error-wise, I agree that it would be better to pass timeout explicitly > with a keyword, but generally users will notice their mistake if they > try to do create_connection(host, port) by ValueError("tuple expected as > first argument, got str instead") Is it better than > TypeError("create_connection takes 1 argument (2 given)") ? Yes, it is better. Currently, the socket.connect() method takes a tuple, and fails with the following exception if 2 separate parameters are passed TypeError: connect() takes exactly one argument (2 given) Which is fine because the function does take exactly one argument. But we're discussing a function with an optional timeout parameter, so that TypeError wouldn't be raised if I called create_connection("localhost", 80). The patch as it currently is, if I am reading it right, would raise one of the following if a string was passed as the address argument, depending on the length of the string. ValueError: need more than 1 value to unpack # len(address) == 1 ValueError: too many values to unpack # len(address) > 2 since it extracts the host and port like so host, port = address Which succeeds, somewhat surprisingly, if a string is passed that is 2 characters long. I was a little surprised to find that this didn't give rise to an error: host, port = "ab". So with a two character hostname, the second letter would be unpacked as a port number. And the function would then fail with the following exception when it reaches the getaddrinfo ("a", "b", 0, SOCK_STREAM) call. socket.gaierror: (10109, 'getaddrinfo failed') I suggest updating the patch to - Explicitly check that the address passed is a tuple of (string, integer) - To raise an exception explaining the parameter expectation when it is not met - To require that the user explicitly name the timeout parameter Regards, Alan. From facundo at taniquetil.com.ar Wed Mar 21 04:14:43 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Wed, 21 Mar 2007 03:14:43 +0000 (UTC) Subject: [Python-Dev] Patchs and bugs resume References: Message-ID: Brett Cannon wrote: > That's some interesting stuff. Took me a second to realize that the > temporal column's total length is the time range from the opening of > the oldest bug to the latest comment made on any bug and that the blue > bar is where within that time frame the bug was opened and the last > comment was made on that bug. Mmmmm, you're right, I should have explained it somewhere... > old and had not been touched in ages. Hopefully you will be able to > easily port this over to the new tracker once it's up (that should > happen 2-4 weeks after 2.5.1 is released). You can be sure I'll port it... -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From facundo at taniquetil.com.ar Wed Mar 21 04:20:42 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Wed, 21 Mar 2007 03:20:42 +0000 (UTC) Subject: [Python-Dev] Adding timeout to socket.py and httplib.py References: <20070320133752.FC64.JCARLSON@uci.edu> <20070320141425.FC68.JCARLSON@uci.edu> <4a951aa00703201525m77015cc6nf9714cdd9fc28f9c@mail.gmail.com> Message-ID: Alan Kennedy wrote: > So is that address = host, port = 80? > > Or is it address = (host, port), timeout=80? The latter, as is in the rest of Python... See your point, you say it's less error prone to make timeout mandatory. I really don't care, so I'll take your advice... -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From facundo at taniquetil.com.ar Wed Mar 21 04:27:23 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Wed, 21 Mar 2007 03:27:23 +0000 (UTC) Subject: [Python-Dev] Adding timeout to socket.py and httplib.py References: <4a951aa00703201525m77015cc6nf9714cdd9fc28f9c@mail.gmail.com> <20070320160541.FC6E.JCARLSON@uci.edu> <4a951aa00703201727s101420a4q67aee1deac33e06f@mail.gmail.com> Message-ID: Alan Kennedy wrote: > - Explicitly check that the address passed is a tuple of (string, integer) It's more probable that a use pass a list of two values, that a host of two letters as you suggested above... > - To raise an exception explaining the parameter expectation when it is not met Won't be necessary if we take into account the explicit timeout parameter... > - To require that the user explicitly name the timeout parameter I already agreed on this, :) So, as a resume: - I'll make "timeout" mandatory - The function signature will be: create_connection(address[, timeout]) See that timeout doesn't have a default value, if you include it, it'll set the socket timeout, if you don't, the defaultimeout will work. The address is a tuple (host, port), as usual In the code, I'll just make "host, port = address", I don't think it will be a problem at all. Remember that this function primary use is for higher level libraries, and that "address" in socket enviroment is always, always, (host, port). Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From anthony at interlink.com.au Wed Mar 21 04:41:00 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed, 21 Mar 2007 14:41:00 +1100 Subject: [Python-Dev] 2.5.1, buildbots and my availability Message-ID: <200703211441.00661.anthony@interlink.com.au> I'm moving house today and tomorrow, and don't expect to have internet access connected up at home til sometime next week. In the meantime, if there's urgent 2.5.1 related issues, bear with me, as I'll only be on email during the working day. cc Neal (hi Neal :) is the best bet. Also, the cygwin and ubuntu/icc buildbots will be offline in the interim - I'll be unplugging them this afternoon to move them. I can still cut the 2.5.1 release - can always do it during the day, even if the stupid ISP takes longer than I expect to connect up the net. From jseutter at gmail.com Wed Mar 21 06:37:12 2007 From: jseutter at gmail.com (Jerry Seutter) Date: Tue, 20 Mar 2007 23:37:12 -0600 Subject: [Python-Dev] test_expat.py and unittest Message-ID: <2c8d48d70703202237w3864e4c7q40a6248e5ff6a742@mail.gmail.com> Hi, I have been working on converting some of the older test files to use unittest. test_expat.py has a section like: class Outputter: def StartElementHandler(self, name, attrs): print 'Start element:\n\t', repr(name), sortdict(attrs) def EndElementHandler(self, name): print 'End element:\n\t', repr(name) def CharacterDataHandler(self, data): data = data.strip() if data: print 'Character data:' print '\t', repr(data) def ProcessingInstructionHandler(self, target, data): print 'PI:\n\t', repr(target), repr(data) def StartNamespaceDeclHandler(self, prefix, uri): print 'NS decl:\n\t', repr(prefix), repr(uri) ...where it defines a set of handlers for an xml document that prints the output to stdout. The test then parses an xml document using this class and the stdout output is compared to a file. There are several lines of text on stdout to be compared: PI: 'xml-stylesheet' 'href="stylesheet.css"' Comment: ' comment data ' Notation declared: ('notation', None, 'notation.jpeg', None) Unparsed entity decl: ('unparsed_entity', None, 'entity.file', None, 'notation') Start element: 'root' {'attr1': 'value1', 'attr2': 'value2\xe1\xbd\x80'} NS decl: 'myns' 'http://www.python.org/namespace' Start element: 'http://www.python.org/namespace!subelement' {} Character data: 'Contents of subelements' End element: 'http://www.python.org/namespace!subelement' End of NS decl: 'myns' Start element: 'sub2' {} Start of CDATA section Character data: 'contents of CDATA section' End of CDATA section End element: 'sub2' External entity ref: (None, 'entity.file', None) End element: 'root' unittest does not seem well suited to this type of testing, because the stdout output comparison is testing many different pieces of functionality. Some subset of it is probably even important. To do this same testing with unittest would require many lines of self.assertEquals(expected_string, string_from_stdout). Does anyone have ideas on how this can be tested in a manner that is not as brittle as the stdout tests, but doesn't require writing significantly more test code? Or if not, is converting this file to use unittest a bad idea? Jerry Seutter -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070320/26f04867/attachment.html From guido at python.org Wed Mar 21 06:43:34 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Mar 2007 22:43:34 -0700 Subject: [Python-Dev] test_expat.py and unittest In-Reply-To: <2c8d48d70703202237w3864e4c7q40a6248e5ff6a742@mail.gmail.com> References: <2c8d48d70703202237w3864e4c7q40a6248e5ff6a742@mail.gmail.com> Message-ID: You could use doctest, which is an accepted testing tool that lets you compare snippets of output directly without having to check in a "golden" file. Despite what is written somewhere, it is not deprecated or discouraged. See Lib/doctest.py. You could also use some advanced testing strategy where the callback, instead of printing, append something to a list (perhaps an instance variable), and later you check that the list contains the expected sequence of values. --Guido On 3/20/07, Jerry Seutter wrote: > Hi, > > I have been working on converting some of the older test files to use > unittest. test_expat.py has a section like: > > class Outputter: > def StartElementHandler(self, name, attrs): > print 'Start element:\n\t', repr(name), sortdict(attrs) > > def EndElementHandler(self, name): > print 'End element:\n\t', repr(name) > > def CharacterDataHandler(self, data): > data = data.strip() > if data: > print 'Character data:' > print '\t', repr(data) > > def ProcessingInstructionHandler(self, target, data): > print 'PI:\n\t', repr(target), repr(data) > > def StartNamespaceDeclHandler(self, prefix, uri): > print 'NS decl:\n\t', repr(prefix), repr(uri) > > > ...where it defines a set of handlers for an xml document that prints the > output to stdout. The test then parses an xml document using this class and > the stdout output is compared to a file. There are several lines of text on > stdout to be compared: > > PI: > 'xml-stylesheet' 'href="stylesheet.css"' > Comment: > ' comment data ' > Notation declared: ('notation', None, 'notation.jpeg', None) > Unparsed entity decl: > ('unparsed_entity', None, 'entity.file', None, 'notation') > Start element: > 'root' {'attr1': 'value1', 'attr2': 'value2\xe1\xbd\x80'} > NS decl: > 'myns' 'http://www.python.org/namespace' > Start element: > 'http://www.python.org/namespace!subelement ' {} > Character data: > 'Contents of subelements' > End element: > 'http://www.python.org/namespace!subelement' > End of NS decl: > 'myns' > Start element: > 'sub2' {} > Start of CDATA section > Character data: > 'contents of CDATA section' > End of CDATA section > End element: > 'sub2' > External entity ref: (None, 'entity.file', None) > End element: > 'root' > > > unittest does not seem well suited to this type of testing, because the > stdout output comparison is testing many different pieces of functionality. > Some subset of it is probably even important. To do this same testing with > unittest would require many lines of self.assertEquals(expected_string, > string_from_stdout). > > Does anyone have ideas on how this can be tested in a manner that is not as > brittle as the stdout tests, but doesn't require writing significantly more > test code? Or if not, is converting this file to use unittest a bad idea? > > Jerry Seutter > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Tue Mar 20 23:18:03 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 21 Mar 2007 10:18:03 +1200 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: <4a951aa00703200853r2486d2c0qf870e075790c025b@mail.gmail.com> References: <4a951aa00703200708t771ff2d3v4dbb7793acc1a478@mail.gmail.com> <4a951aa00703200853r2486d2c0qf870e075790c025b@mail.gmail.com> Message-ID: <46005D9B.1070004@canterbury.ac.nz> Alan Kennedy wrote: > def connect(address, **kwargs): > [snip] > if kwargs.has_key('timeout'): > sock.settimeout(kwargs['timeout']) > [snip] A problem with interfaces like this is that it makes it awkward to pass on a value that you received from higher up. An alternative would be to create and publish a different special value to mean "no timeout". -- Greg From brett at python.org Wed Mar 21 07:51:11 2007 From: brett at python.org (Brett Cannon) Date: Tue, 20 Mar 2007 23:51:11 -0700 Subject: [Python-Dev] Patchs and bugs resume In-Reply-To: References: Message-ID: On 3/20/07, Facundo Batista wrote: > Brett Cannon wrote: > > > That's some interesting stuff. Took me a second to realize that the > > temporal column's total length is the time range from the opening of > > the oldest bug to the latest comment made on any bug and that the blue > > bar is where within that time frame the bug was opened and the last > > comment was made on that bug. > > Mmmmm, you're right, I should have explained it somewhere... > > > > old and had not been touched in ages. Hopefully you will be able to > > easily port this over to the new tracker once it's up (that should > > happen 2-4 weeks after 2.5.1 is released). > > You can be sure I'll port it... > Great! It already led to me closing four bugs, another waiting for people to not speak up, and a sixth for me to fix! -Brett From martin at v.loewis.de Wed Mar 21 08:29:44 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 21 Mar 2007 08:29:44 +0100 Subject: [Python-Dev] Patchs and bugs resume In-Reply-To: References: Message-ID: <4600DEE8.60603@v.loewis.de> >> old and had not been touched in ages. Hopefully you will be able to >> easily port this over to the new tracker once it's up (that should >> happen 2-4 weeks after 2.5.1 is released). > > You can be sure I'll port it... When you do, make sure you take a look at roundup's search facilities. Roundup keeps a 'last activity' field, on which you can search and sort, and a 'creation date' field (likewise). Regards, Martin From g.brandl at gmx.net Wed Mar 21 10:11:36 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 21 Mar 2007 10:11:36 +0100 Subject: [Python-Dev] trunk and 2.5 are borken In-Reply-To: References: Message-ID: Neal Norwitz schrieb: > There are several failures with both the trunk and 2.5 branch. I have > fixed one problem that allows the tests to run on Windows. That was > fixed by reverting 54407 on the trunk. It still needs to be reverted > on the 2.5 branch. You can go back in the buildbot logs to find out > more about the problem. > > The 3 test failures on Windows are: test_urllib test_mailbox and > test_posixpath. > The first 2 fail due to 'Access denied' on @test.2. I'm guessing this > is due to the file not being closed before trying to delete it. I > don't know where this problem showed up, but it was after rev 54406 > which is when all the tests were passing on Windows (and 54407 was > backed out). I now changed the test in said revision to not use a second test directory, but only directories under TESTFN. This specific test should not cause any more problems now. Georg From python-dev at alan.kennedy.name Wed Mar 21 10:59:51 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Wed, 21 Mar 2007 09:59:51 +0000 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. Message-ID: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> [Facundo Batista] > Remember that this function primary use is for > higher level libraries Yes, I see that clearly now. But remember that by adding a new function to the socket module to support httplib et al, you are also adding a function to the socket module that will be used directly by end users. I vote to reject this patch. The underlying problem it is trying to address is that httplib.HTTPConnection objects do not support timeouts. But solving that problem by moving the functionality of the HTTPConnection.connect() method inside the socket module as a standalone function is the *wrong* solution. The proposed new function does not belong in the socket module. In contrast to all of the other socket creation and management functionality in the socket module, the new function does not handle non-blocking IO. Also, the new functions' use-case is too restricted, to client connections with a positive timeout: this functionality is trivially available using existing functionality, e.g. mysock = socket(AF_INET, SOCK_STREAM) mysocket.settimeout(1.0) mysocket.connect( (host, port) ) # etc In contrast to the new function, the existing functionality in the socket module is much more general, and much better designed to handle the wide range of situations in which sockets are used. Socket APIs are hard to do right because sockets are complex and hard to do right. The problem the patch seeks to fix is is in httplib; the problem should be fixed in httplib. I recommend modifying the patch to remove *all* proposed changes to the socket module. Instead, the patch should restrict itself to fixing the httplib module. Sorry, Alan. From scott+python-dev at scottdial.com Wed Mar 21 11:06:53 2007 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 21 Mar 2007 06:06:53 -0400 Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: <460059DE.60307@canterbury.ac.nz> References: <45FC7EA3.4070808@canterbury.ac.nz> <45FDD76B.7080800@canterbury.ac.nz> <20070320070010.GA13982@electricrain.com> <460059DE.60307@canterbury.ac.nz> Message-ID: <460103BD.8040100@scottdial.com> Greg Ewing wrote: > Gregory P. Smith wrote: >> I prefer the default of "clean up after itself" as is to avoid leaving >> temporary file turds on systems caused by us lazy programmers. > > Maybe instead of an option there should be a separate > function with a different name, such as NewUniqueFile. > For the use cases I have in mind, the file isn't really > "temporary" at all. Or rather, only the name is temporary, > as it ultimately gets renamed. > I'm in favor of adding such a function. I've already wrote my way around this missing feature before. The tempfile modules AFAIK is the only portable way to make a unique filename and open it without exposing a race condition. As it is, it's not that difficult to write this function yourselves using mkstemp directly, but I believe there is a great benefit to have it in the stdlib. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From greg.ewing at canterbury.ac.nz Wed Mar 21 11:34:25 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 21 Mar 2007 22:34:25 +1200 Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: <460103BD.8040100@scottdial.com> References: <45FC7EA3.4070808@canterbury.ac.nz> <45FDD76B.7080800@canterbury.ac.nz> <20070320070010.GA13982@electricrain.com> <460059DE.60307@canterbury.ac.nz> <460103BD.8040100@scottdial.com> Message-ID: <46010A31.9060907@canterbury.ac.nz> Scott Dial wrote: > Greg Ewing wrote: > >> Maybe instead of an option there should be a separate >> function with a different name, such as NewUniqueFile. > > I'm in favor of adding such a function. A tangential question -- why are TemporaryFile and NamedTemporaryFile named in TitleCase, when they're functions and not classes? -- Greg From scott+python-dev at scottdial.com Wed Mar 21 12:01:17 2007 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 21 Mar 2007 07:01:17 -0400 Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: <46010A31.9060907@canterbury.ac.nz> References: <45FC7EA3.4070808@canterbury.ac.nz> <45FDD76B.7080800@canterbury.ac.nz> <20070320070010.GA13982@electricrain.com> <460059DE.60307@canterbury.ac.nz> <460103BD.8040100@scottdial.com> <46010A31.9060907@canterbury.ac.nz> Message-ID: <4601107D.5080301@scottdial.com> Greg Ewing wrote: > A tangential question -- why are TemporaryFile and > NamedTemporaryFile named in TitleCase, when they're > functions and not classes? > I wondered the same. At first draft of my email I wrote "class" operating under the assumption that only classes got to be camel-cased. If I had to guess, the rationale was that they are simply wrappers a class. Nevertheless, tempfile dates well back before the PEP 8 style guidelines. I think consistency would dictate that a newly added function should match the other ones, but I have no strong feelings about this. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From gjcarneiro at gmail.com Wed Mar 21 12:14:29 2007 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Wed, 21 Mar 2007 11:14:29 +0000 Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: <4601107D.5080301@scottdial.com> References: <45FC7EA3.4070808@canterbury.ac.nz> <45FDD76B.7080800@canterbury.ac.nz> <20070320070010.GA13982@electricrain.com> <460059DE.60307@canterbury.ac.nz> <460103BD.8040100@scottdial.com> <46010A31.9060907@canterbury.ac.nz> <4601107D.5080301@scottdial.com> Message-ID: On 3/21/07, Scott Dial wrote: > > Greg Ewing wrote: > > A tangential question -- why are TemporaryFile and > > NamedTemporaryFile named in TitleCase, when they're > > functions and not classes? > > > > I wondered the same. At first draft of my email I wrote "class" > operating under the assumption that only classes got to be camel-cased. > If I had to guess, the rationale was that they are simply wrappers a > class. Nevertheless, tempfile dates well back before the PEP 8 style > guidelines. I think consistency would dictate that a newly added > function should match the other ones, but I have no strong feelings > about this. I wouldn't bet on that. The ElementTree XML module was added to Python 2.5 and it does not follow PEP 8 style (method names are not words separated by underscores). -- Gustavo J. A. M. Carneiro "The universe is always one step beyond logic." -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070321/fd423d0d/attachment.htm From ncoghlan at gmail.com Wed Mar 21 12:26:58 2007 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Mar 2007 21:26:58 +1000 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> Message-ID: <46011682.8010307@gmail.com> Alan Kennedy wrote: > The proposed new function does not belong in the socket module. In > contrast to all of the other socket creation and management > functionality in the socket module, the new function does not handle > non-blocking IO. The rest of the socket module isn't going anywhere. If you want to do non-blocking IO, don't use the convenience function that is designed for use with blocking IO (even better, use a library that makes non-blocking IO easier to live with, like asyncore or Twisted). While the name of the proposed function and it's signature need to change (which Facundo has already acknowledged), correctly looking up an address and attempting to connect to it is a non-trivial operation, but one that is duplicated in several stdlib modules (httplib, poplib, ftplib, smtplib at least). This level of repetition for a dozen line function is insane - Facundo's patch is merely the first step towards standardising it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From g.brandl at gmx.net Wed Mar 21 12:39:02 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 21 Mar 2007 12:39:02 +0100 Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: <46010A31.9060907@canterbury.ac.nz> References: <45FC7EA3.4070808@canterbury.ac.nz> <45FDD76B.7080800@canterbury.ac.nz> <20070320070010.GA13982@electricrain.com> <460059DE.60307@canterbury.ac.nz> <460103BD.8040100@scottdial.com> <46010A31.9060907@canterbury.ac.nz> Message-ID: Greg Ewing schrieb: > Scott Dial wrote: >> Greg Ewing wrote: >> >>> Maybe instead of an option there should be a separate >>> function with a different name, such as NewUniqueFile. >> >> I'm in favor of adding such a function. > > A tangential question -- why are TemporaryFile and > NamedTemporaryFile named in TitleCase, when they're > functions and not classes? Probably because they are factory functions potentially returning a _TemporaryFileWrapper. If they were "functions", they'd have to be verbed, e.g. "make_temporary_file()". The class/function distinction is not so clear in Python from the user's point of view since there is no different calling syntax. Georg From facundo at taniquetil.com.ar Wed Mar 21 13:37:06 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Wed, 21 Mar 2007 12:37:06 +0000 (UTC) Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> Message-ID: Alan Kennedy wrote: > But remember that by adding a new function to the socket module to > support httplib et al, you are also adding a function to the socket > module that will be used directly by end users. > > I vote to reject this patch. Well, you can vote to name it _create_connection(), if your concern is what the end user will do with it. Or it's just denial towards this patch? > I recommend modifying the patch to remove *all* proposed changes to > the socket module. Instead, the patch should restrict itself to fixing > the httplib module. -1 to repeat the same functionality in 5 other libraries. As I said above, we can make it non-public. So, as a resume of the choices we still need to settle: a) Repeat the same functionality in 5 other libraries b) Write the function in socket.py, public c) Write the function in socket.py, non public We need to make the decission. The functionality was needed years ago, I don't want to lose a year discussing... The way I see it, we have three posible ways: a) python-dev votes b) python-dev doesn't care, nobody votes, I make it my way c) Guido settles this down Voting is open, ;) Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From skip at pobox.com Wed Mar 21 14:01:33 2007 From: skip at pobox.com (skip at pobox.com) Date: Wed, 21 Mar 2007 08:01:33 -0500 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> Message-ID: <17921.11437.901984.113394@montanaro.dyndns.org> Facundo> Voting is open, ;) ... Facundo> So, as a resume of the choices we still need to settle: Facundo> a) Repeat the same functionality in 5 other libraries Facundo> b) Write the function in socket.py, public Facundo> c) Write the function in socket.py, non public I vote 'c'. Skip From python-dev at alan.kennedy.name Wed Mar 21 14:10:20 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Wed, 21 Mar 2007 13:10:20 +0000 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> Message-ID: <4a951aa00703210610td76800bq53115daee834fb10@mail.gmail.com> [Facundo] > Voting is open, ;) So what are we voting on exactly? The patch as it currently is? The patch has not been updated to reflect recent discussions on the list. Will the patch be updated before the vote? I have one more issue with the patch. I think that the exception handling in the function is wrong. This is (approximately) the way the patch is currently defined. def create_connection(address, **kwargs): msg = "getaddrinfo returns an empty list" host, port = address for res in getaddrinfo(host, port, 0, SOCK_STREAM): af, socktype, proto, canonname, sa = res sock = None try: [snip] return socket except error, err: msg = str(err) # <-- don't do this if sock is not None: sock.close() raise error(msg) Changing the exception to with msg = str(err) is the wrong thing to do: this will hide the exception and make comparisons with symbolic constants fail, for users that want to handle exceptions. The correct way to handle 1. An empty return from getaddrinfo() 2. Other socket errors is as follows def create_connection(address, **kwargs): host, port = address for res in getaddrinfo(host, port, 0, SOCK_STREAM): af, socktype, proto, canonname, sa = res sock = None try: [snip] return new_socket except error, err: if sock is not None: sock.close() raise err else: raise error("getaddrinfo returns an empty list") [Facundo] > Or it's just denial towards this patch? I think this patch is poorly designed and poorly implemented. There are multiple problems in its 17 lines of socket module code; every time I look I find a new problem. Sorry. Alan. From bingham at cenix-bioscience.com Wed Mar 21 14:03:00 2007 From: bingham at cenix-bioscience.com (Aaron Bingham) Date: Wed, 21 Mar 2007 14:03:00 +0100 Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: References: <45FC7EA3.4070808@canterbury.ac.nz> <45FDD76B.7080800@canterbury.ac.nz> <20070320070010.GA13982@electricrain.com> <460059DE.60307@canterbury.ac.nz> <460103BD.8040100@scottdial.com> <46010A31.9060907@canterbury.ac.nz> Message-ID: <46012D04.9040902@cenix-bioscience.com> Georg Brandl wrote: > Greg Ewing schrieb: > >> Scott Dial wrote: >> >> A tangential question -- why are TemporaryFile and >> NamedTemporaryFile named in TitleCase, when they're >> functions and not classes? >> > > Probably because they are factory functions potentially returning a > _TemporaryFileWrapper. If they were "functions", they'd have to be > verbed, e.g. "make_temporary_file()". > > The class/function distinction is not so clear in Python from the user's > point of view since there is no different calling syntax. Actually the distinction is very clear: >>> class _PrivateClass(object): ... pass ... >>> def FunctionNamedToLookLikeClass(): ... return _PrivateClass() ... >>> pc = _PrivateClass() >>> isinstance(pc, _PrivateClass) True >>> fntllc = FunctionNamedToLookLikeClass() >>> isinstance(fntllc, FunctionNamedToLookLikeClass) Traceback (most recent call last): File "", line 1, in TypeError: isinstance() arg 2 must be a class, type, or tuple of classes and types That's sure to be an unpleasant surprise for someone. -- Aaron Bingham Senior Software Engineer bingham at cenix-bioscience.com Tel. +49 (0)351 4173-146 Fax +49 (0)351 4173-109 Cenix BioScience GmbH Tatzberg 47 01307 Dresden, Germany www.cenix-bioscience.com --------------------------------------------------------- Sitz der Gesellschaft (Place of Business): Dresden Gesch?ftsf?hrer (CEO): Dr. Christophe J. Echeverri Amtsgericht (Local Court): Dresden, HRB 19964 Ust-ID (VAT-No.): DE205824437 --------------------------------------------------------- From ncoghlan at gmail.com Wed Mar 21 14:44:22 2007 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Mar 2007 23:44:22 +1000 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: <4a951aa00703210610td76800bq53115daee834fb10@mail.gmail.com> References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> <4a951aa00703210610td76800bq53115daee834fb10@mail.gmail.com> Message-ID: <460136B6.4020001@gmail.com> Alan Kennedy wrote: > I think this patch is poorly designed and poorly implemented. There > are multiple problems in its 17 lines of socket module code; every > time I look I find a new problem. Code which is copied verbatim from httplib, and also occurs with slight variations in several other standard library modules. So all these objections indicate to me is that the operation *needs* to be centralised, any issues corrected, and then the corrected version invoked from all of the places in the standard library that need it. httplib just happens to be the first cab off the rank (as was discussed on this list a short time ago). The objection about "but it doesn't support non-blocking sockets" doesn't make sense to me. This is a function to make creation of a blocking socket more convenient - if you're doing non-blocking IO, why would you even try to use it? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From facundo at taniquetil.com.ar Wed Mar 21 14:44:48 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Wed, 21 Mar 2007 13:44:48 +0000 (UTC) Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> <4a951aa00703210610td76800bq53115daee834fb10@mail.gmail.com> Message-ID: Alan Kennedy wrote: > So what are we voting on exactly? The patch as it currently is? The > patch has not been updated to reflect recent discussions on the list. > Will the patch be updated before the vote? The voting is on a, b or c. The patch will be updated after I know what python-dev want to do. > The correct way to handle > > 1. An empty return from getaddrinfo() > 2. Other socket errors > > is as follows > > def create_connection(address, **kwargs): > host, port = address > for res in getaddrinfo(host, port, 0, SOCK_STREAM): > af, socktype, proto, canonname, sa = res > sock = None > try: > [snip] > return new_socket > except error, err: > if sock is not None: > sock.close() > raise err > else: > raise error("getaddrinfo returns an empty list") Wrong! You're raising an error the first time you can not connect. Your way, the for is superfluous. You're changing functionality here... See? It's not so easy. Maybe I should correct the patch like this: msg = "getaddrinfo returns an empty list" host, port = address for res in getaddrinfo(host, port, 0, SOCK_STREAM): af, socktype, proto, canonname, sa = res sock = None try: [snip] return sock except error, msg: if sock is not None: sock.close() raise error, msg Problems with this approach: - It raises an exception "the old way" - If you try to create three sockets, and all with errors, you'll only see the last one. Why I think that this is the way to go: - Because you're not changing the semantics of the function. Right now it's like this, and maybe we break some code if we change it. > I think this patch is poorly designed and poorly implemented. There > are multiple problems in its 17 lines of socket module code; every > time I look I find a new problem. It's better than yours. Talk is cheap. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From g.brandl at gmx.net Wed Mar 21 14:51:42 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 21 Mar 2007 14:51:42 +0100 Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: <46012D04.9040902@cenix-bioscience.com> References: <45FC7EA3.4070808@canterbury.ac.nz> <45FDD76B.7080800@canterbury.ac.nz> <20070320070010.GA13982@electricrain.com> <460059DE.60307@canterbury.ac.nz> <460103BD.8040100@scottdial.com> <46010A31.9060907@canterbury.ac.nz> <46012D04.9040902@cenix-bioscience.com> Message-ID: Aaron Bingham schrieb: > Georg Brandl wrote: >> Greg Ewing schrieb: >> >>> Scott Dial wrote: >>> >>> A tangential question -- why are TemporaryFile and >>> NamedTemporaryFile named in TitleCase, when they're >>> functions and not classes? >>> >> >> Probably because they are factory functions potentially returning a >> _TemporaryFileWrapper. If they were "functions", they'd have to be >> verbed, e.g. "make_temporary_file()". >> >> The class/function distinction is not so clear in Python from the user's >> point of view since there is no different calling syntax. > Actually the distinction is very clear: > > >>> class _PrivateClass(object): > ... pass > ... > >>> def FunctionNamedToLookLikeClass(): > ... return _PrivateClass() > ... > >>> pc = _PrivateClass() > >>> isinstance(pc, _PrivateClass) > True > >>> fntllc = FunctionNamedToLookLikeClass() > >>> isinstance(fntllc, FunctionNamedToLookLikeClass) > Traceback (most recent call last): > File "", line 1, in > TypeError: isinstance() arg 2 must be a class, type, or tuple of classes > and types > > That's sure to be an unpleasant surprise for someone. duck typing... But you have a valid point. Georg From steven.bethard at gmail.com Wed Mar 21 15:01:17 2007 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 21 Mar 2007 08:01:17 -0600 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> Message-ID: On 3/21/07, Facundo Batista wrote: > So, as a resume of the choices we still need to settle: > > a) Repeat the same functionality in 5 other libraries > b) Write the function in socket.py, public > c) Write the function in socket.py, non public The fact that it's needed in 5 places in the stdlib suggests that there's plenty of user code that could benefit from it too. Hence, I prefer (b), but if that really freaks people out, I'm okay with (c) too. I'm not okay with (a). Thanks, by the way, for creating this patch (and updating it based on python-dev feedback). I think it's really important for the maintainability of Python that duplicate code is factored out whenever possible. STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From python-dev at alan.kennedy.name Wed Mar 21 16:13:34 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Wed, 21 Mar 2007 15:13:34 +0000 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> <4a951aa00703210610td76800bq53115daee834fb10@mail.gmail.com> Message-ID: <4a951aa00703210813p3a24c6eeg3d525336eb6e73fd@mail.gmail.com> [Facundo] > Talk is cheap. I'm sorry if you see my attempts to review your patch as "cheap talk". Maybe I should have just kept my opinion to myself. You'll get your chance to return the favour when I check in my upcoming 1000+ line change to jython 2.3 to bring the jython socket, select and asyncore modules up-to-date with cpython 2.5, including providing ssl support, non-blocking IO, select/poll, etc, for the first time in jython. Alan. From python-dev at alan.kennedy.name Wed Mar 21 16:15:33 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Wed, 21 Mar 2007 15:15:33 +0000 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> <4a951aa00703210610td76800bq53115daee834fb10@mail.gmail.com> Message-ID: <4a951aa00703210815s6dd4db25me26510f758490d75@mail.gmail.com> [Facundo] > Talk is cheap. I'm sorry if you see my attempts to review your patch as "cheap talk". Maybe I should have just kept my opinion to myself. You'll get your chance to return the favour when I check in my upcoming 1000+ line change to jython 2.3 to bring the jython socket, select and asyncore modules up-to-date with cpython 2.5, including providing ssl support, non-blocking IO, select/poll, etc, for the first time in jython. Alan. From jcarlson at uci.edu Wed Mar 21 16:28:04 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 21 Mar 2007 08:28:04 -0700 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> Message-ID: <20070321081901.FC86.JCARLSON@uci.edu> Facundo Batista wrote: > Alan Kennedy wrote: > > I recommend modifying the patch to remove *all* proposed changes to > > the socket module. Instead, the patch should restrict itself to fixing > > the httplib module. > > -1 to repeat the same functionality in 5 other libraries. > > As I said above, we can make it non-public. > > So, as a resume of the choices we still need to settle: > > a) Repeat the same functionality in 5 other libraries > b) Write the function in socket.py, public > c) Write the function in socket.py, non public b or c is fine, I have no preference. In regards to 'there is no way to create a blocking socket this way', Alan is off his rocker. Facundo has already stated that he would like to use something that will allow for the passing of None as a timeout argument to specify no timeout - aka blocking sockets, as per sock.settimeout(None) (through either **kwargs or timeout=sentinel). The function is needed, and the implementation is sufficient for its intended uses. - Josiah From python-dev at alan.kennedy.name Wed Mar 21 16:42:56 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Wed, 21 Mar 2007 15:42:56 +0000 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: <20070321081901.FC86.JCARLSON@uci.edu> References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> <20070321081901.FC86.JCARLSON@uci.edu> Message-ID: <4a951aa00703210842o67020e4ata770d38b2a0b7a23@mail.gmail.com> [Josiah] > In regards to 'there is no way to > create a blocking socket this way', Alan is off his rocker. I am not off my rocker. And I never wrote the words you place in quotes (except in relation to an earlier defect in the patch where the timeout=None value was ignored). What I clearly stated is that the function as is doesn't cater for *non-blocking* sockets. I also clearly stated that I have no problem with the fact that it doesn't handle non-blocking sockets, but this either A: Needs to be enforced in the function by disallowing zero timeouts B: Needs to be recorded in the documentation The use cases for the function are limited: that's fine. But either explicitly limit them or document those limits. > The function is needed, and the implementation is sufficient for its > intended uses. When all the defects are fixed, it will be sufficient. Alan. From martin at v.loewis.de Wed Mar 21 17:12:39 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 21 Mar 2007 17:12:39 +0100 Subject: [Python-Dev] trunk and 2.5 are borken In-Reply-To: References: Message-ID: <46015977.4030108@v.loewis.de> > test_posixpath is failing in: test_relpath This is due to #1339796, r54419. I left a comment in the tracker. Regards, Martin From collinw at gmail.com Wed Mar 21 17:29:21 2007 From: collinw at gmail.com (Collin Winter) Date: Wed, 21 Mar 2007 11:29:21 -0500 Subject: [Python-Dev] trunk and 2.5 are borken In-Reply-To: <46015977.4030108@v.loewis.de> References: <46015977.4030108@v.loewis.de> Message-ID: <43aa6ff70703210929m6f770401j23c60e33d2a37b54@mail.gmail.com> On 3/21/07, "Martin v. L?wis" wrote: > > test_posixpath is failing in: test_relpath > > This is due to #1339796, r54419. I left a comment in the tracker. This is my check-in. I'll have a fix shortly. Collin Winter From facundo at taniquetil.com.ar Wed Mar 21 17:31:45 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Wed, 21 Mar 2007 16:31:45 +0000 (UTC) Subject: [Python-Dev] Adding timeout to socket.py and httplib.py - Updated Message-ID: I updated the patch #1676823, reflecting discussion here: - The function name changed, now it's _create_connection(). Its signature also changed: now, timeout is mandatorily named. - HTTPConnection has the posibility to call timeout with a number, and also with None. In both cases, it updates sock.timeout (changing effectively the default timeout). Docs and tests cases are also updated (note: now there's no doc in libsocket.tex, as this function is now private). Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From martin at v.loewis.de Wed Mar 21 17:47:17 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 21 Mar 2007 17:47:17 +0100 Subject: [Python-Dev] trunk and 2.5 are borken In-Reply-To: References: Message-ID: <46016195.9030505@v.loewis.de> > Patch 1490190 causes test_posix to fail on Tru64 due to chflags(). I have a potential fix for that in the tracker. Martin From tjreedy at udel.edu Wed Mar 21 17:51:37 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 21 Mar 2007 12:51:37 -0400 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.pyand httplib.py. References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> <20070321081901.FC86.JCARLSON@uci.edu> Message-ID: "Josiah Carlson" wrote in message news:20070321081901.FC86.JCARLSON at uci.edu... >Alan is off his rocker. To me, this sort of ad hominen comment is anti-productive and out-of-place in technical discussion. "Facundo Batista" wrote in message news:etrcse$ib2$1 at sea.gmane.org... >It's better than yours. This is a technical claim that can be debated. > Talk is cheap. Though slightly indirect, this strikes me (as well as the target) as an ad hominen slap that detracts from the technical argument in the rest of the post. See comment above. I do my best to restrain and censor such add-on comments and recommend the same to others, for both personal and public benefit. Terry Jan Reedy From tjreedy at udel.edu Wed Mar 21 18:05:53 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 21 Mar 2007 13:05:53 -0400 Subject: [Python-Dev] Patchs and bugs resume References: Message-ID: "| On 3/20/07, Facundo Batista wrote: | > Brett Cannon wrote: | > > old and had not been touched in ages. Hopefully you will be able to | > > easily port this over to the new tracker once it's up (that should | > > happen 2-4 weeks after 2.5.1 is released). | > | > You can be sure I'll port it... I am a fan of Edward Tufte and his advocacy of graceful mixtures of text and small graphics. I think this is an example. A possible improvement, if possible, would be to use thin dark bars for actions (comments) and a lighter bars for periods of inactivity. This would differentiate between items that have had no comments between first and last and those with several. tjr From guido at python.org Wed Mar 21 18:09:54 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2007 10:09:54 -0700 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.pyand httplib.py. In-Reply-To: References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> <20070321081901.FC86.JCARLSON@uci.edu> Message-ID: On 3/21/07, Terry Reedy wrote: > > "Josiah Carlson" wrote in message > news:20070321081901.FC86.JCARLSON at uci.edu... > > >Alan is off his rocker. > > To me, this sort of ad hominen comment is anti-productive and out-of-place > in technical discussion. > > "Facundo Batista" wrote in message > news:etrcse$ib2$1 at sea.gmane.org... > >It's better than yours. > > This is a technical claim that can be debated. > > > Talk is cheap. > > Though slightly indirect, this strikes me (as well as the target) as an ad > hominen slap that detracts from the technical argument in the rest of the > post. See comment above. I do my best to restrain and censor such add-on > comments and recommend the same to others, for both personal and public > benefit. > > Terry Jan Reedy Tempers flare. Alan's first message (in this thread anyway) wasn't a shining sample of diplomacy either, and it appeared to be based in a lack of understanding of the origin of the patch. Let's forget about that and go back to the patch. My only issue at this point is that it tries to distinguish carefully between timeout=None and timeout omitted. As a result the API is somewhat clumsy: a **kwds that is used for only one keyword, and a special object instance used as a default. Is that really worth it? I would personally be totally okay with timeout=None being interpreted as "use the default timeout, if one has been set". Since the default timeout is intended for use only as a "big hammer" to temporarily fix code that doesn't set timeouts but needs it and has no API to pass one (like httplib before the patch), I am personally in favor of going back to defaulting timeout to None and skipping the settimeout() call in _create_connection() if timeout is None. IMO the use case where there is a global timeout set and one library wants to override it with "no timeout" is pretty theoretical, and can just as well be handled by passing sys.maxint as the timeout. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Mar 21 18:14:36 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2007 10:14:36 -0700 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> Message-ID: On 3/21/07, Steven Bethard wrote: > On 3/21/07, Facundo Batista wrote: > > So, as a resume of the choices we still need to settle: > > > > a) Repeat the same functionality in 5 other libraries > > b) Write the function in socket.py, public > > c) Write the function in socket.py, non public > > The fact that it's needed in 5 places in the stdlib suggests that > there's plenty of user code that could benefit from it too. Hence, I > prefer (b), but if that really freaks people out, I'm okay with (c) > too. I'm not okay with (a). I agree with the reasoning leading to choice (b) as optimal. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bingham at cenix-bioscience.com Wed Mar 21 18:25:15 2007 From: bingham at cenix-bioscience.com (Aaron Bingham) Date: Wed, 21 Mar 2007 18:25:15 +0100 Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: References: <45FC7EA3.4070808@canterbury.ac.nz> <45FDD76B.7080800@canterbury.ac.nz> <20070320070010.GA13982@electricrain.com> <460059DE.60307@canterbury.ac.nz> <460103BD.8040100@scottdial.com> <46010A31.9060907@canterbury.ac.nz> <46012D04.9040902@cenix-bioscience.com> Message-ID: <46016A7B.1050506@cenix-bioscience.com> Georg Brandl wrote: > Aaron Bingham schrieb: > >> Georg Brandl wrote: >> >>> Greg Ewing schrieb: >>> >>> >>>> Scott Dial wrote: >>>> >>>> A tangential question -- why are TemporaryFile and >>>> NamedTemporaryFile named in TitleCase, when they're >>>> functions and not classes? >>>> >>>> >>> Probably because they are factory functions potentially returning a >>> _TemporaryFileWrapper. If they were "functions", they'd have to be >>> verbed, e.g. "make_temporary_file()". >>> >>> The class/function distinction is not so clear in Python from the user's >>> point of view since there is no different calling syntax. >>> >> Actually the distinction is very clear: >> >> >>> class _PrivateClass(object): >> ... pass >> ... >> >>> def FunctionNamedToLookLikeClass(): >> ... return _PrivateClass() >> ... >> >>> pc = _PrivateClass() >> >>> isinstance(pc, _PrivateClass) >> True >> >>> fntllc = FunctionNamedToLookLikeClass() >> >>> isinstance(fntllc, FunctionNamedToLookLikeClass) >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: isinstance() arg 2 must be a class, type, or tuple of classes >> and types >> >> That's sure to be an unpleasant surprise for someone. >> > > duck typing... > > But you have a valid point. > Sure, isinstance was only meant as an example. Anything that expects FunctionNamedToLookLikeClass to exhibit class-like behaviors beyond a call returning a new instance will fail. -- Aaron Bingham Senior Software Engineer bingham at cenix-bioscience.com Tel. +49 (0)351 4173-146 Fax +49 (0)351 4173-109 Cenix BioScience GmbH Tatzberg 47 01307 Dresden, Germany www.cenix-bioscience.com --------------------------------------------------------- Sitz der Gesellschaft (Place of Business): Dresden Gesch?ftsf?hrer (CEO): Dr. Christophe J. Echeverri Amtsgericht (Local Court): Dresden, HRB 19964 Ust-ID (VAT-No.): DE205824437 --------------------------------------------------------- From python at rcn.com Wed Mar 21 18:51:37 2007 From: python at rcn.com (Raymond Hettinger) Date: Wed, 21 Mar 2007 13:51:37 -0400 (EDT) Subject: [Python-Dev] Py2.5.1 release schedule Message-ID: <20070321135137.BCG01025@ms09.lnh.mail.rcn.net> What are the current thoughts on when Py2.5.1 will go out? Do we need a bug-day beforehand? Raymond From facundo at taniquetil.com.ar Wed Mar 21 18:49:12 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Wed, 21 Mar 2007 17:49:12 +0000 (UTC) Subject: [Python-Dev] I vote to reject: Adding timeout to socket.pyand httplib.py. References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> <20070321081901.FC86.JCARLSON@uci.edu> Message-ID: Guido van Rossum wrote: > (like httplib before the patch), I am personally in favor of going > back to defaulting timeout to None and skipping the settimeout() call > in _create_connection() if timeout is None. IMO the use case where > there is a global timeout set and one library wants to override it > with "no timeout" is pretty theoretical, and can just as well be > handled by passing sys.maxint as the timeout. In the new version of the patch (I updated it a few minutes ago), in _create_connection() I handle timeout being mandatory with **kwargs. And in HTTPConnection, I handle the posibility of calling it with timeout=None through a sentinel. It works, but my only issue is that it gets ugly in the help(): >>> sentinel = object() >>> def f(a, b=sentinel): ... pass ... >>> help(f) ... f(a, b=) I know I can use a class with __str__ instead of object, but what would one print in that case? In this case, "optional" does not means a value by default... I don't have very strong feelings about how to use the function. I just need a timeout to HTTPConnection, to finally have it in urllib2.urlopen(). Maybe we can settle all this by just putting timeout= and blocking= in create_connection, HTTPConnection, and urlopen(). This way, the signature would be: _create_connection(address, timeout=None, blocking=None) and the behaviour: if timeout is None: if blocking is None: pass elif blocking: sock.setblocking(True) else: sock.setblocking(False) else: if blocking is None or blocking is False: sock.settimeout(timeout) else: raise TypeError("You can not block a socket and also time it out") This way we get off from the "timeout in None means something about blocking" trap. What do you think? Thanks everybody for the patience... Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From barry at python.org Wed Mar 21 18:57:00 2007 From: barry at python.org (Barry Warsaw) Date: Wed, 21 Mar 2007 13:57:00 -0400 Subject: [Python-Dev] Py2.5.1 release schedule In-Reply-To: <20070321135137.BCG01025@ms09.lnh.mail.rcn.net> References: <20070321135137.BCG01025@ms09.lnh.mail.rcn.net> Message-ID: <0C528842-7B1F-438C-B91A-F0493DC8D942@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mar 21, 2007, at 1:51 PM, Raymond Hettinger wrote: > What are the current thoughts on when Py2.5.1 will go out? > Do we need a bug-day beforehand? That might not be a bad idea. I've been working on some email package fixes in the background. While not quite ready yet, I'm also becoming less certain that some of them should actually go into 2.5.1, rather than waiting for 2.6. A bug day would be a good way to coordinate with a few other devs and figure stuff like that out. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRgFx7XEjvBPtnXfVAQK11gP/au7vk5qlqzR3fRDtEbuYcsejuoYH8Cec yy4pP5lAOM2JXR432bDx8cRRm73u00ZD16mxm/5QpGtqkSGnXSy9EpfH2VHr1lss NsCAY/drrWk8qoMA03pO1gCVXhRBaIq36W234gB86RC+ZvWuo/pvyPGkixvGrVe3 qgQQsY/jJf0= =gjEk -----END PGP SIGNATURE----- From guido at python.org Wed Mar 21 18:58:48 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2007 10:58:48 -0700 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.pyand httplib.py. In-Reply-To: References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> <20070321081901.FC86.JCARLSON@uci.edu> Message-ID: On 3/21/07, Facundo Batista wrote: > Guido van Rossum wrote: > > > (like httplib before the patch), I am personally in favor of going > > back to defaulting timeout to None and skipping the settimeout() call > > in _create_connection() if timeout is None. IMO the use case where > > there is a global timeout set and one library wants to override it > > with "no timeout" is pretty theoretical, and can just as well be > > handled by passing sys.maxint as the timeout. > > In the new version of the patch (I updated it a few minutes ago), in > _create_connection() I handle timeout being mandatory with **kwargs. > > And in HTTPConnection, I handle the posibility of calling it with > timeout=None through a sentinel. > > It works, but my only issue is that it gets ugly in the help(): > > >>> sentinel = object() > >>> def f(a, b=sentinel): > ... pass > ... > >>> help(f) > ... > f(a, b=) > > I know I can use a class with __str__ instead of object, but what would > one print in that case? In this case, "optional" does not means a value > by default... This is why I proposed to *get rid of* the distinction between timeout=None and timeout not specified. Let an unspecified timeout default to None, and if timeout is None, skip the settimeout() call. > I don't have very strong feelings about how to use the function. I just > need a timeout to HTTPConnection, to finally have it in > urllib2.urlopen(). Yes, I remember well how it all started with a two-line patch to httplib.py :-) > Maybe we can settle all this by just putting timeout= and > blocking= in create_connection, HTTPConnection, and urlopen(). > This way, the signature would be: > > _create_connection(address, timeout=None, blocking=None) > > and the behaviour: > > if timeout is None: > if blocking is None: > pass > elif blocking: > sock.setblocking(True) > else: > sock.setblocking(False) > else: > if blocking is None or blocking is False: > sock.settimeout(timeout) > else: > raise TypeError("You can not block a socket and also time it out") > > This way we get off from the "timeout in None means something about > blocking" trap. > > What do you think? Overkill. The socket method settimeout() handles all with a single parameter. > Thanks everybody for the patience... Especially you, for volunteering to do this and then getting more feedback than you bargained for!! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fumanchu at amor.org Wed Mar 21 19:00:52 2007 From: fumanchu at amor.org (Robert Brewer) Date: Wed, 21 Mar 2007 11:00:52 -0700 Subject: [Python-Dev] I vote to reject: Adding timeout tosocket.pyand httplib.py. In-Reply-To: Message-ID: <435DF58A933BA74397B42CDEB8145A860A4E9A9F@ex9.hostedexchange.local> Facundo Batista wrote: > It works, but my only issue is that it gets ugly in the help(): > > >>> sentinel = object() > >>> def f(a, b=sentinel): > ... pass > ... > >>> help(f) > ... > f(a, b=) > > I know I can use a class with __str__ instead of object, but > what would one print in that case? I've found "missing" to be the most common (and the most understandable) thing to print in that case. Robert Brewer System Architect Amor Ministries fumanchu at amor.org From amk at amk.ca Wed Mar 21 19:17:45 2007 From: amk at amk.ca (A.M. Kuchling) Date: Wed, 21 Mar 2007 14:17:45 -0400 Subject: [Python-Dev] Py2.5.1 release schedule In-Reply-To: <20070321135137.BCG01025@ms09.lnh.mail.rcn.net> References: <20070321135137.BCG01025@ms09.lnh.mail.rcn.net> Message-ID: <20070321181745.GA5934@localhost.localdomain> On Wed, Mar 21, 2007 at 01:51:37PM -0400, Raymond Hettinger wrote: > What are the current thoughts on when Py2.5.1 will go out? > Do we need a bug-day beforehand? A bug day would be a great idea! I have a mailbox bug that'd greatly benefit from discussion about how to fix the problem. --amk From facundo at taniquetil.com.ar Wed Mar 21 19:44:59 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Wed, 21 Mar 2007 18:44:59 +0000 (UTC) Subject: [Python-Dev] I vote to reject: Adding timeout to socket.pyand httplib.py. References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> <20070321081901.FC86.JCARLSON@uci.edu> Message-ID: Guido van Rossum wrote: > This is why I proposed to *get rid of* the distinction between > timeout=None and timeout not specified. Let an unspecified timeout > default to None, and if timeout is None, skip the settimeout() call. +1 I'll abuse of your dictatorship, and let's see if we can finally specify what I'd do: - Just put a timeout default to None. If None, skip settimeout() call. - Copy the exceptions behaviour that we have actually in the higher level libraries: maybe it's not the best, but we're sure we aren't breaking anything. - As you said in other thread when voting (a,b,c), let's make the function public (but with the more specific name: create_connection). If it's ok, I'll prepare one more update to the patch. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From python at rcn.com Wed Mar 21 20:28:41 2007 From: python at rcn.com (Raymond Hettinger) Date: Wed, 21 Mar 2007 15:28:41 -0400 (EDT) Subject: [Python-Dev] Fwd: Re: Py2.5.1 release schedule Message-ID: <20070321152841.BCG39596@ms09.lnh.mail.rcn.net> [Raymond] >> What are the current thoughts on when Py2.5.1 will go out? >> Do we need a bug-day beforehand? [AMK] >A bug day would be a great idea! I have a mailbox bug that'd >greatly benefit from discussion about how to fix the problem. How about Sunday, April 1st? Raymond From barry at python.org Wed Mar 21 21:08:13 2007 From: barry at python.org (Barry Warsaw) Date: Wed, 21 Mar 2007 16:08:13 -0400 Subject: [Python-Dev] Fwd: Re: Py2.5.1 release schedule In-Reply-To: <20070321152841.BCG39596@ms09.lnh.mail.rcn.net> References: <20070321152841.BCG39596@ms09.lnh.mail.rcn.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mar 21, 2007, at 3:28 PM, Raymond Hettinger wrote: > [Raymond] >>> What are the current thoughts on when Py2.5.1 will go out? >>> Do we need a bug-day beforehand? > > [AMK] >> A bug day would be a great idea! I have a mailbox bug that'd >> greatly benefit from discussion about how to fix the problem. > > How about Sunday, April 1st? I could probably show up for a few hours that day. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRgGQrnEjvBPtnXfVAQJdwgQAkyf/SAG9Du+BzBzYJQ+0F+kfCglc6BSB U6uBretnedHIYmekTKg3e6Myr23B1DVLyJFuvpS8vN15RAePnZ4RLIRIPBtdkCDE Q0mJTZk5mY9xICeA23yMHVCZgUaflKOavn94SHrUYqsxR5FX/w30HxPWeBKqRro8 YGtiQK4BVLA= =N3Pd -----END PGP SIGNATURE----- From oliphant.travis at ieee.org Wed Mar 21 21:36:32 2007 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 21 Mar 2007 13:36:32 -0700 Subject: [Python-Dev] Extended Buffer Interface/Protocol Message-ID: I'm soliciting feedback on my extended buffer protocol that I am proposing for inclusion in Python 3000. As soon as the Python 3.0 implementation is complete, I plan to back-port the result to Python 2.6, therefore, I think there may be some interest on this list. Basically, the extended buffer protocol seeks to allow memory sharing with 1) information about what is "in" the memory (float, int, C-structure, etc.) 2) information about the "shape" of the memory (if any) 3) information about discontiguous memory segments Number 3 is where I could use feedback --- especially from PIL users and developers. Strides are a common way to think about a possibly discontiguous chunk of memory (which appear in NumPy when you select a sub-region from a larger array). The strides vector tells you how many bytes to skip in each dimension to get to the next memory location for that dimension. Because NumPy uses this memory model as do several compute libraries (like BLAS and LAPACK), it makes sense to allow this memory model to be shared between objects in Python. Currently, the proposed buffer interface eliminates the multi-segment option (for Python 3.0) which I think was originally put in place because of the PIL. However, I don't know if it is actually used by any extension types. This is a big reason why Guido wants to drop the multi-segment interface option. The question is should we eliminate the possibility of sharing memory for objects that store data basically as "arrays" of arrays (i.e. true C-style arrays). That is what I'm currently proposing, but I could also see an argument that states that if we are going to support strided memory access, we should also support array of array memory access. If this is added, then it would be another function-call that gets a array-of-array-style memory from the object. What do others think of these ideas? One possible C-API call that Python could grow with the current buffer interface is to allow contiguous-memory mirroring of discontiguous memory, or an iterator object that iterates through every element of any object that exposes the buffer protocol. Thanks for any feedback, -Travis Oliphant From collinw at gmail.com Wed Mar 21 21:37:02 2007 From: collinw at gmail.com (Collin Winter) Date: Wed, 21 Mar 2007 15:37:02 -0500 Subject: [Python-Dev] Contents of test_bool Message-ID: <43aa6ff70703211337t18119326ka27ba82c789c8d83@mail.gmail.com> Is there any reason for test_bool to contain assertions like these? self.assertIs({}.has_key(1), False) self.assertIs({1:1}.has_key(1), True) A significant portion of the file is devoted to making sure various things return bools (isinstance, operator.*) or handle bools correctly (pickle, marshal). Since these don't test the functionality of the bool type, is there a reason not to move these tests to more appropriate test files (eg, test_pickle) or removing them altogether (if they duplicate existing tests)? I've started on this somewhat, but I thought I'd ask before I spent too much time on it. Collin Winter From skip at pobox.com Wed Mar 21 21:37:49 2007 From: skip at pobox.com (skip at pobox.com) Date: Wed, 21 Mar 2007 15:37:49 -0500 Subject: [Python-Dev] Fwd: Re: Py2.5.1 release schedule In-Reply-To: References: <20070321152841.BCG39596@ms09.lnh.mail.rcn.net> Message-ID: <17921.38813.374148.63753@montanaro.dyndns.org> >> How about Sunday, April 1st? Barry> I could probably show up for a few hours that day. I can likely spend a couple hours as well. Afternoon (Central Time) would be better. Depends on Ellen's work schedule. Skip From fuzzyman at voidspace.org.uk Wed Mar 21 21:48:51 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 21 Mar 2007 20:48:51 +0000 Subject: [Python-Dev] Fwd: Re: Py2.5.1 release schedule In-Reply-To: <17921.38813.374148.63753@montanaro.dyndns.org> References: <20070321152841.BCG39596@ms09.lnh.mail.rcn.net> <17921.38813.374148.63753@montanaro.dyndns.org> Message-ID: <46019A33.9010600@voidspace.org.uk> skip at pobox.com wrote: > >> How about Sunday, April 1st? > > Barry> I could probably show up for a few hours that day. > > I can likely spend a couple hours as well. Afternoon (Central Time) would > be better. Depends on Ellen's work schedule. > I'd like to put something back into Python and can be around on that day. Michael > Skip > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > From guido at python.org Wed Mar 21 21:50:14 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2007 13:50:14 -0700 Subject: [Python-Dev] Contents of test_bool In-Reply-To: <43aa6ff70703211337t18119326ka27ba82c789c8d83@mail.gmail.com> References: <43aa6ff70703211337t18119326ka27ba82c789c8d83@mail.gmail.com> Message-ID: I think it was just expedient to group these together when the bool type was added. I wouldn't lose sleep over it either way. On 3/21/07, Collin Winter wrote: > Is there any reason for test_bool to contain assertions like these? > > self.assertIs({}.has_key(1), False) > self.assertIs({1:1}.has_key(1), True) > > A significant portion of the file is devoted to making sure various > things return bools (isinstance, operator.*) or handle bools correctly > (pickle, marshal). Since these don't test the functionality of the > bool type, is there a reason not to move these tests to more > appropriate test files (eg, test_pickle) or removing them altogether > (if they duplicate existing tests)? > > I've started on this somewhat, but I thought I'd ask before I spent > too much time on it. > > Collin Winter > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From nnorwitz at gmail.com Wed Mar 21 21:55:43 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 21 Mar 2007 13:55:43 -0700 Subject: [Python-Dev] Fwd: Re: Py2.5.1 release schedule In-Reply-To: <20070321152841.BCG39596@ms09.lnh.mail.rcn.net> References: <20070321152841.BCG39596@ms09.lnh.mail.rcn.net> Message-ID: On 3/21/07, Raymond Hettinger wrote: > [Raymond] > >> What are the current thoughts on when Py2.5.1 will go out? > >> Do we need a bug-day beforehand? > > [AMK] > >A bug day would be a great idea! I have a mailbox bug that'd > >greatly benefit from discussion about how to fix the problem. > > How about Sunday, April 1st? Is that a joke? ;-) The schedule hasn't changed from my March 7th msg: 2.5.1c1 - Tuesday, April 3 2.5.1 - Thursday April 12 (assuming a c2 is not necessary) My only concern is that 2.5.1 needs to get more stable not less. As long as people are very careful with what's put into 2.5.1 and the buildbots are respected, I think this would be great. I have not received any bug reports through SF or private mail that should block the release. Anthony didn't know of any either. I still have a backlog of mail to go through, but we are in pretty good shape right now. If someone knows of any issues, please let me know. n From g.brandl at gmx.net Wed Mar 21 21:57:21 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 21 Mar 2007 21:57:21 +0100 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py - Updated In-Reply-To: References: Message-ID: Facundo Batista schrieb: > I updated the patch #1676823, reflecting discussion here: > > - The function name changed, now it's _create_connection(). Its > signature also changed: now, timeout is mandatorily named. > > - HTTPConnection has the posibility to call timeout with a number, and > also with None. In both cases, it updates sock.timeout (changing > effectively the default timeout). > > Docs and tests cases are also updated (note: now there's no doc in > libsocket.tex, as this function is now private). Sorry that I didn't manage to review your earlier patch. As it seems, this also had some benefits (the discussion that started on your inquiry). There are others who can judge the new API and implementation better than me, but I can review the formal issues as soon as the API is accepted. Georg From jackdied at jackdied.com Wed Mar 21 22:04:05 2007 From: jackdied at jackdied.com (Jack Diederich) Date: Wed, 21 Mar 2007 17:04:05 -0400 Subject: [Python-Dev] Contents of test_bool In-Reply-To: <43aa6ff70703211337t18119326ka27ba82c789c8d83@mail.gmail.com> References: <43aa6ff70703211337t18119326ka27ba82c789c8d83@mail.gmail.com> Message-ID: <20070321210405.GH13084@performancedrivers.com> On Wed, Mar 21, 2007 at 03:37:02PM -0500, Collin Winter wrote: > Is there any reason for test_bool to contain assertions like these? > > self.assertIs({}.has_key(1), False) > self.assertIs({1:1}.has_key(1), True) > > A significant portion of the file is devoted to making sure various > things return bools (isinstance, operator.*) or handle bools correctly > (pickle, marshal). Since these don't test the functionality of the > bool type, is there a reason not to move these tests to more > appropriate test files (eg, test_pickle) or removing them altogether > (if they duplicate existing tests)? > > I've started on this somewhat, but I thought I'd ask before I spent > too much time on it. Most of them could be moved to their specific type's test module. There are a few (at least on the py3k branch) tests that check if __bool__ functions really return bools and that the proper exceptions are raised. Those should stay in test_bool.py -Jack From g.brandl at gmx.net Wed Mar 21 22:03:29 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 21 Mar 2007 22:03:29 +0100 Subject: [Python-Dev] Fwd: Re: Py2.5.1 release schedule In-Reply-To: <20070321152841.BCG39596@ms09.lnh.mail.rcn.net> References: <20070321152841.BCG39596@ms09.lnh.mail.rcn.net> Message-ID: Raymond Hettinger schrieb: > [Raymond] >>> What are the current thoughts on when Py2.5.1 will go out? >>> Do we need a bug-day beforehand? > > [AMK] >>A bug day would be a great idea! I have a mailbox bug that'd >>greatly benefit from discussion about how to fix the problem. > > How about Sunday, April 1st? Works for me. waiting-for-someone-else-to-deliver-the-joke-on-the-date-ly yours, Georg From oliphant.travis at ieee.org Wed Mar 21 22:21:02 2007 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 21 Mar 2007 14:21:02 -0700 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: References: Message-ID: Attached is the PEP. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pep_buffer.txt Url: http://mail.python.org/pipermail/python-dev/attachments/20070321/db64780b/attachment.txt From greg.ewing at canterbury.ac.nz Wed Mar 21 22:39:47 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Mar 2007 09:39:47 +1200 Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: References: <45FC7EA3.4070808@canterbury.ac.nz> <45FDD76B.7080800@canterbury.ac.nz> <20070320070010.GA13982@electricrain.com> <460059DE.60307@canterbury.ac.nz> <460103BD.8040100@scottdial.com> <46010A31.9060907@canterbury.ac.nz> Message-ID: <4601A623.7050909@canterbury.ac.nz> Georg Brandl wrote: > The class/function distinction is not so clear in Python from the user's > point of view since there is no different calling syntax. There *is* a visible distinction, though -- you can subclass classes but not functions. If these are uppercased because they wrap classes, why not just expose the class directly? Or if that's an implementation detail, it shouldn't affect the casing of the name. -- Greg From greg.ewing at canterbury.ac.nz Wed Mar 21 22:41:51 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Mar 2007 09:41:51 +1200 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> Message-ID: <4601A69F.1090708@canterbury.ac.nz> Facundo Batista wrote: > So, as a resume of the choices we still need to settle: > > a) Repeat the same functionality in 5 other libraries > b) Write the function in socket.py, public > c) Write the function in socket.py, non public or d) Put it in another module Is it time for a sockettools module, maybe? -- Greg From greg.ewing at canterbury.ac.nz Wed Mar 21 22:46:08 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Mar 2007 09:46:08 +1200 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: <17921.11437.901984.113394@montanaro.dyndns.org> References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> <17921.11437.901984.113394@montanaro.dyndns.org> Message-ID: <4601A7A0.1000901@canterbury.ac.nz> skip at pobox.com wrote: > Facundo> c) Write the function in socket.py, non public Also this option has the problem that it would be a strange usage of "non-public", since the function would be designed for use by other modules and not used at all in the module it's supposedly private to. -- Greg From nyamatongwe at gmail.com Wed Mar 21 23:01:55 2007 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Thu, 22 Mar 2007 09:01:55 +1100 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: References: Message-ID: <50862ebd0703211501p5c0de563s1e1f91b3cb1d2027@mail.gmail.com> Travis Oliphant: > 3) information about discontiguous memory segments > > > Number 3 is where I could use feedback --- especially from PIL users and > developers. Strides are a common way to think about a possibly > discontiguous chunk of memory (which appear in NumPy when you select a > sub-region from a larger array). The strides vector tells you how many > bytes to skip in each dimension to get to the next memory location for > that dimension. I think one of the motivations for discontiguous segments was for split buffers which are commonly used in text editors. A split buffer has a gap in the middle where insertions and deletions can often occur without moving much memory. When an insertion or deletion is required elsewhere then the gap is first moved to that position. I have long intended to implement a good split buffer extension for Python but the best I have currently is an extension written using Boost.Python which doesn't implement the buffer interface. Here is a description of split buffers: http://www.cs.cmu.edu/~wjh/papers/byte.html Neil From python-dev at alan.kennedy.name Wed Mar 21 23:25:08 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Wed, 21 Mar 2007 22:25:08 +0000 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: <4601A69F.1090708@canterbury.ac.nz> References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> <4601A69F.1090708@canterbury.ac.nz> Message-ID: <4a951aa00703211525j2be2c684ke20061f9da22b5fc@mail.gmail.com> [Greg Ewing] > or > > d) Put it in another module > > Is it time for a sockettools module, maybe? +1! Alan. From guido at python.org Wed Mar 21 23:40:06 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2007 15:40:06 -0700 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: <50862ebd0703211501p5c0de563s1e1f91b3cb1d2027@mail.gmail.com> References: <50862ebd0703211501p5c0de563s1e1f91b3cb1d2027@mail.gmail.com> Message-ID: On 3/21/07, Neil Hodgson wrote: > Travis Oliphant: > > > 3) information about discontiguous memory segments > > > > > > Number 3 is where I could use feedback --- especially from PIL users and > > developers. Strides are a common way to think about a possibly > > discontiguous chunk of memory (which appear in NumPy when you select a > > sub-region from a larger array). The strides vector tells you how many > > bytes to skip in each dimension to get to the next memory location for > > that dimension. > > I think one of the motivations for discontiguous segments was for > split buffers which are commonly used in text editors. A split buffer > has a gap in the middle where insertions and deletions can often occur > without moving much memory. When an insertion or deletion is required > elsewhere then the gap is first moved to that position. I have long > intended to implement a good split buffer extension for Python but the > best I have currently is an extension written using Boost.Python which > doesn't implement the buffer interface. Here is a description of split > buffers: > > http://www.cs.cmu.edu/~wjh/papers/byte.html But there's always a call to remove the gap (or move it to the end). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Wed Mar 21 23:40:49 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Mar 2007 10:40:49 +1200 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: <50862ebd0703211501p5c0de563s1e1f91b3cb1d2027@mail.gmail.com> References: <50862ebd0703211501p5c0de563s1e1f91b3cb1d2027@mail.gmail.com> Message-ID: <4601B471.1010307@canterbury.ac.nz> Neil Hodgson wrote: > I think one of the motivations for discontiguous segments was for > split buffers which are commonly used in text editors. Note that this is different from the case of an array of pointers to arrays, which is a multi-dimensional array structure, whereas a split buffer is a concatenation of two (possibly different sized) one-dimensional arrays. So an array-of-pointers interface wouldn't be a direct substitute for the existing multi-segment buffer interface. -- Greg From guido at python.org Wed Mar 21 23:45:16 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2007 15:45:16 -0700 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ Message-ID: See python.org/sf/1683368. I'd like to invite opinions on whether it's worth breaking an unknown amount of user code in 2.6 for the sake of stricter argument checking for object.__init__ and object.__new__. I think it probably isn't; but the strict version could be added to 3.0 and a warning issued in 2.6 in -Wpy3k mode. Alternatively, we could introduce the stricter code in 2.6, fix the stdlib modules that it breaks, and hope for the best. Opinions? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Mar 21 23:51:52 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2007 15:51:52 -0700 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: <4a951aa00703211525j2be2c684ke20061f9da22b5fc@mail.gmail.com> References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> <4601A69F.1090708@canterbury.ac.nz> <4a951aa00703211525j2be2c684ke20061f9da22b5fc@mail.gmail.com> Message-ID: On 3/21/07, Alan Kennedy wrote: > [Greg Ewing] > > or > > > > d) Put it in another module > > > > Is it time for a sockettools module, maybe? > > +1! -1. The new module would be just as much a jumble of unrelated APIs as the socket module already is, so there's no particularly good reason to create an arbitrary separation. Also, KISS. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From exarkun at divmod.com Thu Mar 22 02:54:16 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Wed, 21 Mar 2007 20:54:16 -0500 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: Message-ID: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> On Wed, 21 Mar 2007 15:45:16 -0700, Guido van Rossum wrote: >See python.org/sf/1683368. I'd like to invite opinions on whether it's >worth breaking an unknown amount of user code in 2.6 for the sake of >stricter argument checking for object.__init__ and object.__new__. I >think it probably isn't; but the strict version could be added to 3.0 >and a warning issued in 2.6 in -Wpy3k mode. Alternatively, we could >introduce the stricter code in 2.6, fix the stdlib modules that it >breaks, and hope for the best. Opinions? > Perhaps I misunderstand the patch, but it would appear to break not just some inadvisable uses of super(), but an actual core feature of super(). Maybe someone can set me right. Is this correct? class Base(object): def __init__(self, important): # Don't upcall with `important` because object is the base # class and its __init__ doesn't care (or won't accept) it super(Base, self).__init__() self.a = important If so, what are the implications for this? class Other(object): def __init__(self, important): # Don't upcall with `important` because object is the base # class and its __init__ doesn't care (or won't accept) it super(Other, self).__init__() self.b = important class Derived(Base, Other): pass (A similar example could be given where Base and Other take differently named arguments with nothing to do with each other. The end result is the same either way, I think.) I think I understand the desire to pull keyword arguments out at each step of the upcalling process, but I don't see how it can work, since "up" calling isn't always what's going on - given a diamond, there's arbitrary side-calling, so for cooperation to work every method has to pass on every argument, so object.__init__ has to take arbitrary args, since no one knows when their "up" call will actually hit object. Since without diamonds, naive "by-name" upcalling works, I assume that super() is actually intended to be used with diamonds, so this seems relevant. I hope I've just overlooked something. Writing this email feels very strange. Jean-Paul From greg.ewing at canterbury.ac.nz Wed Mar 21 23:32:47 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Mar 2007 10:32:47 +1200 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: References: Message-ID: <4601B28F.2050308@canterbury.ac.nz> Travis Oliphant wrote: > The question is should we eliminate the possibility of sharing memory > for objects that store data basically as "arrays" of arrays (i.e. true > C-style arrays). Can you clarify what you mean by this? Are you talking about an array of pointers to other arrays? (This is not what I would call an "array of arrays", even in C.) Supporting this kind of thing could be a slippery slope, since there can be arbitrary levels of complexity to such a structure. E.g do you support a 1d array of pointers to 3d arrays of pointers to 2d arrays? Etc. The more different kinds of format you support, the less likely it becomes that the thing consuming the data will be willing to go to the trouble required to understand it. > One possible C-API call that Python could grow with the current buffer > interface is to allow contiguous-memory mirroring of discontiguous > memory, I don't think the buffer protocol itself should incorporate anything that requires implicitly copying the data, since the whole purpose of it is to provide direct access to the data without need for copying. It would be okay to supply some utility functions for re-packing data, though. > or an iterator object that iterates through every element of any > object that exposes the buffer protocol. Again, for efficiency reasons I wouldn't like to involve Python objects and iteration mechanisms in this. The buffer interface is meant to give you raw access to the data at raw C speeds. Anything else is outside its scope, IMO. -- Greg From nyamatongwe at gmail.com Thu Mar 22 03:13:57 2007 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Thu, 22 Mar 2007 13:13:57 +1100 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: <4601B471.1010307@canterbury.ac.nz> References: <50862ebd0703211501p5c0de563s1e1f91b3cb1d2027@mail.gmail.com> <4601B471.1010307@canterbury.ac.nz> Message-ID: <50862ebd0703211913i301de6ceqf1937b60454968bb@mail.gmail.com> Greg Ewing: > So an array-of-pointers interface wouldn't be a direct > substitute for the existing multi-segment buffer > interface. Providing an array of (pointer,length) wouldn't be too much extra work for a split vector implementation. Guido van Rossum: > But there's always a call to remove the gap (or move it to the end). Yes, although its something you try to avoid. I'm not saying that this is an important use-case since no one seems to have produced a split vector implementation that provides the buffer protocol. Numeric-style array handling is much more common so deserves priority. Neil From roundup-admin at psf.upfronthosting.co.za Sun Mar 18 01:00:46 2007 From: roundup-admin at psf.upfronthosting.co.za (Tracker) Date: Sun, 18 Mar 2007 00:00:46 +0000 (UTC) Subject: [Python-Dev] Summary of Tracker Issues Message-ID: <20070318000046.4F093782AD@psf.localdomain> ACTIVITY SUMMARY (03/11/07 - 03/18/07) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1645 open ( +1) / 8581 closed ( +0) / 10226 total ( +1) Average duration of open issues: 738 days. Median duration of open issues: 686 days. Open Issues Breakdown open 1645 ( +1) pending 0 ( +0) Issues Created Or Reopened (1) ______________________________ New issue test for email 03/14/07 http://bugs.python.org/issue1001 created dubois Top Issues Most Discussed (1) _____________________________ 4 Test issue 8 days open http://bugs.python.org/issue1000 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070318/7c999d69/attachment.html From ewan at xensource.com Sun Mar 18 11:49:03 2007 From: ewan at xensource.com (Ewan Mellor) Date: Sun, 18 Mar 2007 10:49:03 +0000 Subject: [Python-Dev] Bug in inspect module In-Reply-To: <43aa6ff70703171809pb952de2yf7398c22b01d5ca1@mail.gmail.com> References: <20070317203826.GA9617@leeni.uk.xensource.com> <43aa6ff70703171809pb952de2yf7398c22b01d5ca1@mail.gmail.com> Message-ID: <20070318104903.GA3371@leeni.uk.xensource.com> On Sat, Mar 17, 2007 at 08:09:06PM -0500, Collin Winter wrote: > On 3/17/07, Ewan Mellor wrote: > >I have a problem being reported (by Xen users) where inspect.stack() is > >throwing IndexError. I think that this is a bug in inspect.py -- > >findsource > >generally throws IOError when it can't find a particular source file, but > >in > >the case where it finds a source file, but that file is shorter than > >expected, > >then findsource throws IndexError, and this is propagated all the way out > >of > >inspect.stack(). > > > >I'm not sure why inspect.py is finding source files that don't match the > >code > >being executed -- it seems to be dependent upon the install environment, > >because it's not affecting most people. That said, I think that it's > >reasonable to cope with a too-short source file in the same way as we cope > >with missing source files -- by throwing IOError from findsource and > >handling > >that exception later. > > FYI: this has been reported at least once before: see bug #1628987 > (http://python.org/sf/1628987). The problem (in the bug report, at > least) is that the source file changes between compilation and when > findsource() is called, and findsource() picks up the modified > version. Thanks for that. It's certainly a mismatch between the compiled code and the source file subsequently found. That said, I don't think that it only affects the case when people are editing source files and not recompiling -- this isn't the sort of thing that my users would be doing at this point. I suspect that the source file lookup is broken in some way, and so the wrong file ends up in the line cache. It's not affecting me, unfortunately, so I can't easily investigate. I see that you closed this bug as "Won't Fix". My two line patch seems like a reasonable workaround, and certainly more simple than the other proposals -- do you think it's worth dropping in? Cheers, Ewan. From jon at unequivocal.co.uk Mon Mar 19 18:34:01 2007 From: jon at unequivocal.co.uk (Jon Ribbens) Date: Mon, 19 Mar 2007 17:34:01 +0000 Subject: [Python-Dev] Status of thread cancellation In-Reply-To: References: Message-ID: <20070319173401.GA20271@snowy.squish.net> Nick Maclaren wrote: > Well, I have seen it hundreds of times on a dozen different Unices; > it is very common. You don't always SEE the stuck process - sometimes > the 'kill -9' causes the pid to become invisible to ps etc., and > just occasionally it can continue to use CPU until the system is > rebooted. You're describing something caused by a buggy operating system. I have never seen any modern Unix exhibit any of the behaviours you describe. I have seen such things in the 1990's though. From scott at scottdial.com Wed Mar 21 12:01:03 2007 From: scott at scottdial.com (Scott Dial) Date: Wed, 21 Mar 2007 07:01:03 -0400 Subject: [Python-Dev] Rationale for NamedTemporaryFile? In-Reply-To: <46010A31.9060907@canterbury.ac.nz> References: <45FC7EA3.4070808@canterbury.ac.nz> <45FDD76B.7080800@canterbury.ac.nz> <20070320070010.GA13982@electricrain.com> <460059DE.60307@canterbury.ac.nz> <460103BD.8040100@scottdial.com> <46010A31.9060907@canterbury.ac.nz> Message-ID: <4601106F.9030104@scottdial.com> Greg Ewing wrote: > A tangential question -- why are TemporaryFile and > NamedTemporaryFile named in TitleCase, when they're > functions and not classes? > I wondered the same. At first draft of my email I wrote "class" operating under the assumption that only classes got to be camel-cased. If I had to guess, the rationale was that they are simply wrappers a class. Nevertheless, tempfile dates well back before the PEP 8 style guidelines. I think consistency would dictate that a newly added function should match the other ones, but I have no strong feelings about this. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From alan at alan.kennedy.name Wed Mar 21 16:07:39 2007 From: alan at alan.kennedy.name (Alan Kennedy) Date: Wed, 21 Mar 2007 15:07:39 +0000 Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> <4a951aa00703210610td76800bq53115daee834fb10@mail.gmail.com> Message-ID: <4a951aa00703210807j45c163fcu1d07e242e7de103@mail.gmail.com> [Facundo] > Talk is cheap. I'm sorry if you see my attempts to review your patch as "cheap talk". Maybe I should have just kept my opinion to myself. You'll get your chance to return the favour when I check in my upcoming 1000+ line change to jython 2.3 to bring the jython socket, select and asyncore modules up-to-date with cpython 2.5, including providing ssl support, non-blocking IO, select/poll, etc, for the first time in jython. Alan. From b.wolowiec at students.mimuw.edu.pl Wed Mar 21 19:42:03 2007 From: b.wolowiec at students.mimuw.edu.pl (=?iso-8859-2?q?Bart=B3omiej_Wo=B3owiec?=) Date: Wed, 21 Mar 2007 19:42:03 +0100 Subject: [Python-Dev] regexp in Python Message-ID: <200703211942.03812.b.wolowiec@students.mimuw.edu.pl> For some time I'm interested in regular expressions and Finite State Machine. Recently, I saw that Python uses "Secret Labs' Regular Expression Engine", which very often works too slow. Its pesymistic time complexity is O(2^n), although other solutions, with time complexity O(n*m) ( O(n*m^2), m is the length of the regular expression and n is the length of the text, introduction to problem: http://swtch.com/~rsc/regexp/regexp1.html ) For example: $ cat test.py import re import sys if sys.argv[1:]: n = int(sys.argv[1]) regexp = 'a?'*n + 'a'*n print 'regexp=', regexp str = 'a'*n + 'b' + 'a'*n print 'str=', str print re.findall(regexp, str) $ time python test.py 20 regexp= a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?aaaaaaaaaaaaaaaaaaaa str= aaaaaaaaaaaaaaaaaaaabaaaaaaaaaaaaaaaaaaaa ['aaaaaaaaaaaaaaaaaaaa', 'aaaaaaaaaaaaaaaaaaaa'] real 0m0.592s user 0m0.508s sys 0m0.028s $ time python test.py 21 regexp= a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?aaaaaaaaaaaaaaaaaaaaa str= aaaaaaaaaaaaaaaaaaaaabaaaaaaaaaaaaaaaaaaaaa ['aaaaaaaaaaaaaaaaaaaaa', 'aaaaaaaaaaaaaaaaaaaaa'] real 0m1.018s user 0m1.000s sys 0m0.004s $ time python test.py 22 regexp= a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?aaaaaaaaaaaaaaaaaaaaaa str= aaaaaaaaaaaaaaaaaaaaaabaaaaaaaaaaaaaaaaaaaaaa ['aaaaaaaaaaaaaaaaaaaaaa', 'aaaaaaaaaaaaaaaaaaaaaa'] real 0m2.062s user 0m1.992s sys 0m0.028s $ time python test.py 23 regexp= a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?aaaaaaaaaaaaaaaaaaaaaaa str= aaaaaaaaaaaaaaaaaaaaaaabaaaaaaaaaaaaaaaaaaaaaaa ['aaaaaaaaaaaaaaaaaaaaaaa', 'aaaaaaaaaaaaaaaaaaaaaaa'] real 0m4.159s user 0m4.092s sys 0m0.024s Unfortuntely not for all regular expressions quick implementation may be used (e.g. when it uses backreferences), but for major part of regular expressions this implementation works much faster. In addition to Google Summer of Code I would willingly write a faster implementation of regular expressions in Python. Naturally, the implementation would be 100% compatible with the existing regexp engine. What do you think about my proposition? -- Bartlomiej Wolowiec From python at rcn.com Thu Mar 22 03:27:52 2007 From: python at rcn.com (Raymond Hettinger) Date: Wed, 21 Mar 2007 22:27:52 -0400 (EDT) Subject: [Python-Dev] Calling base class methods from C Message-ID: <20070321222752.BCH60580@ms09.lnh.mail.rcn.net> The xxsubtype.c module gives an example of calling a parent method if it is in a slot: static int spamlist_init(spamlistobject *self, PyObject *args, PyObject *kwds) { if (PyList_Type.tp_init((PyObject *)self, args, kwds) < 0) return -1; self->state = 0; return 0; } How you call non-slotted named methods in parent classes? class List(list): def append(self, x): print x List.append(self, x) # What is the C equivalent of this call? Raymond From oliphant.travis at ieee.org Thu Mar 22 03:48:25 2007 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 21 Mar 2007 19:48:25 -0700 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: <4601B28F.2050308@canterbury.ac.nz> References: <4601B28F.2050308@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Travis Oliphant wrote: > > >>The question is should we eliminate the possibility of sharing memory >>for objects that store data basically as "arrays" of arrays (i.e. true >>C-style arrays). > > > Can you clarify what you mean by this? Are you talking > about an array of pointers to other arrays? (This is > not what I would call an "array of arrays", even in C.) I'm talking about arrays of pointers to other arrays: i.e. if somebody defined in C float B[10][20] then B would B an array of pointers to arrays of floats. > > Supporting this kind of thing could be a slippery slope, > since there can be arbitrary levels of complexity to > such a structure. E.g do you support a 1d array of > pointers to 3d arrays of pointers to 2d arrays? Etc. > Yes, I saw that. But, it could actually be supported, in general. The shape information is available. If a 3-d array is meant then ndims is 3 and you would re-cast the returned pointer appropriately. In other words, suppose that instead of strides you can request a variable through the buffer interface with type void **segments. Then, by passing the address to a void * variable to the routine you would receive the array. Then, you could handle 1-d, 2-d, and 3-d cases using something like this: This is pseudocode: void *segments; int ndims; Py_ssize_t *shape; char *format; (&ndims, &shape, &format, and &segments) are passed to the buffer interface. if strcmp(format, "f") != 0 raise an error. if (ndims == 1) var = (float *)segments for (i=0; i The more different kinds of format you support, the less > likely it becomes that the thing consuming the data > will be willing to go to the trouble required to > understand it. That is certainly true. I'm really only going through the trouble, since the multiple segment already exists and the PIL has this memory model (although I have not heard PIL developers clamoring for support, --- I'm just being sensitive to that extension type). > > >>One possible C-API call that Python could grow with the current buffer >>interface is to allow contiguous-memory mirroring of discontiguous >>memory, > > > I don't think the buffer protocol itself should incorporate > anything that requires implicitly copying the data, since > the whole purpose of it is to provide direct access to the > data without need for copying. No, this would not be the buffer protocol, but merely a C-API that would use the buffer protocol - i.e. it is just a utility function as you mention. > > It would be okay to supply some utility functions for > re-packing data, though. > > >>or an iterator object that iterates through every element of any >>object that exposes the buffer protocol. > > > Again, for efficiency reasons I wouldn't like to involve > Python objects and iteration mechanisms in this. I was thinking more of a C-iterator, like NumPy provides. This can be very efficient (as long as the loop is not in Python). It sure provides a nice abstraction that lets you deal with discontiguous arrays as if they were contiguous, though. The > buffer interface is meant to give you raw access to the > data at raw C speeds. Anything else is outside its scope, Sure. These things are just ideas about *future* utility functions that might make use of the buffer interface and motivate its design. Thanks for your comments. -Travis From greg.ewing at canterbury.ac.nz Thu Mar 22 04:21:50 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Mar 2007 15:21:50 +1200 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: References: Message-ID: <4601F64E.2040506@canterbury.ac.nz> Guido van Rossum wrote: > See python.org/sf/1683368. I'd like to invite opinions on whether it's > worth breaking an unknown amount of user code in 2.6 for the sake of > stricter argument checking for object.__init__ Personally I have never written code that relies on being able to pass arbitrary args to object.__init__ and don't ever intend to, so this wouldn't bother me. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From guido at python.org Thu Mar 22 04:37:34 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2007 20:37:34 -0700 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> Message-ID: On 3/21/07, Jean-Paul Calderone wrote: > On Wed, 21 Mar 2007 15:45:16 -0700, Guido van Rossum wrote: > >See python.org/sf/1683368. I'd like to invite opinions on whether it's > >worth breaking an unknown amount of user code in 2.6 for the sake of > >stricter argument checking for object.__init__ and object.__new__. I > >think it probably isn't; but the strict version could be added to 3.0 > >and a warning issued in 2.6 in -Wpy3k mode. Alternatively, we could > >introduce the stricter code in 2.6, fix the stdlib modules that it > >breaks, and hope for the best. Opinions? > > > > Perhaps I misunderstand the patch, but it would appear to break not just > some inadvisable uses of super(), but an actual core feature of super(). > Maybe someone can set me right. Is this correct? > > class Base(object): > def __init__(self, important): > # Don't upcall with `important` because object is the base > # class and its __init__ doesn't care (or won't accept) it > super(Base, self).__init__() > self.a = important > > If so, what are the implications for this? > > class Other(object): > def __init__(self, important): > # Don't upcall with `important` because object is the base > # class and its __init__ doesn't care (or won't accept) it > super(Other, self).__init__() > self.b = important > > class Derived(Base, Other): > pass > > > (A similar example could be given where Base and Other take differently > named arguments with nothing to do with each other. The end result is > the same either way, I think.) > > I think I understand the desire to pull keyword arguments out at each > step of the upcalling process, but I don't see how it can work, since > "up" calling isn't always what's going on - given a diamond, there's > arbitrary side-calling, so for cooperation to work every method has to > pass on every argument, so object.__init__ has to take arbitrary args, > since no one knows when their "up" call will actually hit object. > > Since without diamonds, naive "by-name" upcalling works, I assume that > super() is actually intended to be used with diamonds, so this seems > relevant. > > I hope I've just overlooked something. Writing this email feels very > strange. > > Jean-Paul There are different philosophies about the correct style for cooperative super calls. The submitter of the bug report likes to remove "consumed" arguments and pass the others on, having something at the root that complains about any unused arguments. It has the problem that you mention: if multiple classes might be interested in the *same* argument they won't see it. The other style is to pass *all* arguments down, and let everyone cherry-pick them. The last call then just throws them away. This has the problem that misspelled arguments are silently ignored rather than being diagnosed at the point where you can do something about it. I don't know what the "best practice" is (like Greg Ewing, I don't use either style myself) but I've got a feeling that it must be easier to solve the former problem than the latter (also I don't know that the former actually occurs in practice). When using more traditional styles, or single inheritance, it certainly makes more sense to reject excess arguments than to ignore them; the original code was clearly intending to do this, but due to the minimalist coding, it didn't catch enough. I've pretty much made up my mind at this point that the right way forward is to be strict; in 3.0 we can afford to be strict as in the strict version of the patch, while in 2.6 we'll be less strict, but still stricter than 2.5; 2.6 in -Wpy3k mode will issue warnings for those cases where 3.0 will issue an error but 2.6 doesn't. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Thu Mar 22 04:45:24 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Mar 2007 15:45:24 +1200 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> Message-ID: <4601FBD4.70506@canterbury.ac.nz> Jean-Paul Calderone wrote: > Perhaps I misunderstand the patch, but it would appear to break not just > some inadvisable uses of super(), but an actual core feature of super(). > Maybe someone can set me right. Is this correct? > > class Base(object): > def __init__(self, important): > > If so, what are the implications for this? > > class Other(object): > def __init__(self, important): I think the idea was that each __init__ method *extracts* the arguments that apply to it and doesn't pass them on. Then, by the time object.__init__ is reached, there should be none left. This only works if the sets of keywords recognised by each __init__ method are all disjoint. Another approach is for none of them to extract anything, and always pass everything on. Then you need something at the top that accepts anything, and you get no error checking. This kind of thing seems to be a general feature of super, i.e. in order for it to work properly, all the classes using it have to agree on some set of rules or other. This creates isolated ghettos of classes that can't be mixed with each other. If you try to mix in a class which doesn't know about the rules, or follows a different set of rules, everything falls apart. Every time I've considered using super, I've eventually decided against it for reasons like this. I've always found a better way that doesn't introduce so may strange interactions between the classes involved. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From janssen at parc.com Thu Mar 22 02:53:58 2007 From: janssen at parc.com (Bill Janssen) Date: Wed, 21 Mar 2007 18:53:58 PDT Subject: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py. In-Reply-To: References: <4a951aa00703210259r790591a3w4fcb4d8e85beaa8b@mail.gmail.com> <4601A69F.1090708@canterbury.ac.nz> <4a951aa00703211525j2be2c684ke20061f9da22b5fc@mail.gmail.com> Message-ID: <07Mar21.175400pst."57996"@synergy1.parc.xerox.com> Guido van Rossum wrote: > > > Is it time for a sockettools module, maybe? > > > > +1! > > -1. The new module would be just as much a jumble of unrelated APIs as > the socket module already is, so there's no particularly good reason > to create an arbitrary separation. Also, KISS. I agree with Guido on this one. Bill From greg.ewing at canterbury.ac.nz Thu Mar 22 04:54:26 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Mar 2007 15:54:26 +1200 Subject: [Python-Dev] Calling base class methods from C In-Reply-To: <20070321222752.BCH60580@ms09.lnh.mail.rcn.net> References: <20070321222752.BCH60580@ms09.lnh.mail.rcn.net> Message-ID: <4601FDF2.8080601@canterbury.ac.nz> Raymond Hettinger wrote: > class List(list): > def append(self, x): > print x > List.append(self, x) # What is the C equivalent of this call? Something like PyObject *meth, *result; meth = PyObject_GetAttrString(PyList_Type, "append") result = PyObject_CallFunctionObjArgs(meth, self, x, NULL) plus appropriate error checking. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From jcarlson at uci.edu Thu Mar 22 05:05:11 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 21 Mar 2007 21:05:11 -0700 Subject: [Python-Dev] regexp in Python In-Reply-To: <200703211942.03812.b.wolowiec@students.mimuw.edu.pl> References: <200703211942.03812.b.wolowiec@students.mimuw.edu.pl> Message-ID: <20070321210418.FC9C.JCARLSON@uci.edu> Bart?omiej Wo?owiec wrote: > > For some time I'm interested in regular expressions and Finite State Machine. > Recently, I saw that Python uses "Secret Labs' Regular Expression Engine", > which very often works too slow. Its pesymistic time complexity is O(2^n), > although other solutions, with time complexity O(n*m) ( O(n*m^2), m is the > length of the regular expression and n is the length of the text, > introduction to problem: http://swtch.com/~rsc/regexp/regexp1.html ) [snip] > In addition to Google Summer of Code I would willingly write a faster > implementation of regular expressions in Python. Naturally, the > implementation would be 100% compatible with the existing regexp engine. > What do you think about my proposition? Submit it as a Google Summer of Code project. - Josiah From aahz at pythoncraft.com Thu Mar 22 05:12:22 2007 From: aahz at pythoncraft.com (Aahz) Date: Wed, 21 Mar 2007 21:12:22 -0700 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> Message-ID: <20070322041222.GA10709@panix.com> On Wed, Mar 21, 2007, Guido van Rossum wrote: > On 3/21/07, Jean-Paul Calderone wrote: >> >> I think I understand the desire to pull keyword arguments out at each >> step of the upcalling process, but I don't see how it can work, since >> "up" calling isn't always what's going on - given a diamond, there's >> arbitrary side-calling, so for cooperation to work every method has to >> pass on every argument, so object.__init__ has to take arbitrary args, >> since no one knows when their "up" call will actually hit object. >> >> Since without diamonds, naive "by-name" upcalling works, I assume that >> super() is actually intended to be used with diamonds, so this seems >> relevant. > > There are different philosophies about the correct style for > cooperative super calls. Maybe so, but this would massively break my company's application, if we were actually using new-style classes and the built-in super(). We have a web application that commonly passes all form fields down as **kwargs; the application uses lots of mixin classes with identically-named methods. We have a home-brew super() that crawls the stack. Call me a strong -1 on this now that JP has explained what it does. I can't believe we're the only people doing this. I guess it doesn't matter as much for 3.0 because we're probably going to have to do a massive rewrite for that, anyway. (This codebase started in 1.4 and we're still running against 2.2.) But my take is that this is still an ugly fix. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Typing is cheap. Thinking is expensive." --Roy Smith From tjreedy at udel.edu Thu Mar 22 05:19:12 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 22 Mar 2007 00:19:12 -0400 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> Message-ID: "Guido van Rossum" wrote in message news:ca471dc20703212037g3e03df1fq433a988257d10112 at mail.gmail.com... | There are different philosophies about the correct style for | cooperative super calls. | | The submitter of the bug report likes to remove "consumed" arguments | and pass the others on, having something at the root that complains | about any unused arguments. It has the problem that you mention: if | multiple classes might be interested in the *same* argument they won't | see it. The other style is to pass *all* arguments down, and let | everyone cherry-pick them. The last call then just throws them away. | This has the problem that misspelled arguments are silently ignored | rather than being diagnosed at the point where you can do something | about it. | | I don't know what the "best practice" is (like Greg Ewing, I don't use | either style myself) but I've got a feeling that it must be easier to | solve the former problem than the latter (also I don't know that the | former actually occurs in practice). When using more traditional | styles, or single inheritance, it certainly makes more sense to reject | excess arguments than to ignore them; the original code was clearly | intending to do this, but due to the minimalist coding, it didn't | catch enough. It seems to me that to get the exact behavior one wants at the apex of a diamond structure, one should subclass object and override .__init__ with a function that does not call object.__init__ and use that subclass as the apex instead of object itself. Wouldn't this mask the behavior of object.__init__ and whatever changes decided on? (But having said that, I have no opiniou on what the default should be for those who don't do this.) Terry Jan Reedy From rhamph at gmail.com Thu Mar 22 05:21:18 2007 From: rhamph at gmail.com (Adam Olsen) Date: Wed, 21 Mar 2007 22:21:18 -0600 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> Message-ID: On 3/21/07, Jean-Paul Calderone wrote: > On Wed, 21 Mar 2007 15:45:16 -0700, Guido van Rossum wrote: > >See python.org/sf/1683368. I'd like to invite opinions on whether it's > >worth breaking an unknown amount of user code in 2.6 for the sake of > >stricter argument checking for object.__init__ and object.__new__. I > >think it probably isn't; but the strict version could be added to 3.0 > >and a warning issued in 2.6 in -Wpy3k mode. Alternatively, we could > >introduce the stricter code in 2.6, fix the stdlib modules that it > >breaks, and hope for the best. Opinions? > > > > Perhaps I misunderstand the patch, but it would appear to break not just > some inadvisable uses of super(), but an actual core feature of super(). > Maybe someone can set me right. Is this correct? > > class Base(object): > def __init__(self, important): > # Don't upcall with `important` because object is the base > # class and its __init__ doesn't care (or won't accept) it > super(Base, self).__init__() > self.a = important > > If so, what are the implications for this? > > class Other(object): > def __init__(self, important): > # Don't upcall with `important` because object is the base > # class and its __init__ doesn't care (or won't accept) it > super(Other, self).__init__() > self.b = important > > class Derived(Base, Other): > pass > > > (A similar example could be given where Base and Other take differently > named arguments with nothing to do with each other. The end result is > the same either way, I think.) The common name is actually critical. Your argument names are essentially a shared namespace, just like that on the object itself, and they're both modifying it on the assumption of being the only thing that does so. There's two ways to fix your example. First, adding a common base class which is the "owner" of that name: class Owner(object): def __init__(self, important, **kwargs): super(Owner, self).__init__(**kwargs) # important is skipped class Left(Owner): def __init__(self, important, **kwargs): super(Left, self).__init__(important=important, **kwargs) class Right(Owner): def __init__(self, important, **kwargs): super(Right, self).__init__(important=important, **kwargs) class Derived(Left, Right): pass >>> Derived("hi") The other is to rename the argument, removing the namespace conflict: class Left(object): def __init__(self, oranges, **kwargs): super(Left, self).__init__(oranges=oranges, **kwargs) class Right(object): def __init__(self, apples, **kwargs): super(Right, self).__init__(apples=apples, **kwargs) class Derived(Left, Right): pass >>> Derived(apples=3, oranges=8) In this second version you could clean up Derived's interface by adding either "def __init__(self, apples, oranges, **kwargs)" and passing them both explicitly, or by adding "def __init__(self, *, **kwargs)" and requiring they by given to you by name. Either way you're completely safe. > > I think I understand the desire to pull keyword arguments out at each > step of the upcalling process, but I don't see how it can work, since > "up" calling isn't always what's going on - given a diamond, there's > arbitrary side-calling, so for cooperation to work every method has to > pass on every argument, so object.__init__ has to take arbitrary args, > since no one knows when their "up" call will actually hit object. > > Since without diamonds, naive "by-name" upcalling works, I assume that > super() is actually intended to be used with diamonds, so this seems > relevant. > > I hope I've just overlooked something. Writing this email feels very > strange. super() has always felt strange to me. Now, with PEP 3102 and the strict __init__, not so much. -- Adam Olsen, aka Rhamphoryncus From rhamph at gmail.com Thu Mar 22 05:23:27 2007 From: rhamph at gmail.com (Adam Olsen) Date: Wed, 21 Mar 2007 22:23:27 -0600 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> Message-ID: On 3/21/07, Adam Olsen wrote: > On 3/21/07, Jean-Paul Calderone wrote: > > On Wed, 21 Mar 2007 15:45:16 -0700, Guido van Rossum wrote: > > >See python.org/sf/1683368. I'd like to invite opinions on whether it's > > >worth breaking an unknown amount of user code in 2.6 for the sake of > > >stricter argument checking for object.__init__ and object.__new__. I > > >think it probably isn't; but the strict version could be added to 3.0 > > >and a warning issued in 2.6 in -Wpy3k mode. Alternatively, we could > > >introduce the stricter code in 2.6, fix the stdlib modules that it > > >breaks, and hope for the best. Opinions? > > > > > > > Perhaps I misunderstand the patch, but it would appear to break not just > > some inadvisable uses of super(), but an actual core feature of super(). > > Maybe someone can set me right. Is this correct? > > > > class Base(object): > > def __init__(self, important): > > # Don't upcall with `important` because object is the base > > # class and its __init__ doesn't care (or won't accept) it > > super(Base, self).__init__() > > self.a = important > > > > If so, what are the implications for this? > > > > class Other(object): > > def __init__(self, important): > > # Don't upcall with `important` because object is the base > > # class and its __init__ doesn't care (or won't accept) it > > super(Other, self).__init__() > > self.b = important > > > > class Derived(Base, Other): > > pass > > > > > > (A similar example could be given where Base and Other take differently > > named arguments with nothing to do with each other. The end result is > > the same either way, I think.) > > The common name is actually critical. Your argument names are > essentially a shared namespace, just like that on the object itself, > and they're both modifying it on the assumption of being the only > thing that does so. > > There's two ways to fix your example. First, adding a common base > class which is the "owner" of that name: > > class Owner(object): > def __init__(self, important, **kwargs): > super(Owner, self).__init__(**kwargs) # important is skipped > > class Left(Owner): > def __init__(self, important, **kwargs): > super(Left, self).__init__(important=important, **kwargs) > > class Right(Owner): > def __init__(self, important, **kwargs): > super(Right, self).__init__(important=important, **kwargs) > > class Derived(Left, Right): > pass > > >>> Derived("hi") > > > The other is to rename the argument, removing the namespace conflict: > > class Left(object): > def __init__(self, oranges, **kwargs): > super(Left, self).__init__(oranges=oranges, **kwargs) > > class Right(object): > def __init__(self, apples, **kwargs): > super(Right, self).__init__(apples=apples, **kwargs) > > class Derived(Left, Right): > pass Hmm, where's that "undo post" button... That should be: class Left(object): def __init__(self, oranges, **kwargs): super(Left, self).__init__(**kwargs) class Right(object): def __init__(self, apples, **kwargs): super(Right, self).__init__(**kwargs) class Derived(Left, Right): pass And I would have gotten an error when I tested it had I been using the strict __init__. > > >>> Derived(apples=3, oranges=8) > > In this second version you could clean up Derived's interface by > adding either "def __init__(self, apples, oranges, **kwargs)" and > passing them both explicitly, or by adding "def __init__(self, *, > **kwargs)" and requiring they by given to you by name. Either way > you're completely safe. > > > > > > I think I understand the desire to pull keyword arguments out at each > > step of the upcalling process, but I don't see how it can work, since > > "up" calling isn't always what's going on - given a diamond, there's > > arbitrary side-calling, so for cooperation to work every method has to > > pass on every argument, so object.__init__ has to take arbitrary args, > > since no one knows when their "up" call will actually hit object. > > > > Since without diamonds, naive "by-name" upcalling works, I assume that > > super() is actually intended to be used with diamonds, so this seems > > relevant. > > > > I hope I've just overlooked something. Writing this email feels very > > strange. > > super() has always felt strange to me. Now, with PEP 3102 and the > strict __init__, not so much. > > -- > Adam Olsen, aka Rhamphoryncus > -- Adam Olsen, aka Rhamphoryncus From greg.ewing at canterbury.ac.nz Thu Mar 22 05:25:41 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Mar 2007 16:25:41 +1200 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: References: <4601B28F.2050308@canterbury.ac.nz> Message-ID: <46020545.7000707@canterbury.ac.nz> Travis Oliphant wrote: > I'm talking about arrays of pointers to other arrays: > > i.e. if somebody defined in C > > float B[10][20] > > then B would B an array of pointers to arrays of floats. No, it wouldn't, it would be a contiguously stored 2-dimensional array of floats. An array of pointers would be float *B[10]; followed by code to allocate 10 arrays of 20 floats each and initialise B to point to them. > Yes, I saw that. But, it could actually be supported, in general. Certainly it *can* be supported, but the question is how many different format variations it's reasonable to expect the consumer of the data to be able to deal with. Because each variation requires the consumer to use different code to access the data, if it wants to avoid making a copy. > else if (ndims == 3) > > var = (float ***)segments > for (i=0; i for (j=0; j for (k=0; j # process var[i][j][k] This assumes that the 3-dimensional case is using the array-of-pointers implementation at all levels. But there are other possibilities, e.g. a 1d array of pointers to contiguous 2d arrays, or a contiguous 2d array of pointers to 1d arrays. It's hard to deal with all of those using a common piece of code. I can imagine cases like that coming up in practice. For example, an image object might store its data as four blocks of memory for R, G, B and A planes, each of which is a contiguous 2d array with shape and stride -- but you want to view it as a 3d array byte[plane][x][y]. (Actually you'd probably *prefer* to view it as byte[x][y][plane], which would make things even more difficult...) > I was thinking more of a C-iterator, like NumPy provides. This can be > very efficient (as long as the loop is not in Python). > > It sure provides a nice abstraction that lets you deal with > discontiguous arrays as if they were contiguous, though. Something like that might be useful. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From guido at python.org Thu Mar 22 05:36:52 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2007 21:36:52 -0700 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: <20070322041222.GA10709@panix.com> References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> <20070322041222.GA10709@panix.com> Message-ID: On 3/21/07, Aahz wrote: > Maybe so, but this would massively break my company's application, if we > were actually using new-style classes and the built-in super(). We have > a web application that commonly passes all form fields down as **kwargs; > the application uses lots of mixin classes with identically-named > methods. We have a home-brew super() that crawls the stack. For what? > Call me a strong -1 on this now that JP has explained what it does. I > can't believe we're the only people doing this. I guess it doesn't > matter as much for 3.0 because we're probably going to have to do a > massive rewrite for that, anyway. (This codebase started in 1.4 and > we're still running against 2.2.) But my take is that this is still an > ugly fix. A trivial fix to get the behavior you want is to introduce a new class Object that all your company's classes derive from which has an __init__ that ignores its arguments and does nothing. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Mar 22 05:41:05 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2007 21:41:05 -0700 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: <4601FBD4.70506@canterbury.ac.nz> References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> <4601FBD4.70506@canterbury.ac.nz> Message-ID: On 3/21/07, Greg Ewing wrote: > Every time I've considered using super, I've eventually > decided against it for reasons like this. I've always > found a better way that doesn't introduce so may > strange interactions between the classes involved. I end up feeling the same way, for purely pragmatic reasons. However, cooperative multiple inheritance is a respected religion, and when practiced systematically can work well. Also make a big distinction between super calls of __init__ (which are a Pythonic wart and don't exist in other languages practicing multiple inheritance AFAIK) and super calls of regular methods (which are virtually problem-free as long as you agree on a base class that defines a method and derive from that class when you want to extend it). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Mar 22 05:42:34 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2007 21:42:34 -0700 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> Message-ID: On 3/21/07, Terry Reedy wrote: > It seems to me that to get the exact behavior one wants at the apex of a > diamond structure, one should subclass object and override .__init__ with > a function that does not call object.__init__ and use that subclass as the > apex instead of object itself. Wouldn't this mask the behavior of > object.__init__ and whatever changes decided on? Yup, that's what I recommended for Aahz. > (But having said that, I have no opiniou on what the default should be for > those who don't do this.) I do now -- for the single inheritance case, refusing extra args makes the most sense too, so that sohuld be the default in 3.0. With a Py3k warning in 2.6. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Mar 22 05:45:44 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2007 21:45:44 -0700 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> Message-ID: On 3/21/07, Adam Olsen wrote: > super() has always felt strange to me. When used in __init__? Or in general? If the former, that's because it's a unique Python wart to even be able to use super for __init__. > Now, with PEP 3102 and the strict __init__, not so much. Works for me. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From rhamph at gmail.com Thu Mar 22 05:51:30 2007 From: rhamph at gmail.com (Adam Olsen) Date: Wed, 21 Mar 2007 22:51:30 -0600 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> Message-ID: On 3/21/07, Guido van Rossum wrote: > On 3/21/07, Adam Olsen wrote: > > super() has always felt strange to me. > > When used in __init__? Or in general? If the former, that's because > it's a unique Python wart to even be able to use super for __init__. In general. Too many things could fail without errors, so it wasn't obvious how to use it correctly. None of the articles I've read helped either. > > Now, with PEP 3102 and the strict __init__, not so much. > > Works for me. :-) -- Adam Olsen, aka Rhamphoryncus From pje at telecommunity.com Thu Mar 22 05:53:24 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 21 Mar 2007 23:53:24 -0500 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: References: <4601FBD4.70506@canterbury.ac.nz> <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> <4601FBD4.70506@canterbury.ac.nz> Message-ID: <5.1.1.6.0.20070321234721.02e05008@sparrow.telecommunity.com> At 09:41 PM 3/21/2007 -0700, Guido van Rossum wrote: >On 3/21/07, Greg Ewing wrote: > > Every time I've considered using super, I've eventually > > decided against it for reasons like this. I've always > > found a better way that doesn't introduce so may > > strange interactions between the classes involved. > >I end up feeling the same way, for purely pragmatic reasons. However, >cooperative multiple inheritance is a respected religion, and when >practiced systematically can work well. > >Also make a big distinction between super calls of __init__ (which are >a Pythonic wart and don't exist in other languages practicing multiple >inheritance AFAIK) and super calls of regular methods (which are >virtually problem-free as long as you agree on a base class that >defines a method and derive from that class when you want to extend >it). FYI, the only time I ever use super() for __init__ calls is for metaclasses. However, there the signature is fixed, so this problem doesn't really arise. However, in Py3K the metaclass __init__ will accept keyword arguments. What's the right thing to do there? Should we pass them all through? I'm not sure if there's any inherent symmetry between this and object.__init__ -- but it seems to be an interesting question in its own right. The whole point of being co-operative in a metaclass is to allow other metaclasses to be safely mixed in -- and they may be metaclasses from a completely different library or framework. So, it's not as simple as saying "use a specialized base class" -- the class author may not even *know* that metaclasses are involved, let alone multiple ones. From greg.ewing at canterbury.ac.nz Thu Mar 22 06:02:05 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Mar 2007 17:02:05 +1200 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: <5.1.1.6.0.20070321234721.02e05008@sparrow.telecommunity.com> References: <4601FBD4.70506@canterbury.ac.nz> <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> <4601FBD4.70506@canterbury.ac.nz> <5.1.1.6.0.20070321234721.02e05008@sparrow.telecommunity.com> Message-ID: <46020DCD.5080708@canterbury.ac.nz> Phillip J. Eby wrote: > The whole point of being co-operative in a metaclass is to allow other > metaclasses to be safely mixed in -- and they may be metaclasses from a > completely different library or framework. Some of these use cases might now be addressable using class decorators instead of mixing metaclasses. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From blake at firefox.com Thu Mar 22 07:39:44 2007 From: blake at firefox.com (Blake Ross) Date: Wed, 21 Mar 2007 23:39:44 -0700 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ Message-ID: <4fe763f0703212339h19430caeh143d463b2ee1a7b7@mail.gmail.com> At 09:41 PM 3/21/2007 -0700, Guido van Rossum wrote: > Also make a big distinction between super calls of __init__ (which are > a Pythonic wart and don't exist in other languages practicing multiple > inheritance AFAIK) Since I filed the bug, I should clarify that the primary reason I'm using super-init is to avoid multiple construction of a shared base (even if the base is just object, since I'd prefer not to rely on object's initializer being a no-op). C++ ensures that virtual bases are only constructed once, so there's no concern as to multiple construction. Is there a Python pattern better than super-init that provides the same guarantee? (Apologies if this appears in the wrong place; I just joined the list and I'm not seeing a way to participate in an existing thread.) From martin at v.loewis.de Thu Mar 22 08:07:25 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 22 Mar 2007 08:07:25 +0100 Subject: [Python-Dev] Calling base class methods from C In-Reply-To: <20070321222752.BCH60580@ms09.lnh.mail.rcn.net> References: <20070321222752.BCH60580@ms09.lnh.mail.rcn.net> Message-ID: <46022B2D.1040001@v.loewis.de> > class List(list): > def append(self, x): > print x > List.append(self, x) # What is the C equivalent of this call? Always literally what you write in Python (assuming you meant list.append): PyObject_CallMethod(PyList_Type, "append", "OO", self, x); HTH, Martin From terry at jon.es Thu Mar 22 11:21:15 2007 From: terry at jon.es (Terry Jones) Date: Thu, 22 Mar 2007 11:21:15 +0100 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: Your message at 20:37:34 on Wednesday, 21 March 2007 References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> Message-ID: <17922.22683.487075.64946@terry-jones-computer.local> >>>>> "G" == Guido van Rossum writes: G> There are different philosophies about the correct style for G> cooperative super calls. G> The submitter of the bug report likes to remove "consumed" arguments G> and pass the others on, having something at the root that complains G> about any unused arguments. It has the problem that you mention: if G> multiple classes might be interested in the *same* argument they G> won't see it. The other style is to pass *all* arguments down, and G> let everyone cherry-pick them. The last call then just throws them G> away. This has the problem that misspelled arguments are silently G> ignored rather than being diagnosed at the point where you can do G> something about it. G> I don't know what the "best practice" is (like Greg Ewing, I don't G> use either style myself) but I've got a feeling that it must be G> easier to solve the former problem than the latter (also I don't G> know that the former actually occurs in practice). I'm following up to the above, as I'm not sure where the best place to jump into this discussion is. I think the problem is more general that just consuming and/or passing on. In the typical style of passing args to super classes that I've seen, __init__ methods in the call chain also don't really even know *whether* they should use an argument. There is also the problem of argument change over time. E.g., suppose I'm the writer of a class X and, unbeknownst to me, someone writes class Y(X,Z), where Z is some other class. Suppose Z's __init__ expects an argument called z. If I one day add an argument called z, that has totally different semantics, things get really hairy. I'm the writer of X, I'm unaware of classes Y and Z, and if someone passes me an z argument then naturally I want to (try to) use it. There are other examples like this, and argument name clashes can be more specific (like 'limit' or 'debug'). In general, we're faced with a situation that is also both time-sensitive and semantics-sensitive. Class writers shouldn't need to know all future argument changes to other classes potentially upstream or downstream from them in the call chain. Given an unstructured mess of arguments, you don't know what to consume, what to ignore, what to pass on, etc., and you don't know the future. Although this sounds much more difficult than simple diamond inheritance, I think there's an easy way to solve it. When writing a class, you know 1) what arguments your __init__ method wants, and 2) what arguments your superclass(es) __init__ methods want. You know the second thing from reading the API/docs of your superclasses - that's why you're calling them. You want to be able, for example, to route a 'limit' argument to one of your super classes and another different 'limit' argument to another of your super classes. You want this to work even if one or more of your superclasses (now or in the future) in turn decides to call some other super class that also has a 'limit' argument with entirely different (or the same) semantics. Etc. An easy solution is for __init__ to receive and pass a dictionary whose keys are class names and whose values are dictionaries of arguments intended for that class. That way, an __init__ method in a class knows exactly which arguments are intended for it. It can detect extra args, missing args, misspelt args, etc. It can also prepare args for its known immediate superclasses (those from which it subclasses) - and it does know these classes as it is explicitly subclassing from them. It can leave arguments that are intended for the __init__ methods in other classes alone. It can create different arguments of the same name for its different superclasses. It can change its signature (adding and dropping args) without affecting the args passed to its superclass. Classes earlier in the call chain can do the same without affecting the args received by this class. It is also immune to differences in superclass name ordering by subclasses (i.e., a subclass of X and Y should be able to be class Z(X,Y) or class(Y,Z) without screwing up the args received by either X.__init__ or Y.__init__. An __init__ could also safely del its own arguments once it has extracted/used them. In some odd cases it might like to pass them along, perhaps even adding something (like a None key to its sub-dictionary) to indicate that it has already been called (yes, this sounds like weird - I'm just mentioning that it would be possible). There's also no need for any special top-level subclasses of object that simply ignore all their arguments (though you _could_ add one if you wanted to be strict and insist that all __init__ methods del their args). This approach adheres nicely to the explicit is better than implicit maxim. I think the implicit situation is unsolvable. I guess this could all be dressed up nicely using a metaclass that pulled out any available args, made them available to the __init__ as regular keyword args, and made available the special dictionary that holds arguments for later classes (any given __init__ may want to add/alter this). I don't know enough about metaclasses to know if this could be done completely cleanly though. I'd probably prefer it all to be explicit and manual. Sorry for such a long post without much code. I can write some example classes if the above is unclear or people think this worth looking at further. Terry From greg.ewing at canterbury.ac.nz Thu Mar 22 12:26:37 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Mar 2007 23:26:37 +1200 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: <4fe763f0703212339h19430caeh143d463b2ee1a7b7@mail.gmail.com> References: <4fe763f0703212339h19430caeh143d463b2ee1a7b7@mail.gmail.com> Message-ID: <460267ED.2050109@canterbury.ac.nz> Blake Ross wrote: > C++ ensures that virtual bases > are only constructed once, As far as I remember, C++ ensures this by not calling the base constructors automatically at all, leaving you to explicitly call the constructors of all the virtual bases that you inherit. You could adopt the same solution in Python - have methods called init_xxx in each class Xxx, and call them all from the __init__ method of the most-derived class. -- Greg From thomas at python.org Thu Mar 22 12:55:02 2007 From: thomas at python.org (Thomas Wouters) Date: Thu, 22 Mar 2007 12:55:02 +0100 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> Message-ID: <9e804ac0703220455s67771deaj35e132f486f9a3b0@mail.gmail.com> On 3/22/07, Adam Olsen wrote: > > On 3/21/07, Guido van Rossum wrote: > > On 3/21/07, Adam Olsen wrote: > > > super() has always felt strange to me. > > > > When used in __init__? Or in general? If the former, that's because > > it's a unique Python wart to even be able to use super for __init__. > > In general. Too many things could fail without errors, so it wasn't > obvious how to use it correctly. None of the articles I've read > helped either. I've been thinking about writing an article that explains how to use super(), so let's start here :) This is a long post that I'll probably eventually copy-paste-and-edit into an article of some sort, when I get the time. Please do comment, except with 'MI is insane' -- I already know that. Nevertheless, I think MI has its uses. super() is actually extremely straight-forward (except for the call syntax) and the only sane way (that I can think of) of doing co-operative multiple inheritance. Note 'co-operative'. It's not the same thing as using MI for 'mixins'. Co-operative MI isn't always (or often) useful, but it's the only way to make complexer-than-mixins MI not make mush of your brain when users do something unexpected. If you can do with composition or refactoring to make the desire for MI go away, that is probably a better solution. I'm not advocating MI over simpler solutions, but I am advocating *correct* MI over *incorrect* MI :) (And if Python 3.0 is indeed going to have abstract baseclasses, MI is going to be a lot more common, so getting it right is getting more and more important.) Bottom line is, for any given method (with as specific signature and semantics), you need a baseclass that implements that method but does not call its superclass. (It can be an empty implementation, if you want, as long as it at least has it and does not raise an exception. I believe object.__init__'s odd behaviour was a (possibly unconcious) attempt to make it such a baseclass for __init__.) Any class that wants to provide implementation of that method (in a co-operative manner) has at least derive from that baseclass. No class that does not derive from that baseclass must implement the method(s) (at least, not if it is to be part of the MI hierarchy.) Each method should use super() to call the baseclass version. In order to change the signature of a method in part of the MI tree, you need to extract the changed signature in its own hierarchy: provide a new baseclass (which inherits from the original baseclass) with the changed signature, but rather than not call the baseclass method, it calls the baseclass method with the original signature. All classes that implement the changed signature should then derive from that second baseclass. Classes multiply-inheriting from two classes where one of the two's methods are changed with respect to the common baseclass, should put the most-specific class first (this is generally good advice when doing multiple inheritance :) Mixing baseclasses with different signatures but similar semantics is not much easier this way, although IMHO the semantics are less surprising. (This goes for mixing baseclasses that change the signature of *different methods*, too: both classes are 'more specific' than the other class.) Say you have: class A: def spam(self, msg): print msg class B(A): def spam(self, msg, times): for i in range(times): super(B, self).spam(msg) class C(A): def spam(self, msg, finish): super(C, self).spam(msg) print finish There is no way straightforward way to combine them into a class D(B, C). You could make each subclass take arbitrary keyword(-only) arguments and 'consume' them (as suggested in this thread) only in the baseclasses-for-that-signature: class B(A): def spam(self, msg, times, **kwargs): for i in range(times): super(B, self).spam(msg, **kwargs) class C(A): def spam(self, msg, finish, **kwargs): super(C, self).spam(msg, **kwargs) print finish ... but that means adding **kwargs to *all* changed-in-derived-class methods that *might* want to co-operate with changed-differently methods. Notice, though, that the baseclass does *not* take arbitrary keywords (they should all have been consumed by the derived-baseclasses), so you still get checking for unused/misspelled keyword arguments. It happens in a slightly confusing place (the basemostest class for a method) but the message is just as clear. Another way to do it is to insert a new class between B and C, which implements the baseclass signature (not B's) but calls its superclass with C's signature. It has to either invent the extra argument to C's signature on the spot, or fetch it from someplace else, though: # 'helper' class to insert inbetween B and C class BtoC(C): def spam(self, msg): super(BtoC, self).spam(msg, self.finish) # New signature-baseclass for B and C combined class BandC(B, BtoC, C): def spam(self, msg, times, finish): # This obviously doesn't do the right thing for nested calls or calls from multiple threads. self.finish = finish super(BandC, self).spam(msg, times) However, the main problem with this solution is that subclasses of BandC no longer satisfy the Liskov substitution principle with respect to positional arguments to 'spam'. So, here, too, you would have to make them keyword-only arguments (although since that would be a method signature change, you could do it in a subclass of B and a subclass of C; you wouldn't have Liskov substitutability from BandC to B or C, but you would have it from BandC to KeywordOnlyB or KeywordOnlyC.) Then there's the case where you want a derived class's method to implement the functionality from a baseclass in a different way, and *not* call the baseclass method (so not entirely co-operative). That gets even trickier if you do want others to co-operate with your class, and do want co-operation with other methods. The simple solution is to put your overruling class at the end of the baseclass list, and only have it inherit from the basemost class. The problem is that anyone multiply-inheriting from your overruling class must put it at the end of its baseclass list (or the other classes' methods may get silently ignored.) So if you want to be able to mix well with other classes (in arbitrary order), you have to extract the overruling-methods into a separate baseclass with *only* the overruling bits, and make sure that one is always last in the inheritance tree right before the baseclass (or at least not before any methods you want to co-operate with.) At that point (and possibly way before) your inheritance tree is getting so complicated, you should probably give up and use composition, or refactor code so you don't have so many conflicts. However, the main point of all of this is simple: the singly-inheriting classes inbetween all these wacky glue-jobs *don't care* about any of this. They only care about their direct baseclass method signature(s), and about using super() to call them. You only have to get wacky if you want to do wacky things. Multiply-inheriting classes that don't want to change the signature of any methods don't need to care all that much either: all they need to check is whether all direct baseclasses have the same signatures. And use super() to call them, of course. Most of the sanity-checks one would want in such a co-operative MI tree (do signatures match, is there a single baseclass for a changed signature, if there is an 'overriding class' is it in the right place in the inheritance tree, etc) can pretty easily be done in a metaclass, possibly with the help of some decorators. I've been meaning to write such a metaclass, but haven't had the time. Likewise, I've been meaning to write all this down in an article. Comments, corrections, questions and blatant accusations of perversion of the fragile MI-unaware programmer's mind all welcome. Omg-look-at-the-long-post-I-feel-like-pje-now'ly y'rs, -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070322/6d107e5a/attachment.htm From python-dev at alan.kennedy.name Thu Mar 22 14:22:00 2007 From: python-dev at alan.kennedy.name (Alan Kennedy) Date: Thu, 22 Mar 2007 13:22:00 +0000 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: References: <4a951aa00703201525m77015cc6nf9714cdd9fc28f9c@mail.gmail.com> <20070320160541.FC6E.JCARLSON@uci.edu> <4a951aa00703201727s101420a4q67aee1deac33e06f@mail.gmail.com> Message-ID: <4a951aa00703220622g71586e5bwd149c920e191acfc@mail.gmail.com> [Alan] >> - Explicitly check that the address passed is a tuple of (string, integer) [Facundo] > In the code, I'll just make "host, port = address", I don't think it > will be a problem at all. Remember that this function primary use is for > higher level libraries, and that "address" in socket enviroment is > always, always, (host, port). It's rather unfortunate that the tuple needs to be unpacked at all. Instead, it should be possible to simply pass the address tuple directly to the socsket.getaddrinfo() function, and let it worry about the tuple-ness of the address, raising exceptions accordingly. The socket.getaddrinfo() function, unlike every other python socket function, takes separate host and port parameters. Which forces every user of the socket.getaddrinfo function to do the same unnecessary and potentially error-prone address tuple unpacking. I have raised a feature request to change this. [1685962] socket.getaddrinfo() should take an address tuple. Regards, Alan. From guido at python.org Thu Mar 22 15:36:28 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Mar 2007 07:36:28 -0700 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: <4fe763f0703212339h19430caeh143d463b2ee1a7b7@mail.gmail.com> References: <4fe763f0703212339h19430caeh143d463b2ee1a7b7@mail.gmail.com> Message-ID: On 3/21/07, Blake Ross wrote: > At 09:41 PM 3/21/2007 -0700, Guido van Rossum wrote: > > > Also make a big distinction between super calls of __init__ (which are > > a Pythonic wart and don't exist in other languages practicing multiple > > inheritance AFAIK) > > Since I filed the bug, I should clarify that the primary reason I'm > using super-init is to avoid multiple construction of a shared base > (even if the base is just object, since I'd prefer not to rely on > object's initializer being a no-op). C++ ensures that virtual bases > are only constructed once, so there's no concern as to multiple > construction. Is there a Python pattern better than super-init that > provides the same guarantee? > > (Apologies if this appears in the wrong place; I just joined the list > and I'm not seeing a way to participate in an existing thread.) Welcome to the group! You seem to have stirred up quite the discussion. Regarding where to ask for advice on that particular issue, python-dev is *not* the place; comp.lang.python would be more appropriate. Though perhaps Greg Ewing's suggestion is all you need. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Mar 22 15:41:20 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Mar 2007 07:41:20 -0700 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: <4a951aa00703220622g71586e5bwd149c920e191acfc@mail.gmail.com> References: <4a951aa00703201525m77015cc6nf9714cdd9fc28f9c@mail.gmail.com> <20070320160541.FC6E.JCARLSON@uci.edu> <4a951aa00703201727s101420a4q67aee1deac33e06f@mail.gmail.com> <4a951aa00703220622g71586e5bwd149c920e191acfc@mail.gmail.com> Message-ID: On 3/22/07, Alan Kennedy wrote: > [Alan] > >> - Explicitly check that the address passed is a tuple of (string, integer) > > [Facundo] > > In the code, I'll just make "host, port = address", I don't think it > > will be a problem at all. Remember that this function primary use is for > > higher level libraries, and that "address" in socket enviroment is > > always, always, (host, port). > > It's rather unfortunate that the tuple needs to be unpacked at all. Why? > Instead, it should be possible to simply pass the address tuple > directly to the socsket.getaddrinfo() function, and let it worry about > the tuple-ness of the address, raising exceptions accordingly. > > The socket.getaddrinfo() function, unlike every other python socket > function, takes separate host and port parameters. Which forces every > user of the socket.getaddrinfo function to do the same unnecessary and > potentially error-prone address tuple unpacking. > > I have raised a feature request to change this. > > [1685962] socket.getaddrinfo() should take an address tuple. It's unlikely to be granted. Getaddrinfo(), like gethostname() and a few other things, lives at a different abstraction level than the basic socket object; it is only relevant for IP sockets, not for other types of addresses. The Python call just wraps the system call which has a similar API. While from a purist POV you might want to move all IP-related APIs out of the "pure" socket module (and this would include SSL), in practice, nobody cares. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Thu Mar 22 16:17:49 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 22 Mar 2007 16:17:49 +0100 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: <460267ED.2050109@canterbury.ac.nz> References: <4fe763f0703212339h19430caeh143d463b2ee1a7b7@mail.gmail.com> <460267ED.2050109@canterbury.ac.nz> Message-ID: <46029E1D.4020103@v.loewis.de> Greg Ewing schrieb: > Blake Ross wrote: >> C++ ensures that virtual bases >> are only constructed once, > > As far as I remember, C++ ensures this by not calling > the base constructors automatically at all, leaving > you to explicitly call the constructors of all the > virtual bases that you inherit. That's not true. A virtual base constructor is always called directly from the most-specific constructor (i.e. from the constructor of the object being created). If no explicit base constructor call is present in that constructor, an implicit call to the default constructor is made. So if the base class does not have a default constructor, you indeed need to explicitly add a constructor call into each subclass. HTH, Martin From terry at jon.es Thu Mar 22 16:25:56 2007 From: terry at jon.es (Terry Jones) Date: Thu, 22 Mar 2007 16:25:56 +0100 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: Your message at 11:21:15 on Thursday, 22 March 2007 References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> <17922.22683.487075.64946@terry-jones-computer.local> Message-ID: <17922.40964.824346.989555@terry-jones-computer.local> Following up to myself, with some examples. I probably haven't done this as cleanly as is possible, but below are a bunch of classes and subclasses that cleanly deal with passing around arguments, even when those args conflict in name, etc., as outlined in my previous mail. Here's the general class __init__ pattern: class myclass(A, B, C, ...): def __init__(self, **kwargs): initargs = kwargs.setdefault('__initargs__', {}) # 1 Aargs = initargs.setdefault(A, {}) # 2 Aargs['limit'] = 10 # 3 Bargs = initargs.setdefault(B, {}) # 4 Bargs['limit'] = 'wednesday' # 5 super(myclass, self).__init__(**kwargs) # 6 myargs = initargs.get(myclass, {}) # 7 limit = myargs.get('limit', 7) # 8 if 'huh?' in myargs: raise Exception # 9 In words, the steps are: 1. Pull the __initargs__ key out of **kwargs, or create it. This gives you the top-level dictionary containing your own arguments, if any, and the args for your superclass(es), if any. 2. Get the arguments for superclass A, and 3. Add an argument for A.__init__ 4 & 5. Do the same for superclass B. We don't alter args for Superclass C. 6. Call super, passing **kwargs along (this contains the original __initargs__ that was sent to us, or the one we made in step 1 if there was no prior __initargs__. 7. Our arguments, if any, are in initargs too. Get them. 8. Pull out one of our args, or set a default. 9. Check we didn't get any unexpected args, etc. Some comments: - This isn't too ugly, I don't think, but it does require mentioning your class and superclasses by name. There are many advantages (see last mail) if you're willing to do this. - You can combine positional args and explicit keyword args in a class if you choose, and you can then disambiguate or complain if you like (class sub4 in the example code below does this). You can also take those explicit args and stuff them into the __initargs__ for your superclasses, as/if needed. Explicit args might be a slight pain to deal with, but you need them as you don't want to destroy the simple class instantiation mechanism of python that passes args normally to __init__. - It does require that classes cooperate to a small extent. E.g., you don't probably don't want the __init__ class in one of your super classes to go fiddling with initargs[myclass] before myclass even gets to see its own args (though this particular nit could be solved by having myclass copy / remove its args before calling super). - A class could also optionally delete it args once done. In that case it's cleaner to have line 7 use initargs.setdefault(myclass, {}) so a del initargs[myclass] always just works. - If you don't want to mess with the args to your superclasses, don't. In that case lines 2-5 go away in the above (and 1 and 6 could be swapped). - You can add *args to all your __init__ methods. This allows your class to sit in the middle of a class chain where a lower method wants to pass a positional argument on to a higher class (that wasn't written according to the above pattern). So the args just gets passed along. It's not as nice, but it doesn't break anything (by silently not passing on any positional args that may be present). Classes written using the above pattern can just ignore all positional args. - If I were encouraging doing something like the above in python proper, I think I'd allow line 1 to read initargs = kwargs.setdefault(None, {}) which adopts the convention that the __init__ keywords are passed in the None slot of kwargs. Currently you can't do this, as python complains that all keyword args must be strings. While this goes against explicit is better that implicit, using __initargs__ as I have done is polluting the keyword space and would probably one day cause someone a problem. There's some example code below. Terry class err(Exception): pass class sum(object): def __init__(self, **kwargs): print "-> in sum" initargs = kwargs.setdefault('__initargs__', {}) super(sum, self).__init__(**kwargs) myargs = initargs.get(sum, {}) limit = myargs.get('limit', 5) print "my limit is %d" % limit class daylimitmaker(object): def __init__(self, **kwargs): print "-> in daylimitmaker" initargs = kwargs.setdefault('__initargs__', {}) super(daylimitmaker, self).__init__(**kwargs) myargs = initargs.get(daylimitmaker, {}) limit = myargs.get('limit', 'sunday (last day of the week)') print "my limit is '%s'" % limit class lastday(object): def __init__(self, **kwargs): print "-> in lastday" initargs = kwargs.setdefault('__initargs__', {}) super(lastday, self).__init__(**kwargs) myargs = initargs.get(lastday, {}) limit = myargs.get('limit', 'yesterday') print "my limit is '%s'" % limit class onelimitarg(object): def __init__(self, **kwargs): print "-> in onelimitarg" initargs = kwargs.setdefault('__initargs__', {}) super(onelimitarg, self).__init__(**kwargs) myargs = initargs.get(onelimitarg, {}) if list(myargs.keys()) != ('limit'): raise err, "hey, someone passed me unknown arguments: %s" % myargs limit = myargs.get('limit', 1) print "my limit is %d" % limit class sub0(sum): '''Subclass sum, and pass it an explicit limit.''' def __init__(self, **kwargs): print "-> in sub0" initargs = kwargs.setdefault('__initargs__', {}) sumargs = initargs.setdefault(sum, {}) sumargs['limit'] = 99 super(sub0, self).__init__(**kwargs) class sub1(sum, lastday): '''Subclass sum and lastday, passing both an explicit limit.''' def __init__(self, **kwargs): print "-> in sub1" initargs = kwargs.setdefault('__initargs__', {}) sumargs = initargs.setdefault(sum, {}) sumargs['limit'] = 4 lastdayargs = initargs.setdefault(lastday, {}) lastdayargs['limit'] = 'friday' super(sub1, self).__init__(**kwargs) myargs = initargs.get(sub1, {}) limit = myargs.get('limit', 23) # etc... class sub2(lastday, sum): '''Subclass lastday and sum (opposite order from sub1), passing both an explicit limit.''' def __init__(self, **kwargs): print "-> in sub2" initargs = kwargs.setdefault('__initargs__', {}) sumargs = initargs.setdefault(sum, {}) sumargs['limit'] = 10 lastdayargs = initargs.setdefault(lastday, {}) lastdayargs['limit'] = 'wednesday' super(sub2, self).__init__(**kwargs) myargs = initargs.get(sub2, {}) # etc... class sub3(lastday, sum): '''Subclass lastday and sum, letting both use their default limit.''' def __init__(self, **kwargs): print "-> in sub3" initargs = kwargs.setdefault('__initargs__', {}) super(sub3, self).__init__(**kwargs) myargs = initargs.get(sub3, {}) # etc... class sub4(lastday, sum): '''Subclass lastday and sum, let our __init__ take a limit arg and check for ambiguity.''' def __init__(self, limit=10, **kwargs): '''This is deliberately ugly, but it can be done.''' print "-> in sub4" initargs = kwargs.setdefault('__initargs__', {}) sumargs = initargs.setdefault(sum, {}) sumargs['limit'] = limit super(sub4, self).__init__(**kwargs) myargs = initargs.get(sub4, {}) if 'limit' in myargs: raise err, "hey, someone passed me an __initargs__ limit argument, but i like to get an explicit limit arg.\n" class sub5(sub3, sub4): def __init__(self, **kwargs): print "-> in sub5" initargs = kwargs.setdefault('__initargs__', {}) super(sub5, self).__init__(**kwargs) myargs = initargs.get(sub5, {}) # etc... class sub6(sum, daylimitmaker): def __init__(self, **kwargs): print "-> in sub6" initargs = kwargs.setdefault('__initargs__', {}) super(sub6, self).__init__(**kwargs) myargs = initargs.get(sub6, {}) # etc... class sub7(onelimitarg): def __init__(self, **kwargs): print "-> in sub7" initargs = kwargs.setdefault('__initargs__', {}) onelimitargs = initargs.setdefault(onelimitarg, {}) onelimitargs['limit'] = 10 onelimitargs['dummy'] = 'dummy' # Pass an unknown arg. super(sub7, self).__init__(**kwargs) if __name__ == '__main__': for klass in (sub0, sub1, sub2, sub3, sub4, sub5, sub6, sub7): print "Instantiating %s: mro = %s" % (klass.__name__, [k.__name__ for k in klass.__mro__]) try: klass() except err, e: print e print print "Instantiating sub4(limit=22)" sub4(limit=22) # Call with explicit arg, the way sub4 likes it. print try: print "Instantiating sub4 with __initargs__ limit." # (The way sub4 doesn't like it.) sub4(__initargs__={sub4 : {'limit' : None}}) except err, e: print e From facundo at taniquetil.com.ar Thu Mar 22 16:28:23 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Thu, 22 Mar 2007 15:28:23 +0000 (UTC) Subject: [Python-Dev] Patchs and bugs resume References: <4600DEE8.60603@v.loewis.de> Message-ID: Martin v. L?wis wrote: > When you do, make sure you take a look at roundup's search facilities. > Roundup keeps a 'last activity' field, on which you can search and sort, > and a 'creation date' field (likewise). Could you please point me to documentation about the new tracker? I want to study the best way to extract information from it (right now, I'm just pulling htmls from SF and parsing them, and that's not easy, fast, nor clean). Thank you! -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From pje at telecommunity.com Thu Mar 22 16:33:46 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 22 Mar 2007 10:33:46 -0500 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: <46020DCD.5080708@canterbury.ac.nz> References: <5.1.1.6.0.20070321234721.02e05008@sparrow.telecommunity.com> <4601FBD4.70506@canterbury.ac.nz> <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> <4601FBD4.70506@canterbury.ac.nz> <5.1.1.6.0.20070321234721.02e05008@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070322103053.02e95e40@sparrow.telecommunity.com> At 05:02 PM 3/22/2007 +1200, Greg Ewing wrote: >Phillip J. Eby wrote: > > > The whole point of being co-operative in a metaclass is to allow other > > metaclasses to be safely mixed in -- and they may be metaclasses from a > > completely different library or framework. > >Some of these use cases might now be addressable using >class decorators instead of mixing metaclasses. I think we can rule that hypothesis out, since Zope and PEAK have had class decorators for maybe 3-4 years now. Class decorators don't let you provide metaclass methods or properties, for example. (Unless you just use the decorator to mix in the metaclass, which puts you right back where you started.) From martin at v.loewis.de Thu Mar 22 16:55:04 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 22 Mar 2007 16:55:04 +0100 Subject: [Python-Dev] Patchs and bugs resume In-Reply-To: References: <4600DEE8.60603@v.loewis.de> Message-ID: <4602A6D8.9040404@v.loewis.de> > Could you please point me to documentation about the new tracker? I want > to study the best way to extract information from it (right now, I'm > just pulling htmls from SF and parsing them, and that's not easy, fast, > nor clean). The tracker software is roundup. It's documentation is at http://roundup.sourceforge.net/ However, I suggest that you just create/recover your account at bugs.python.org (recover using your SF account name), then go to Search, customize a search, perform it, and look at the 'Download as CSV' link. HTH, Martin From oliphant.travis at ieee.org Thu Mar 22 19:53:16 2007 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 22 Mar 2007 11:53:16 -0700 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: <46020545.7000707@canterbury.ac.nz> References: <4601B28F.2050308@canterbury.ac.nz> <46020545.7000707@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Travis Oliphant wrote: > > >>I'm talking about arrays of pointers to other arrays: >> >>i.e. if somebody defined in C >> >>float B[10][20] >> >>then B would B an array of pointers to arrays of floats. > > > No, it wouldn't, it would be a contiguously stored > 2-dimensional array of floats. An array of pointers > would be > > float *B[10]; > > followed by code to allocate 10 arrays of 20 floats > each and initialise B to point to them. > You are right, of course, that example was not correct. I think the point is still valid, though. One could still use the shape to indicate how many levels of pointers-to-pointers there are (i.e. how many pointer dereferences are needed to select out an element). Further dimensionality could then be reported in the format string. This would not be hard to allow. It also would not be hard to write a utility function to copy such shared memory into a contiguous segment to provide a C-API that allows casual users to avoid the details of memory layout when they are writing an algorithm that just uses the memory. > I can imagine cases like that coming up in practice. > For example, an image object might store its data > as four blocks of memory for R, G, B and A planes, > each of which is a contiguous 2d array with shape > and stride -- but you want to view it as a 3d > array byte[plane][x][y]. All we can do is have the interface actually be able to describe it's data. Users would have to take that information and write code accordingly. In this case, for example, one possibility is that the object would raise an error if strides were requested. It would also raise an error if contiguous data was requested (or I guess it could report the R channel only if it wanted to). Only if segments were requested could it return an array of pointers to the four memory blocks. It could then report itself as a 2-d array of shape (4, H) where H is the height. Each element of the array would be reported as "%sB" % W where W is the width of the image (i.e. each element of the 2-d array would be a 1-d array of length W. Alternatively it could report itself as a 1-d array of shape (4,) with elements "(H,W)B" A user would have to write the algorithm correctly in order to access the memory correctly. Alternatively, a utility function that copies into a contiguous buffer would allow the consumer to "not care" about exactly how the memory is layed out. But, the buffer interface would allow the utility function to figure it out and do the right thing for each exporter. This flexibility would not be available if we don't allow for segmented memory in the buffer interface. So, I don't think it's that hard to at least allow the multiple-segment idea into the buffer interface (as long as all the segments are the same size, mind you). It's only one more argument to the getbuffer call. -Travis From facundo at taniquetil.com.ar Thu Mar 22 20:19:00 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Thu, 22 Mar 2007 19:19:00 +0000 (UTC) Subject: [Python-Dev] Adding timeout to socket.py and httplib.py - Updated References: Message-ID: Georg Brandl wrote: > There are others who can judge the new API and implementation better than > me, but I can review the formal issues as soon as the API is accepted. The API is accepted now, I proposed it and Guido say ok 24hs ago, ;) I'll update the patch to that API, and let you know through here... Thank you!!! -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From titus at caltech.edu Thu Mar 22 21:26:36 2007 From: titus at caltech.edu (Titus Brown) Date: Thu, 22 Mar 2007 13:26:36 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070313223629.GA27802@caltech.edu> <20070316163704.GC3107@caltech.edu> Message-ID: <20070322202636.GA23873@caltech.edu> Hi all, I posted about adding 'get_output', 'get_status_output', and 'get_status_output_errors' to subprocess here, http://ivory.idyll.org/blog/mar-07/replacing-commands-with-subprocess and got some interesting responses. Briefly, my original proposal was to add these three functions: output = get_output(cmd, input=None, cwd=None, env=None) (status, output) = get_status_output(cmd, input=None, cwd=None, env=None) (status, output, errout) = get_status_output_errors(cmd, input=None, cwd=None, env=None) Commenters convinced me to propose a few additions. In order of estimated plausibility, * first, all sensical keyword args to subprocess.Popen (everything but universal_newlines, stdout, stderr, and bufsize, which don't make much sense in the context of the proposed functions) should be accepted by the 'get_' functions. This complicates the function signatures but does make them potentially much more useful. * second, I'd like to add a 'require_success' bool keyword, that is by default False (and does nothing in that case). However, when True, the functions would emulate check_call, i.e. they would raise CalledProcessError when the returncode was not zero. * third, the 'popen2' module should be deprecated for 2.6. I don't see that it has anything in it that subprocess doesn't have. Thoughts? --titus p.s. This has been a fun learning process... ;) From guido at python.org Thu Mar 22 21:38:10 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Mar 2007 13:38:10 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <20070322202636.GA23873@caltech.edu> References: <20070313223629.GA27802@caltech.edu> <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> Message-ID: Sounds good to me. In 3.0 we should probably not have os.popen*(), nor the popen2 module at all, and do everything via the subprocess module. I wonder if we should even get rid of os.system(); then there should be a subprocess.system() instead. And do we even need os.fork(), os.exec*(), os.spawn*()? On 3/22/07, Titus Brown wrote: > Hi all, > > I posted about adding 'get_output', 'get_status_output', and > 'get_status_output_errors' to subprocess here, > > http://ivory.idyll.org/blog/mar-07/replacing-commands-with-subprocess > > and got some interesting responses. > > Briefly, my original proposal was to add these three functions: > > output = get_output(cmd, input=None, cwd=None, env=None) > > (status, output) = get_status_output(cmd, input=None, cwd=None, env=None) > (status, output, errout) = get_status_output_errors(cmd, input=None, > cwd=None, env=None) > > Commenters convinced me to propose a few additions. In order of > estimated plausibility, > > * first, all sensical keyword args to subprocess.Popen > (everything but universal_newlines, stdout, stderr, and bufsize, > which don't make much sense in the context of the proposed functions) > should be accepted by the 'get_' functions. > > This complicates the function signatures but does make them > potentially much more useful. > > * second, I'd like to add a 'require_success' bool keyword, that is > by default False (and does nothing in that case). However, when > True, the functions would emulate check_call, i.e. they would raise > CalledProcessError when the returncode was not zero. > > * third, the 'popen2' module should be deprecated for 2.6. I don't see > that it has anything in it that subprocess doesn't have. > > Thoughts? > > --titus > > p.s. This has been a fun learning process... ;) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Thu Mar 22 21:48:51 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 22 Mar 2007 21:48:51 +0100 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070313223629.GA27802@caltech.edu> <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> Message-ID: <4602EBB3.6000109@v.loewis.de> Guido van Rossum schrieb: > Sounds good to me. In 3.0 we should probably not have os.popen*(), nor > the popen2 module at all, and do everything via the subprocess module. > I wonder if we should even get rid of os.system(); then there should > be a subprocess.system() instead. And do we even need os.fork(), > os.exec*(), os.spawn*()? I don't know about about *os*.fork; I surely like to have posix.fork. The posix module exposes many OS functions as-is. This has the advantage that their semantics are crystal-clear: they do whatever the system call does (which, ideally, is what POSIX specifies for it). So you can do systems programming in Python, and only need good knowledge of the underlying system calls (i.e. using Python as a better C). For the subsystem module, the semantics is not so clear, in border cases. Regards, Martin From jon+python-dev at unequivocal.co.uk Thu Mar 22 22:03:24 2007 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Thu, 22 Mar 2007 21:03:24 +0000 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <4602EBB3.6000109@v.loewis.de> References: <20070313223629.GA27802@caltech.edu> <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> <4602EBB3.6000109@v.loewis.de> Message-ID: <20070322210324.GK741@snowy.squish.net> "\"Martin v. L?wis\"" wrote: > > And do we even need os.fork(), os.exec*(), os.spawn*()? > > I don't know about about *os*.fork; I surely like to have posix.fork. > The posix module exposes many OS functions as-is. This has the > advantage that their semantics are crystal-clear: they do whatever the > system call does (which, ideally, is what POSIX specifies for it). > So you can do systems programming in Python, and only need good > knowledge of the underlying system calls (i.e. using Python as a > better C). I definitely agree. Removing the POSIX system call mappings would make Python less useful and general-purpose. Yes it's nice to have high-level utility functions like those in the subprocess module, but I think it's very important for the low-level functions to be there too when you need them. From rhamph at gmail.com Thu Mar 22 22:06:03 2007 From: rhamph at gmail.com (Adam Olsen) Date: Thu, 22 Mar 2007 15:06:03 -0600 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: <9e804ac0703220455s67771deaj35e132f486f9a3b0@mail.gmail.com> References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> <9e804ac0703220455s67771deaj35e132f486f9a3b0@mail.gmail.com> Message-ID: On 3/22/07, Thomas Wouters wrote: > On 3/22/07, Adam Olsen wrote: > > In general. Too many things could fail without errors, so it wasn't > > obvious how to use it correctly. None of the articles I've read > > helped either. > > I've been thinking about writing an article that explains how to use > super(), so let's start here :) This is a long post that I'll probably > eventually copy-paste-and-edit into an article of some sort, when I get the > time. Please do comment, except with 'MI is insane' -- I already know that. > Nevertheless, I think MI has its uses. I'm going to be blunt, and I apologize if I offend. In short, your article is no better than any of the others. TOOWTDI What you've done is list off various ways why multiple inheritance and super() can fail, and then provide a toolbox from which a programmer can cobble together a solution to fit their exact needs. It's not pythonic. What we need is a *single* style that can be applied consistently to 90+% of problems while still requiring minimal effort to read later. Using keyword arguments and consuming them is the best I've seen so far. Sure it's a little verbose, but the verbosity is repetitive and easy to tune out. It also requires the classes to cooperate. News flash: Python isn't C++ or Java. Python uses a shared __dict__ rather than private namespaces in each class. Python *always* requires the classes to cooperate. If you want to combine uncooperative classes you need to use delegation. I'm sure somebody could whip up a metaclass to automate it, especially with the new metaclass syntax, not to mention ABCs to say "I'm string-ish" when you're delegating str. -- Adam Olsen, aka Rhamphoryncus From guido at python.org Thu Mar 22 22:22:48 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Mar 2007 14:22:48 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <20070322210324.GK741@snowy.squish.net> References: <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> <4602EBB3.6000109@v.loewis.de> <20070322210324.GK741@snowy.squish.net> Message-ID: On 3/22/07, Jon Ribbens wrote: > "\"Martin v. L?wis\"" wrote: > > > And do we even need os.fork(), os.exec*(), os.spawn*()? > > > > I don't know about about *os*.fork; I surely like to have posix.fork. > > The posix module exposes many OS functions as-is. This has the > > advantage that their semantics are crystal-clear: they do whatever the > > system call does (which, ideally, is what POSIX specifies for it). > > So you can do systems programming in Python, and only need good > > knowledge of the underlying system calls (i.e. using Python as a > > better C). > > I definitely agree. Removing the POSIX system call mappings would make > Python less useful and general-purpose. > > Yes it's nice to have high-level utility functions like those in the > subprocess module, but I think it's very important for the low-level > functions to be there too when you need them. Sure. os.fork() and the os.exec*() family can stay. But os.spawn*(), that abomination invented by Microsoft? I also hear no opposition against killign os.system() and os.popen(). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Mar 22 22:28:33 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Mar 2007 14:28:33 -0700 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> <9e804ac0703220455s67771deaj35e132f486f9a3b0@mail.gmail.com> Message-ID: Can we move this to c.l.py or python-ideas? I don't think it has any bearing on the decision on whether object.__init__() or object.__new__() should reject excess arguments. Or if it does I've lost the connection through the various long articles. I also would like to ask Mr. Olsen to tone down his rhetoric a bit. There's nothing unpythonic about designing an API using positional arguments. --Guido On 3/22/07, Adam Olsen wrote: > On 3/22/07, Thomas Wouters wrote: > > On 3/22/07, Adam Olsen wrote: > > > In general. Too many things could fail without errors, so it wasn't > > > obvious how to use it correctly. None of the articles I've read > > > helped either. > > > > I've been thinking about writing an article that explains how to use > > super(), so let's start here :) This is a long post that I'll probably > > eventually copy-paste-and-edit into an article of some sort, when I get the > > time. Please do comment, except with 'MI is insane' -- I already know that. > > Nevertheless, I think MI has its uses. > > I'm going to be blunt, and I apologize if I offend. In short, your > article is no better than any of the others. > > TOOWTDI > > What you've done is list off various ways why multiple inheritance and > super() can fail, and then provide a toolbox from which a programmer > can cobble together a solution to fit their exact needs. It's not > pythonic. What we need is a *single* style that can be applied > consistently to 90+% of problems while still requiring minimal effort > to read later. > > Using keyword arguments and consuming them is the best I've seen so > far. Sure it's a little verbose, but the verbosity is repetitive and > easy to tune out. It also requires the classes to cooperate. News > flash: Python isn't C++ or Java. Python uses a shared __dict__ rather > than private namespaces in each class. Python *always* requires the > classes to cooperate. > > If you want to combine uncooperative classes you need to use > delegation. I'm sure somebody could whip up a metaclass to automate > it, especially with the new metaclass syntax, not to mention ABCs to > say "I'm string-ish" when you're delegating str. > > -- > Adam Olsen, aka Rhamphoryncus > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fuzzyman at voidspace.org.uk Thu Mar 22 22:34:46 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 22 Mar 2007 21:34:46 +0000 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> <4602EBB3.6000109@v.loewis.de> <20070322210324.GK741@snowy.squish.net> Message-ID: <4602F676.1080907@voidspace.org.uk> Guido van Rossum wrote: > On 3/22/07, Jon Ribbens wrote: > >> "\"Martin v. L?wis\"" wrote: >> >>>> And do we even need os.fork(), os.exec*(), os.spawn*()? >>>> >>> I don't know about about *os*.fork; I surely like to have posix.fork. >>> The posix module exposes many OS functions as-is. This has the >>> advantage that their semantics are crystal-clear: they do whatever the >>> system call does (which, ideally, is what POSIX specifies for it). >>> So you can do systems programming in Python, and only need good >>> knowledge of the underlying system calls (i.e. using Python as a >>> better C). >>> >> I definitely agree. Removing the POSIX system call mappings would make >> Python less useful and general-purpose. >> >> Yes it's nice to have high-level utility functions like those in the >> subprocess module, but I think it's very important for the low-level >> functions to be there too when you need them. >> > > Sure. os.fork() and the os.exec*() family can stay. But os.spawn*(), > that abomination invented by Microsoft? I also hear no opposition > against killign os.system() and os.popen() Except that 'os.system' is really easy to use and I use it rarely enough that I *always* have to RTFM for subprocess which makes you jump through a few more (albeit simple) hoops. Additionally, AFAIK subprocess is still broken for py2exe'd applications which is a problem. All the best, Michael Foord From martin at v.loewis.de Thu Mar 22 22:37:48 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 22 Mar 2007 22:37:48 +0100 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> <4602EBB3.6000109@v.loewis.de> <20070322210324.GK741@snowy.squish.net> Message-ID: <4602F72C.5040901@v.loewis.de> > Sure. os.fork() and the os.exec*() family can stay. But os.spawn*(), > that abomination invented by Microsoft? Right, I personally would not miss it. It's also not a system call, but a library function on both Windows and Unix (the equivalent of exposing fork would be to expose CreateProcessEx - something that I think Python should do out of the box, and not just when PythonWin is installed - but you can now get it through ctypes). > I also hear no opposition > against killign os.system() and os.popen(). Both are library functions; I can implement them in Python on top of what is there (plus popen is based on stdio, which we declared evil). So yes, the can go. Regards, Martin From titus at caltech.edu Thu Mar 22 22:44:08 2007 From: titus at caltech.edu (Titus Brown) Date: Thu, 22 Mar 2007 14:44:08 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <4602F676.1080907@voidspace.org.uk> References: <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> <4602EBB3.6000109@v.loewis.de> <20070322210324.GK741@snowy.squish.net> <4602F676.1080907@voidspace.org.uk> Message-ID: <20070322214408.GB2577@caltech.edu> On Thu, Mar 22, 2007 at 09:34:46PM +0000, Michael Foord wrote: -> Guido van Rossum wrote: -> > Sure. os.fork() and the os.exec*() family can stay. But os.spawn*(), -> > that abomination invented by Microsoft? I also hear no opposition -> > against killign os.system() and os.popen() -> -> Except that 'os.system' is really easy to use and I use it rarely enough -> that I *always* have to RTFM for subprocess which makes you jump through -> a few more (albeit simple) hoops. Hopefully the patch I'm making will change that, no? I could add in a 'system'-alike call easily enough; that was suggested. But I think returncode = subprocess.call("program") is pretty simple, isn't it? -> Additionally, AFAIK subprocess is still broken for py2exe'd applications -> which is a problem. Explain? cheers, --titus From jcarlson at uci.edu Thu Mar 22 22:50:36 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 22 Mar 2007 14:50:36 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070322210324.GK741@snowy.squish.net> Message-ID: <20070322144807.FCB7.JCARLSON@uci.edu> "Guido van Rossum" wrote: > Sure. os.fork() and the os.exec*() family can stay. But os.spawn*(), > that abomination invented by Microsoft? I also hear no opposition > against killign os.system() and os.popen(). As long as os.system migrates to subprocess.system (as you originally suggested), I'm +1 with everything mentioned in this thread. - Josiah From guido at python.org Thu Mar 22 22:47:58 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Mar 2007 14:47:58 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <4602F676.1080907@voidspace.org.uk> References: <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> <4602EBB3.6000109@v.loewis.de> <20070322210324.GK741@snowy.squish.net> <4602F676.1080907@voidspace.org.uk> Message-ID: On 3/22/07, Michael Foord wrote: > Guido van Rossum wrote: > > Sure. os.fork() and the os.exec*() family can stay. But os.spawn*(), > > that abomination invented by Microsoft? I also hear no opposition > > against killign os.system() and os.popen() > > Except that 'os.system' is really easy to use and I use it rarely enough > that I *always* have to RTFM for subprocess which makes you jump through > a few more (albeit simple) hoops. So let's add subprocess.system() which takes care of the hoops (but still allows you more flexibility through optional keyword parameters). > Additionally, AFAIK subprocess is still broken for py2exe'd applications > which is a problem. That doesn't sound like an API design concern -- it's a bug that just needs to be fixed somewhere. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fuzzyman at voidspace.org.uk Thu Mar 22 22:53:32 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 22 Mar 2007 21:53:32 +0000 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <20070322214408.GB2577@caltech.edu> References: <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> <4602EBB3.6000109@v.loewis.de> <20070322210324.GK741@snowy.squish.net> <4602F676.1080907@voidspace.org.uk> <20070322214408.GB2577@caltech.edu> Message-ID: <4602FADC.2060700@voidspace.org.uk> Titus Brown wrote: > On Thu, Mar 22, 2007 at 09:34:46PM +0000, Michael Foord wrote: > -> Guido van Rossum wrote: > -> > Sure. os.fork() and the os.exec*() family can stay. But os.spawn*(), > -> > that abomination invented by Microsoft? I also hear no opposition > -> > against killign os.system() and os.popen() > -> > -> Except that 'os.system' is really easy to use and I use it rarely enough > -> that I *always* have to RTFM for subprocess which makes you jump through > -> a few more (albeit simple) hoops. > > Hopefully the patch I'm making will change that, no? > > I could add in a 'system'-alike call easily enough; that was suggested. > But I think > > returncode = subprocess.call("program") > > is pretty simple, isn't it? > Probably. I can just never remember it (I didn't remember it when typing that email). My fault, os.system is just easier to remember - make an os system call. :-) > -> Additionally, AFAIK subprocess is still broken for py2exe'd applications > -> which is a problem. > > Explain? > Programs created with py2exe (or frozen in other ways I believe - including the Wing IDE for example), fail with an error similar to the following : Traceback (most recent call last): File "main.py", line 133, in main() File "main.py", line 125, in main launch_launcher() File "main.py", line 98, in launch_launcher subprocess.Popen([path], stderr = childstderr) File "subprocess.pyc", line 586, in ___init___ File "subprocess.pyc", line 681, in _get_handles File "subprocess.pyc", line 722, in _make_inheritable TypeError: an integer is required The problem is detailed here : http://www.py2exe.org/index.cgi/Py2ExeSubprocessInteractions "if you used py2exe to create a Windows program as opposed to a console program, and you launched the py2exe program by clicking rather than by running if from a command window, then the parent process has no handle to inherit. subprocess.Popen will throw an exception (TypeError: an integer is required) because GetStdHandle returns None" (Although the workaround as listed there is reported not to work and I haven't put the effort into experimenting.) If subprocess is supposed to be the main way that users launch sub-processes it would be nice if it worked for py2exe without the users (who are often newbies) having to put a workaround in place. Currently of course 'os.system' is often sufficient workaround. :-) All the best, Michael Foord > cheers, > --titus > > From titus at caltech.edu Thu Mar 22 22:54:51 2007 From: titus at caltech.edu (Titus Brown) Date: Thu, 22 Mar 2007 14:54:51 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> <4602EBB3.6000109@v.loewis.de> <20070322210324.GK741@snowy.squish.net> <4602F676.1080907@voidspace.org.uk> Message-ID: <20070322215451.GC2577@caltech.edu> On Thu, Mar 22, 2007 at 02:47:58PM -0700, Guido van Rossum wrote: -> On 3/22/07, Michael Foord wrote: -> > Guido van Rossum wrote: -> > > Sure. os.fork() and the os.exec*() family can stay. But os.spawn*(), -> > > that abomination invented by Microsoft? I also hear no opposition -> > > against killign os.system() and os.popen() -> > -> > Except that 'os.system' is really easy to use and I use it rarely enough -> > that I *always* have to RTFM for subprocess which makes you jump through -> > a few more (albeit simple) hoops. -> -> So let's add subprocess.system() which takes care of the hoops (but -> still allows you more flexibility through optional keyword -> parameters). How would this differ from subprocess.call()? http://docs.python.org/lib/node530.html (And should I add this in my patch, or is this for Py3k?) --titus From nd at perlig.de Thu Mar 22 23:12:26 2007 From: nd at perlig.de (=?iso-8859-1?q?Andr=E9_Malo?=) Date: Thu, 22 Mar 2007 23:12:26 +0100 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <20070322215451.GC2577@caltech.edu> References: <20070322215451.GC2577@caltech.edu> Message-ID: <200703222312.26936@news.perlig.de> * Titus Brown wrote: > On Thu, Mar 22, 2007 at 02:47:58PM -0700, Guido van Rossum wrote: > -> On 3/22/07, Michael Foord wrote: > -> > Guido van Rossum wrote: > -> > > Sure. os.fork() and the os.exec*() family can stay. But > os.spawn*(), -> > > that abomination invented by Microsoft? I also hear > no opposition -> > > against killign os.system() and os.popen() > -> > > -> > Except that 'os.system' is really easy to use and I use it rarely > enough -> > that I *always* have to RTFM for subprocess which makes you > jump through -> > a few more (albeit simple) hoops. > -> > -> So let's add subprocess.system() which takes care of the hoops (but > -> still allows you more flexibility through optional keyword > -> parameters). > > How would this differ from subprocess.call()? > > http://docs.python.org/lib/node530.html It doesn't implement the system() spec: nd -- Winnetous Erbe: From rhamph at gmail.com Thu Mar 22 23:29:29 2007 From: rhamph at gmail.com (Adam Olsen) Date: Thu, 22 Mar 2007 16:29:29 -0600 Subject: [Python-Dev] Breaking calls to object.__init__/__new__ In-Reply-To: References: <20070322015416.18920.1264754307.divmod.quotient.4707@ohm> <9e804ac0703220455s67771deaj35e132f486f9a3b0@mail.gmail.com> Message-ID: On 3/22/07, Guido van Rossum wrote: > Can we move this to c.l.py or python-ideas? I don't think it has any > bearing on the decision on whether object.__init__() or > object.__new__() should reject excess arguments. Or if it does I've > lost the connection through the various long articles. It's about use-cases involving what object.__init__() does. However, as it supports the decision that has been already made, you are right in that it no longer belongs on python-dev. I'll move further replies somewhere else. > I also would like to ask Mr. Olsen to tone down his rhetoric a bit. > There's nothing unpythonic about designing an API using positional > arguments. Again, I apologize for that. But the unpythonic comment referred only to providing an assortment of unobvious choices, rather than a single obvious one (perhaps with specialty options rarely used). It was not in reference to my previous argument against positional arguments. -- Adam Olsen, aka Rhamphoryncus From titus at caltech.edu Fri Mar 23 00:21:20 2007 From: titus at caltech.edu (Titus Brown) Date: Thu, 22 Mar 2007 16:21:20 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <200703222312.26936@news.perlig.de> References: <20070322215451.GC2577@caltech.edu> <200703222312.26936@news.perlig.de> Message-ID: <20070322232120.GA4512@caltech.edu> On Thu, Mar 22, 2007 at 11:12:26PM +0100, Andr? Malo wrote: -> * Titus Brown wrote: -> -> > On Thu, Mar 22, 2007 at 02:47:58PM -0700, Guido van Rossum wrote: -> > -> On 3/22/07, Michael Foord wrote: -> > -> > Guido van Rossum wrote: -> > -> > > Sure. os.fork() and the os.exec*() family can stay. But -> > os.spawn*(), -> > > that abomination invented by Microsoft? I also hear -> > no opposition -> > > against killign os.system() and os.popen() -> > -> > -> > -> > Except that 'os.system' is really easy to use and I use it rarely -> > enough -> > that I *always* have to RTFM for subprocess which makes you -> > jump through -> > a few more (albeit simple) hoops. -> > -> -> > -> So let's add subprocess.system() which takes care of the hoops (but -> > -> still allows you more flexibility through optional keyword -> > -> parameters). -> > -> > How would this differ from subprocess.call()? -> > -> > http://docs.python.org/lib/node530.html -> -> It doesn't implement the system() spec: -> -> -> nd OK, but I'm still confused. This isn't about moving os.system into subprocess, it's about reimplementing os.system *with* subprocess.Popen, right? And how would that be substantially different from call()? Different defaults? (like shell=True, close_fds=False?) --titus From brett at python.org Fri Mar 23 00:57:15 2007 From: brett at python.org (Brett Cannon) Date: Thu, 22 Mar 2007 16:57:15 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070313223629.GA27802@caltech.edu> <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> Message-ID: On 3/22/07, Guido van Rossum wrote: > Sounds good to me. In 3.0 we should probably not have os.popen*(), nor > the popen2 module at all, and do everything via the subprocess module. > I wonder if we should even get rid of os.system(); then there should > be a subprocess.system() instead. And do we even need os.fork(), > os.exec*(), os.spawn*()? > popen2 is already slated for removal in PEP 3108 based on subprocess making is obsolete (as is commands). -Brett > On 3/22/07, Titus Brown wrote: > > Hi all, > > > > I posted about adding 'get_output', 'get_status_output', and > > 'get_status_output_errors' to subprocess here, > > > > http://ivory.idyll.org/blog/mar-07/replacing-commands-with-subprocess > > > > and got some interesting responses. > > > > Briefly, my original proposal was to add these three functions: > > > > output = get_output(cmd, input=None, cwd=None, env=None) > > > > (status, output) = get_status_output(cmd, input=None, cwd=None, env=None) > > (status, output, errout) = get_status_output_errors(cmd, input=None, > > cwd=None, env=None) > > > > Commenters convinced me to propose a few additions. In order of > > estimated plausibility, > > > > * first, all sensical keyword args to subprocess.Popen > > (everything but universal_newlines, stdout, stderr, and bufsize, > > which don't make much sense in the context of the proposed functions) > > should be accepted by the 'get_' functions. > > > > This complicates the function signatures but does make them > > potentially much more useful. > > > > * second, I'd like to add a 'require_success' bool keyword, that is > > by default False (and does nothing in that case). However, when > > True, the functions would emulate check_call, i.e. they would raise > > CalledProcessError when the returncode was not zero. > > > > * third, the 'popen2' module should be deprecated for 2.6. I don't see > > that it has anything in it that subprocess doesn't have. > > > > Thoughts? > > > > --titus > > > > p.s. This has been a fun learning process... ;) > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From greg.ewing at canterbury.ac.nz Fri Mar 23 02:07:55 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 23 Mar 2007 13:07:55 +1200 Subject: [Python-Dev] Adding timeout to socket.py and httplib.py In-Reply-To: References: <4a951aa00703201525m77015cc6nf9714cdd9fc28f9c@mail.gmail.com> <20070320160541.FC6E.JCARLSON@uci.edu> <4a951aa00703201727s101420a4q67aee1deac33e06f@mail.gmail.com> <4a951aa00703220622g71586e5bwd149c920e191acfc@mail.gmail.com> Message-ID: <4603286B.6010203@canterbury.ac.nz> Guido van Rossum wrote: > It's unlikely to be granted. ... The Python call just wraps the > system call which has a similar API. What about letting it accept both? Maintaining strict consistency with the C API here at the cost of causing pain on almost all uses of the function seems to be a case of purity beating practicality. -- Greg From greg.ewing at canterbury.ac.nz Fri Mar 23 02:24:17 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 23 Mar 2007 13:24:17 +1200 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <20070322202636.GA23873@caltech.edu> References: <20070313223629.GA27802@caltech.edu> <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> Message-ID: <46032C41.7010203@canterbury.ac.nz> Titus Brown wrote: > * second, I'd like to add a 'require_success' bool keyword, that is > by default False (and does nothing in that case). However, when > True, the functions would emulate check_call, i.e. they would raise > CalledProcessError when the returncode was not zero. By the no-constant-parameters principle, I think it would be better to have another set of functions for this. They should be named without 'status' in the names, since if the function returns at all, the status is always going to be zero. -- Greg From greg.ewing at canterbury.ac.nz Fri Mar 23 02:29:04 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 23 Mar 2007 13:29:04 +1200 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070313223629.GA27802@caltech.edu> <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> Message-ID: <46032D60.9020404@canterbury.ac.nz> Guido van Rossum wrote: > I wonder if we should even get rid of os.system(); then there should > be a subprocess.system() instead. That sounds okay, since system() isn't actually a system call, despite its name. > And do we even need os.fork(), os.exec*(), os.spawn*()? Since fork() and exec() are directly wrapping actual system calls, I think they should stay in os. I'm not sure about spawn() -- on Unix it's not a direct system call, but it might be on Windows. -- Greg From greg.ewing at canterbury.ac.nz Fri Mar 23 02:44:26 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 23 Mar 2007 13:44:26 +1200 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <20070322214408.GB2577@caltech.edu> References: <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> <4602EBB3.6000109@v.loewis.de> <20070322210324.GK741@snowy.squish.net> <4602F676.1080907@voidspace.org.uk> <20070322214408.GB2577@caltech.edu> Message-ID: <460330FA.20202@canterbury.ac.nz> Titus Brown wrote: > I could add in a 'system'-alike call easily enough; that was suggested. > But I think > > returncode = subprocess.call("program") > > is pretty simple, isn't it? Something to keep in mind is that system() doesn't directly launch a process running the command, it uses a shell. So it's not just simple sugar for some subprocess.* call. -- Greg From glyph at divmod.com Fri Mar 23 03:25:33 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Fri, 23 Mar 2007 02:25:33 -0000 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070313223629.GA27802@caltech.edu> <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> Message-ID: <20070323022533.7769.1013921136.divmod.xquotient.1528@joule.divmod.com> On 22 Mar, 08:38 pm, guido at python.org wrote: >And do we even need os.fork(), os.exec*(), os.spawn*()? Maybe not os.spawn*, but Twisted's spawnProcess is implemented (on UNIX) in terms of fork/exec and I believe it should stay that way. The subprocess module isn't really usable for asynchronous subprocess communication. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070323/2709dc32/attachment.htm From glyph at divmod.com Fri Mar 23 03:36:39 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Fri, 23 Mar 2007 02:36:39 -0000 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <4602F72C.5040901@v.loewis.de> References: <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> <4602EBB3.6000109@v.loewis.de> <20070322210324.GK741@snowy.squish.net> <4602F72C.5040901@v.loewis.de> Message-ID: <20070323023639.7769.1193786111.divmod.xquotient.1541@joule.divmod.com> On 22 Mar, 09:37 pm, martin at v.loewis.de wrote: >>Sure. os.fork() and the os.exec*() family can stay. But os.spawn*(), >>that abomination invented by Microsoft? > >Right, I personally would not miss it. It's also not a system call, >but a library function on both Windows and Unix (the equivalent >of exposing fork would be to expose CreateProcessEx - something >that I think Python should do out of the box, and not just when >PythonWin is installed - but you can now get it through ctypes). >>I also hear no opposition >>against killign os.system() and os.popen(). > >Both are library functions; I can implement them in Python on top >of what is there (plus popen is based on stdio, which we declared >evil). So yes, the can go. In the long term (read: 3k) I think I agree completely. It seems that this is a clear-cut case of TATMWTDI ("there are too many ways to do it") and the subprocess module should satisfy all of these use-cases. I also like Martin's earlier suggestion of calling the remaining OS process-manipulation functions "posix.fork", etc. I think it would be a lot clearer to read and maintain the implementation of subprocess (and asynchronous equivalents, like Twisted's process support) if the platform back-ends were explicitly using APIs in platform-specific modules. The current Twisted implementation misleadingly looks like the UNIX implementation is cross-platform because it uses functions in the "os" module, whereas the Windows implementation uses win32all. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070323/f492cea7/attachment.html From mike.klaas at gmail.com Fri Mar 23 03:38:38 2007 From: mike.klaas at gmail.com (Mike Klaas) Date: Thu, 22 Mar 2007 19:38:38 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <460330FA.20202@canterbury.ac.nz> References: <20070322202636.GA23873@caltech.edu> <4602EBB3.6000109@v.loewis.de> <20070322210324.GK741@snowy.squish.net> <4602F676.1080907@voidspace.org.uk> <20070322214408.GB2577@caltech.edu> <460330FA.20202@canterbury.ac.nz> Message-ID: <3d2ce8cb0703221938h775890ddg9ce2f8c3d8da8938@mail.gmail.com> On 3/22/07, Greg Ewing wrote: > Titus Brown wrote: > > > I could add in a 'system'-alike call easily enough; that was suggested. > > But I think > > > > returncode = subprocess.call("program") > > > > is pretty simple, isn't it? > > Something to keep in mind is that system() doesn't > directly launch a process running the command, it > uses a shell. So it's not just simple sugar for > some subprocess.* call. >>> subprocess.call("ls | grep tmp", shell=True) svn-commit.2.tmp svn-commit.tmp The more important difference is the encoding of the return value: system() has magic to encode signal-related termination of the child process. -Mike From jason.orendorff at gmail.com Fri Mar 23 03:53:21 2007 From: jason.orendorff at gmail.com (Jason Orendorff) Date: Thu, 22 Mar 2007 22:53:21 -0400 Subject: [Python-Dev] minidom and DOM level 2 Message-ID: The lib ref claims that minidom supports DOM Level 1. Does anyone know what parts of Level 2 are not implemented? I wasn't able to find anything offhand. It seems to be more a matter of what's not documented, or what's not covered by the regression tests. So. I'd be happy to do some diffing between the implementation, documentation, tests, and the Recommendation, and submit patches for whatever needs it. If anyone thinks that's worthwhile. Anyone? -j From brett at python.org Fri Mar 23 06:22:06 2007 From: brett at python.org (Brett Cannon) Date: Thu, 22 Mar 2007 22:22:06 -0700 Subject: [Python-Dev] minidom and DOM level 2 In-Reply-To: References: Message-ID: On 3/22/07, Jason Orendorff wrote: > The lib ref claims that minidom supports DOM Level 1. Does anyone > know what parts of Level 2 are not implemented? I wasn't able to find > anything offhand. It seems to be more a matter of what's not > documented, or what's not covered by the regression tests. > > So. I'd be happy to do some diffing between the implementation, > documentation, tests, and the Recommendation, and submit patches for > whatever needs it. If anyone thinks that's worthwhile. Anyone? > It obviously wouldn't hurt to have level 2 compliance. So I am definitely not going to tell you to not do it. =) -Brett From martin at v.loewis.de Fri Mar 23 08:45:54 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 23 Mar 2007 08:45:54 +0100 Subject: [Python-Dev] minidom and DOM level 2 In-Reply-To: References: Message-ID: <460385B2.1030003@v.loewis.de> Jason Orendorff schrieb: > The lib ref claims that minidom supports DOM Level 1. Does anyone > know what parts of Level 2 are not implemented? Most prominently, minidom only implements the Core module of DOM level 2, none of Views, Events, Style, Traversal and Range, or HTML. Whether anything is lacking in DOM Core level 2, I would have to check in detail. Regards, Martin From martin at v.loewis.de Fri Mar 23 09:47:41 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 23 Mar 2007 09:47:41 +0100 Subject: [Python-Dev] minidom and DOM level 2 In-Reply-To: References: Message-ID: <4603942D.4060508@v.loewis.de> Jason Orendorff schrieb: > The lib ref claims that minidom supports DOM Level 1. Does anyone > know what parts of Level 2 are not implemented? I wasn't able to find > anything offhand. I now looked at it closely, and the only thing missing from DOM Level 2 Core (that I could find) is the EntityReference interface, and Document::createEntityReference. I'm not sure what semantics goes with it. Regards, Martin From fredrik at pythonware.com Fri Mar 23 12:15:53 2007 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 23 Mar 2007 12:15:53 +0100 Subject: [Python-Dev] regexp in Python References: <200703211942.03812.b.wolowiec@students.mimuw.edu.pl> Message-ID: Bartlomiej Wolowiec wrote: > For some time I'm interested in regular expressions and Finite State Machine. > Recently, I saw that Python uses "Secret Labs' Regular Expression Engine", > which very often works too slow. Its pesymistic time complexity is O(2^n), > although other solutions, with time complexity O(n*m) ( O(n*m^2), m is the > length of the regular expression and n is the length of the text, > introduction to problem: http://swtch.com/~rsc/regexp/regexp1.html ) that article almost completely ignores all the subtle capturing and left- to-right semantics that a perl-style engine requires, though. trust me, this is a much larger can of worms than you might expect. but if you're ready to open it, feel free to hack away. > major part of regular expressions the contrived example you used has nothing whatsoever to do with "major part of regular expressions" as seen in the wild, though. I'd be much more interested in optimizations that focuses on patterns you've found in real code. From jason.orendorff at gmail.com Fri Mar 23 13:02:39 2007 From: jason.orendorff at gmail.com (Jason Orendorff) Date: Fri, 23 Mar 2007 08:02:39 -0400 Subject: [Python-Dev] minidom and DOM level 2 In-Reply-To: <4603942D.4060508@v.loewis.de> References: <4603942D.4060508@v.loewis.de> Message-ID: On 3/23/07, "Martin v. L?wis" wrote: > Jason Orendorff schrieb: > > The lib ref claims that minidom supports DOM Level 1. Does anyone > > know what parts of Level 2 are not implemented? I wasn't able to find > > anything offhand. > > I now looked at it closely, and the only thing missing from DOM Level > 2 Core (that I could find) is the EntityReference interface, and > Document::createEntityReference. I'm not sure what semantics goes with it. OK, I think this is worthwhile then. :) I'll read the spec and submit a patch. -j From hrvoje.niksic at avl.com Fri Mar 23 13:35:10 2007 From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=) Date: Fri, 23 Mar 2007 13:35:10 +0100 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070313223629.GA27802@caltech.edu> <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> Message-ID: <1174653310.5624.7.camel@localhost> On Thu, 2007-03-22 at 13:38 -0700, Guido van Rossum wrote: > Sounds good to me. In 3.0 we should probably not have os.popen*(), nor > the popen2 module at all, and do everything via the subprocess module. > I wonder if we should even get rid of os.system(); then there should > be a subprocess.system() instead. And do we even need os.fork(), > os.exec*(), os.spawn*()? Please don't remove os.fork and os.exec*. Some people need to fine-tune process creation and don't need portability to non-Unix OS'es. For them, the functions that call the underlying system API and little or nothing else are a god-send. From steven.bethard at gmail.com Fri Mar 23 17:30:37 2007 From: steven.bethard at gmail.com (Steven Bethard) Date: Fri, 23 Mar 2007 10:30:37 -0600 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: <1174653310.5624.7.camel@localhost> References: <20070313223629.GA27802@caltech.edu> <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> <1174653310.5624.7.camel@localhost> Message-ID: On 3/23/07, Hrvoje Nik?i? wrote: > On Thu, 2007-03-22 at 13:38 -0700, Guido van Rossum wrote: > > Sounds good to me. In 3.0 we should probably not have os.popen*(), nor > > the popen2 module at all, and do everything via the subprocess module. > > I wonder if we should even get rid of os.system(); then there should > > be a subprocess.system() instead. And do we even need os.fork(), > > os.exec*(), os.spawn*()? > > Please don't remove os.fork and os.exec*. Some people need to fine-tune > process creation and don't need portability to non-Unix OS'es. For > them, the functions that call the underlying system API and little or > nothing else are a god-send. Right, but if you're really using only Posix, you can simply use ``posix.fork`` and ``posix.exec*`` and then you're even being explicit about the fact. STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From titus at caltech.edu Fri Mar 23 18:37:56 2007 From: titus at caltech.edu (Titus Brown) Date: Fri, 23 Mar 2007 10:37:56 -0700 Subject: [Python-Dev] deprecate commands.getstatus() In-Reply-To: References: <20070313223629.GA27802@caltech.edu> <20070316163704.GC3107@caltech.edu> <20070322202636.GA23873@caltech.edu> <1174653310.5624.7.camel@localhost> Message-ID: <20070323173756.GA12992@caltech.edu> On Fri, Mar 23, 2007 at 10:30:37AM -0600, Steven Bethard wrote: -> On 3/23/07, Hrvoje Nik??i?? wrote: -> > On Thu, 2007-03-22 at 13:38 -0700, Guido van Rossum wrote: -> > > Sounds good to me. In 3.0 we should probably not have os.popen*(), nor -> > > the popen2 module at all, and do everything via the subprocess module. -> > > I wonder if we should even get rid of os.system(); then there should -> > > be a subprocess.system() instead. And do we even need os.fork(), -> > > os.exec*(), os.spawn*()? -> > -> > Please don't remove os.fork and os.exec*. Some people need to fine-tune -> > process creation and don't need portability to non-Unix OS'es. For -> > them, the functions that call the underlying system API and little or -> > nothing else are a god-send. -> -> Right, but if you're really using only Posix, you can simply use -> ``posix.fork`` and ``posix.exec*`` and then you're even being explicit -> about the fact. Yes, but: http://docs.python.org/lib/module-posix.html """Do not import this module directly.""" Unless people want me to try to extract something coherent from the recent discussion, I'm going to avoid doing anything with os.* functions. That can be done separately from the contemplated subprocess patch, anyway. The whole thread is here: http://www.gossamer-threads.com/lists/engine?post=553519;list=python and I will finish up a patch to do this: http://www.gossamer-threads.com/lists/python/dev/555743#555743 (add get_*output* functions to subprocess, modify docs appropriately, add 'require_success', and put in a docs deprecation for popen2/commands). I'll post again when I have a patch ready so there's something concrete to complain about ;). thanks, --titus From jimjjewett at gmail.com Fri Mar 23 18:48:52 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 23 Mar 2007 13:48:52 -0400 Subject: [Python-Dev] [Python-checkins] r54539 - in python/trunk: Lib/string.py Misc/NEWS Objects/typeobject.c In-Reply-To: <20070323045846.0C6001E4005@bag.python.org> References: <20070323045846.0C6001E4005@bag.python.org> Message-ID: (Please note that most replies should trim at least one list from the Cc) On 3/23/07, guido.van.rossum wrote: > Note: without the change to string.py, lots of spurious warnings happen. > What's going on there? I assume it was a defensive measure for subclasses of both Template and X, where X has its own metaclass (which might have an __init__ that shouldn't be ignored). This is where cooperative classes get ugly. You could argue that the "correct" code is to not bother making the super call if it would go all the way to object (or even type), but there isn't a good way to spell that. > + # A super call makes no sense since type() doesn't define __init__(). > + # (Or does it? And should type.__init__() accept three args?) > + # super(_TemplateMetaclass, cls).__init__(name, bases, dct) In this particular case, you could define a type.__init__ that did nothing. (If the signature were wrong, type.__new__ would have already caught it. If __new__ and __init__ are seeing something different, then the change was probably intentional.) The problem isn't really limited to type.__init__ though. You'll sometimes see similar patterns for __del__, close, save, etc. The main difference is that they have to either catch an error or check first, since object doesn't have an implementation of those methods. object.__init__ doesn't really do anything either, except check for errors. Getting rid of it should have the same effect as complaining about parameters. The ideal solution (discussion of which probably ought to stay on python-ideas) might be to replace object.__init__ with some sort of PlaceHolder object that raises an error *unless* called through a cooperative super. This PlaceHolder would also be useful for AbstractBaseClasses/Interfaces. PlaceHolder still wouldn't deal with the original concern of verifying that all arguments had already been stripped and used; but the ABCs might be able to. -jJ > Modified: python/trunk/Lib/string.py > ============================================================================== > --- python/trunk/Lib/string.py (original) > +++ python/trunk/Lib/string.py Fri Mar 23 05:58:42 2007 > @@ -108,7 +108,9 @@ > """ > > def __init__(cls, name, bases, dct): > - super(_TemplateMetaclass, cls).__init__(name, bases, dct) > + # A super call makes no sense since type() doesn't define __init__(). > + # (Or does it? And should type.__init__() accept three args?) > + # super(_TemplateMetaclass, cls).__init__(name, bases, dct) > if 'pattern' in dct: > pattern = cls.pattern > else: From facundo at taniquetil.com.ar Fri Mar 23 18:55:13 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Fri, 23 Mar 2007 17:55:13 +0000 (UTC) Subject: [Python-Dev] Final (final?) update to patch about socket timeout... Message-ID: ...in socket.py and httplib.py, with tests and docs. The patch is #1676823. Basically what I did now is: - Just put a timeout default to None. If None, skip settimeout() call. - Copy the exceptions behaviour that we have actually in the higher level libraries, to be sure we aren't breaking anything. - The function is now public, named "create_connection". Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From mike.klaas at gmail.com Fri Mar 23 18:59:34 2007 From: mike.klaas at gmail.com (Mike Klaas) Date: Fri, 23 Mar 2007 10:59:34 -0700 Subject: [Python-Dev] regexp in Python In-Reply-To: References: <200703211942.03812.b.wolowiec@students.mimuw.edu.pl> Message-ID: <3d2ce8cb0703231059k7c591bcdx5ca5a966e4d42b22@mail.gmail.com> On 3/23/07, Fredrik Lundh wrote: > Bartlomiej Wolowiec wrote: > > > For some time I'm interested in regular expressions and Finite State Machine. > > Recently, I saw that Python uses "Secret Labs' Regular Expression Engine", > > which very often works too slow. Its pesymistic time complexity is O(2^n), > > although other solutions, with time complexity O(n*m) ( O(n*m^2), m is the > > length of the regular expression and n is the length of the text, > > introduction to problem: http://swtch.com/~rsc/regexp/regexp1.html ) > > that article almost completely ignores all the subtle capturing and left- > to-right semantics that a perl-style engine requires, though. trust me, > this is a much larger can of worms than you might expect. but if you're > ready to open it, feel free to hack away. > > > major part of regular expressions > > the contrived example you used has nothing whatsoever to do with > "major part of regular expressions" as seen in the wild, though. I'd > be much more interested in optimizations that focuses on patterns > you've found in real code. A fruitful direction that is not as ambitious as re-writing the entire engine would be to add "independent group" assertions to python's RE syntax [ (?> ... ) in perl]. They are rather handy for optimizing the malperforming cases alluded to here (which rarely occur as the OP posted, but tend to crop up in less malignant forms). -Mike From jason.orendorff at gmail.com Fri Mar 23 19:18:17 2007 From: jason.orendorff at gmail.com (Jason Orendorff) Date: Fri, 23 Mar 2007 14:18:17 -0400 Subject: [Python-Dev] Hindsight on Py_UNICODE_WIDE? Message-ID: Scheme is adding Unicode support in an upcoming standard: (DRAFT) http://www.r6rs.org/document/lib-html/r6rs-lib-Z-H-3.html I have two questions for the python-dev team about Python's Unicode experiences. If it's convenient, please take a moment to reply. Thanks in advance. 1. In hindsight, what do you think about PEP 261, the Py_UNICODE_WIDE build option? On balance, has this been good, bad, or indifferent? What's good/bad about it? 2. The idea of multiple string representations has come up (that is, where all strings are Unicode, but in memory some are 8-bit, some 16-bit, and some 32-bit--each string uses the narrowest possible representation). This has been discussed here for Python 3000. My question is: Is this for real? How far along is it? How likely is it? Thanks, Jason From guido at python.org Fri Mar 23 19:22:48 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 23 Mar 2007 11:22:48 -0700 Subject: [Python-Dev] Final (final?) update to patch about socket timeout... In-Reply-To: References: Message-ID: Looks good. I forget -- can you check this in yourself? If so, do it! If not, let me know and I'll do it for you. Thanks for doing this! --Guido On 3/23/07, Facundo Batista wrote: > ...in socket.py and httplib.py, with tests and docs. > > The patch is #1676823. > > Basically what I did now is: > > - Just put a timeout default to None. If None, skip settimeout() call. > > - Copy the exceptions behaviour that we have actually in the higher > level libraries, to be sure we aren't breaking anything. > > - The function is now public, named "create_connection". > > Regards, > > -- > . Facundo > . > Blog: http://www.taniquetil.com.ar/plog/ > PyAr: http://www.python.org/ar/ > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Mar 23 19:43:21 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 23 Mar 2007 11:43:21 -0700 Subject: [Python-Dev] Hindsight on Py_UNICODE_WIDE? In-Reply-To: References: Message-ID: On 3/23/07, Jason Orendorff wrote: > Scheme is adding Unicode support in an upcoming standard: > (DRAFT) http://www.r6rs.org/document/lib-html/r6rs-lib-Z-H-3.html > > I have two questions for the python-dev team about Python's Unicode > experiences. If it's convenient, please take a moment to reply. > Thanks in advance. > > 1. In hindsight, what do you think about PEP 261, the Py_UNICODE_WIDE > build option? On balance, has this been good, bad, or indifferent? > What's good/bad about it? Don't ask me, I've never thought about it. > 2. The idea of multiple string representations has come up (that is, > where all strings are Unicode, but in memory some are 8-bit, some > 16-bit, and some 32-bit--each string uses the narrowest possible > representation). This has been discussed here for Python 3000. My > question is: Is this for real? How far along is it? How likely is > it? Unlikely to happen at this point; nobody's stepped up to implement this, and the implementation effort would be *huuuuge* -- on top of the already *huuge* effort of unifying str and unicode. (And no, it would not make that effort any easier.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From facundo at taniquetil.com.ar Fri Mar 23 19:55:04 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Fri, 23 Mar 2007 18:55:04 +0000 (UTC) Subject: [Python-Dev] Final (final?) update to patch about socket timeout... References: Message-ID: Guido van Rossum wrote: > Looks good. I forget -- can you check this in yourself? If so, do it! > If not, let me know and I'll do it for you. Thanks for doing this! Done. You're welcome. I'll start now with the patch about the *other* higher level libraries, :) Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From guido at python.org Fri Mar 23 19:59:13 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 23 Mar 2007 11:59:13 -0700 Subject: [Python-Dev] Final (final?) update to patch about socket timeout... In-Reply-To: References: Message-ID: On 3/23/07, Facundo Batista wrote: > Guido van Rossum wrote: > > > Looks good. I forget -- can you check this in yourself? If so, do it! > > If not, let me know and I'll do it for you. Thanks for doing this! > > Done. You're welcome. > > I'll start now with the patch about the *other* higher level libraries, > :) Great -- those should be very easy. If you feel confident you don't have to get them reviewed before you check in. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mal at egenix.com Fri Mar 23 20:04:41 2007 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 23 Mar 2007 20:04:41 +0100 Subject: [Python-Dev] Hindsight on Py_UNICODE_WIDE? In-Reply-To: References: Message-ID: <460424C9.7000500@egenix.com> On 2007-03-23 19:18, Jason Orendorff wrote: > Scheme is adding Unicode support in an upcoming standard: > (DRAFT) http://www.r6rs.org/document/lib-html/r6rs-lib-Z-H-3.html > > I have two questions for the python-dev team about Python's Unicode > experiences. If it's convenient, please take a moment to reply. > Thanks in advance. > > 1. In hindsight, what do you think about PEP 261, the Py_UNICODE_WIDE > build option? On balance, has this been good, bad, or indifferent? > What's good/bad about it? Having narrow and wide builds introduces a level of complexity that seems unnecessary. Few people ever use non-BMP code points and the ones who do can easily get away with UTF-16 surrogates. Most Unixes have chosen to go with UCS4 as storage format, so you have little choice if you want to take advantage of mapping directly to wchar on Unix. Windows has chosen UTF-16 as internal storage format and wchar is 16-bit on that platform. You may also want to consider looking at PEP 263: http://www.python.org/dev/peps/pep-0263 Source code encoding is a great thing ! You can now write native Unicode in Python source code. The only downside is the extra complexity added by the fact that the tokenizer in Py2 works on 8-bit characters. For this reason we had to decode the source code to Unicode, then encode it to UTF-8, pass it to the tokenizer and then decode the UTF-8 literal strings for Unicode back into Unicode again. Ideally, the tokenizer in Py3k should be rewritten to work directly on Unicode. > 2. The idea of multiple string representations has come up (that is, > where all strings are Unicode, but in memory some are 8-bit, some > 16-bit, and some 32-bit--each string uses the narrowest possible > representation). This has been discussed here for Python 3000. My > question is: Is this for real? How far along is it? How likely is > it? My suggestion for Scheme is not to go down that route. It adds complexity for little added value and also makes the implementation slower (due to the frequent conversion from one internal format to another). Can't comment on Py3k - I'm out of that loop. If you want to know more about how Unicode was added to Python 2.x and how it can be used, I suggest you read the following: Unicode integration (one of the first PEPs ever written :-): http://www.python.org/dev/peps/pep-0100 Unicode in Python: http://www.egenix.com/files/python/EuroPython2002-Python-and-Unicode.pdf Designing Unicode-aware Applications in Python: http://www.egenix.com/files/python/EPC2006-Developing-Unicode-aware-applications-in-Python.pdf Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 23 2007) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From facundo at taniquetil.com.ar Fri Mar 23 20:22:54 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Fri, 23 Mar 2007 19:22:54 +0000 (UTC) Subject: [Python-Dev] Final (final?) update to patch about socket timeout... References: Message-ID: Facundo Batista wrote: > Guido van Rossum wrote: > >> Looks good. I forget -- can you check this in yourself? If so, do it! >> If not, let me know and I'll do it for you. Thanks for doing this! > > Done. You're welcome. Tests failed because of this commit *only* in "alpha Tru64 5.1 trunk" buildbot. The test that failed is one that does: sock = socket.create_connection((HOST, PORT), timeout=10) self.assertEqual(sock.gettimeout(), 10) And failed with timeout, obviously because it couldn't connect to itself in 10 seconds. The "connect to itself" part is implemented in the very same way (i.e. copied) that in the rest of the tests, even using the same port and everything. Note that the previous regression test, in this machine, also failed in test_socket, *before* my commit. My questions are: could this machine be so loaded that it couldn't connect in 10 seconds? Or something is happening in this machine/architecture? Shall I try to increment the timeout to 60, in these tests (where actually the timeout can be anything) to see what happens? Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From alexander.belopolsky at gmail.com Fri Mar 23 20:45:51 2007 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 23 Mar 2007 15:45:51 -0400 Subject: [Python-Dev] Proposal: Allow any mapping after ** in calls Message-ID: Python allows arbitrary sequences after * in calls, but an expression following ** must be a (subclass of) dict. I believe * and ** should be treated similarly and since f(*UserList(..)) is valid, f(**UserDict(..)) should be valid as well. Of course, I can work around this limitation by writing f(**dict(UserDict(..))), but this is unnatural given that I don't have to do f(*tuple(UserList(..))). I have submitted a patch on SF that makes Python accept arbitrary mappings after **. See . From python at rcn.com Sat Mar 24 01:25:49 2007 From: python at rcn.com (Raymond Hettinger) Date: Fri, 23 Mar 2007 20:25:49 -0400 (EDT) Subject: [Python-Dev] Exceptions for readonly attributes Message-ID: <20070323202549.BCO34118@ms09.lnh.mail.rcn.net> Re: www.python.org/sf/1687163 I noticed that RO members raise a TypeError upon an attempted write. In contrast, we get an AttributeError when writing to a readonly property or to a readonly method (such as those for builtin types). IMO, the TypeError should really be an AttributeError. However, changing it makes four of the tests fail. The question for you guys is whether or not to make the fix. Raymond From orsenthil at users.sourceforge.net Sat Mar 24 01:35:20 2007 From: orsenthil at users.sourceforge.net (Senthil) Date: Sat, 24 Mar 2007 06:05:20 +0530 Subject: [Python-Dev] RFC - GoogleSOC proposal -cleanupurllib Message-ID: <7c42eba10703231735i30c14d19u52a28322208d715e@mail.gmail.com> Hi All, I have written a proposal to cleanup urllib as part of Google SoC. I am attaching the file 'soc1' with this email. Requesting you to go through the proposal and provide any feedback which I can incorporate in my submission. Thanks, Senthil -- O.R.Senthil Kumaran http://phoe6.livejournal.com -------------- next part -------------- A non-text attachment was scrubbed... Name: soc1 Type: application/octet-stream Size: 9363 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20070324/e6a9efe2/attachment-0001.obj From facundo at taniquetil.com.ar Sat Mar 24 02:32:48 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Sat, 24 Mar 2007 01:32:48 +0000 (UTC) Subject: [Python-Dev] Final (final?) update to patch about socket timeout... References: Message-ID: Facundo Batista wrote: > Tests failed because of this commit *only* in "alpha Tru64 5.1 trunk" buildbot. Also it fails in "g4 osx.4 trunk". In all the other platforms it works ok. > The test that failed is one that does: > > sock = socket.create_connection((HOST, PORT), timeout=10) > self.assertEqual(sock.gettimeout(), 10) Which is very strange, because this is the third test in "testTimeoutAttribute" (this method actually tries different ways of calling the function, no real functionality). In other tests, sometimes fail in the second test of the method. In the test setUp() method, I'm setting the serving socket up, with a listen(1). I can try to to put a listen(5). I can try to reorder the tests. I can try a lot of things to see why there's a problem here. But I do not have access to any of both architectures (Alpha and G4). And I don't want to try everything through commits in the repository (or should I?). I do not want to leave these failing tests. I'll fix them, or comment them out (actually, the tests are very superficial). What do you do in these cases? How do you deal with failing test cases in platforms to which you do not have access? Thanks! -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From guido at python.org Sat Mar 24 05:02:53 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 23 Mar 2007 21:02:53 -0700 Subject: [Python-Dev] Exceptions for readonly attributes In-Reply-To: <20070323202549.BCO34118@ms09.lnh.mail.rcn.net> References: <20070323202549.BCO34118@ms09.lnh.mail.rcn.net> Message-ID: Sounds like something maybe to do in 3.0. On 3/23/07, Raymond Hettinger wrote: > Re: www.python.org/sf/1687163 > > I noticed that RO members raise a TypeError upon an attempted write. In contrast, we get an AttributeError when writing to a readonly property or to a readonly method (such as those for builtin types). > > IMO, the TypeError should really be an AttributeError. However, changing it makes four of the tests fail. > > The question for you guys is whether or not to make the fix. > > > Raymond > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From nyamatongwe at gmail.com Sat Mar 24 07:08:06 2007 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Sat, 24 Mar 2007 17:08:06 +1100 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: <50862ebd0703211913i301de6ceqf1937b60454968bb@mail.gmail.com> References: <50862ebd0703211501p5c0de563s1e1f91b3cb1d2027@mail.gmail.com> <4601B471.1010307@canterbury.ac.nz> <50862ebd0703211913i301de6ceqf1937b60454968bb@mail.gmail.com> Message-ID: <50862ebd0703232308w3df020ard7827340b556c109@mail.gmail.com> I have developed a split vector type that implements the buffer protocol at http://scintilla.sourceforge.net/splitvector-1.0.zip It acts as a mutable string implementing most of the sequence protocol as well as the buffer protocol. splitvector.SplitVector('c') creates a vector containing 8 bit characters and splitvector.SplitVector('u') is for Unicode. A writable attribute bufferAppearence can be set to 0 (default) to respond to buffer protocol calls by moving the gap to the end and returning the address of all of the data. Setting bufferAppearence to 1 responds as a two segment buffer. I haven't found any code that understands responding with two segments. sre and file.write handle SplitVector fine when it responds as a single segment: import re, splitvector x = splitvector.SplitVector("c") x[:] = "The life of brian" r = re.compile("l[a-z]*", re.M) print x y = r.search(x) print y.group(0) x.bufferAppearence = 1 y = r.search(x) print y.group(0) produces The life of brian life Traceback (most recent call last): File "qt.py", line 9, in y = r.search(x) TypeError: expected string or buffer It is likely that adding multi-segment ability to sre would complexify and slow it down. OTOH multi-segment buffers may be well-suited to scatter/gather I/O calls like writev. Neil From orsenthil at gmail.com Sat Mar 24 10:17:39 2007 From: orsenthil at gmail.com (Senthil Kumaran) Date: Sat, 24 Mar 2007 14:47:39 +0530 Subject: [Python-Dev] Fwd: RFC - GoogleSOC proposal -cleanupurllib In-Reply-To: <7c42eba10703231735i30c14d19u52a28322208d715e@mail.gmail.com> References: <7c42eba10703231735i30c14d19u52a28322208d715e@mail.gmail.com> Message-ID: <7c42eba10703240217s413b1e70md953906d5188065e@mail.gmail.com> Resending this; as i think my sf.net email alias got blocked by the python-dev. ---------- Forwarded message ---------- From: Senthil Date: Mar 24, 2007 6:05 AM Subject: RFC - GoogleSOC proposal -cleanupurllib To: python-dev at python.org Hi All, I have written a proposal to cleanup urllib as part of Google SoC. I am attaching the file 'soc1' with this email. Requesting you to go through the proposal and provide any feedback which I can incorporate in my submission. Thanks, Senthil -- O.R.Senthil Kumaran homepage: http://puggy.symonds.net/~senthil blog: http://phoe6.livejournal.com -------------- next part -------------- A non-text attachment was scrubbed... Name: soc1 Type: application/octet-stream Size: 9363 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20070324/f20e8f2e/attachment.obj From martin at v.loewis.de Sat Mar 24 11:45:54 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 24 Mar 2007 11:45:54 +0100 Subject: [Python-Dev] Hindsight on Py_UNICODE_WIDE? In-Reply-To: References: Message-ID: <46050162.5010709@v.loewis.de> > 1. In hindsight, what do you think about PEP 261, the Py_UNICODE_WIDE > build option? On balance, has this been good, bad, or indifferent? > What's good/bad about it? Unlike MAL, I think it was good choice, primarily for political reasons. People kept complaining that Python doesn't "really" support Unicode, and they went silence since this change was made. These days, Linux distributions always ship Python in UCS-4 mode (triggered by the fact that wchar_t is also UCS-4 on Linux); Windows distributions always chose UCS-2 (for the same reason: wchar_t is two bytes). If it weren't for Windows, I would suggest that always using UCS-4 is the simplest solution. FWIW, Smalltalk implementations (VisualWorks and Squeak in particular) seem to go the "multiple internal representations" route, even though they have mutable strings. If you put a wider character into a smaller string, the smaller string become:s transparently wider. HTH, Martin From jcarlson at uci.edu Sat Mar 24 19:25:15 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 24 Mar 2007 11:25:15 -0700 Subject: [Python-Dev] Fwd: RFC - GoogleSOC proposal -cleanupurllib In-Reply-To: <7c42eba10703240217s413b1e70md953906d5188065e@mail.gmail.com> References: <7c42eba10703231735i30c14d19u52a28322208d715e@mail.gmail.com> <7c42eba10703240217s413b1e70md953906d5188065e@mail.gmail.com> Message-ID: <20070324103704.FCD5.JCARLSON@uci.edu> The original went through. You likely didn't get any responses because the proposal has text that is significantly longer than most other SoC proposals, and perhaps people just haven't had the time to read it yet. Also, I don't believe anyone else has posted the full text of their proposal publicly, usually it is "I would like to do this kind of thing, what do people think?". If I were to summarize your proposal, it would be "merge, clean up, and add features to urllib, urllib2, and urlparse". Sounds reasonable for Python 3, but remember that you will need to update any standard library module that currently uses urllib, urllib2, and urlparse to make sure that they pass the test suite, probably add tests to the test suite for your new module, and include documentation (I can't remember if any of that was in your proposal). - Josiah "Senthil Kumaran" wrote: > Resending this; as i think my sf.net email alias got blocked by the python-dev. > > > ---------- Forwarded message ---------- > From: Senthil > Date: Mar 24, 2007 6:05 AM > Subject: RFC - GoogleSOC proposal -cleanupurllib > To: python-dev at python.org > > > Hi All, > I have written a proposal to cleanup urllib as part of Google SoC. I > am attaching the file 'soc1' with this email. Requesting you to go > through the proposal and provide any feedback which I can incorporate > in my submission. From mike at skew.org Sat Mar 24 22:48:04 2007 From: mike at skew.org (Mike Brown) Date: Sat, 24 Mar 2007 15:48:04 -0600 (MDT) Subject: [Python-Dev] Fwd: RFC - GoogleSOC proposal -cleanupurllib In-Reply-To: <7c42eba10703240217s413b1e70md953906d5188065e@mail.gmail.com> Message-ID: <200703242148.l2OLm4J1085642@chilled.skew.org> Senthil Kumaran wrote: > I have written a proposal to cleanup urllib as part of Google SoC. I am > attaching the file 'soc1' with this email. Requesting you to go through the > proposal and provide any feedback which I can incorporate in my submission. >From your proposal: > 2) In all modules, Follow the new RFC 2396 in favour of RFC 1738 and RFC 1808. > [...] > In all modules, follow the new RFC 2396 in favor of RFC 1738, RFC 1808. The > standards for URI described in RFC 2396 is different from older RFCs and > urllib, urllib2 modules implement the URL specifications based on the older > URL specification. This will need changes in urlparse and other parse > modules to handle URLS as specified in the RFC2396. The "new" RFC 2396 was superseded by STD 66 (RFC 3986) two years ago. Your failure to notice this development doesn't bode well :) j/k, although it does undermine confidence somewhat. I think the bugfixes sound great, but major enhancements and API refactorings need to be undertaken more cautiously. In any case, I have a few suggestions: - Read http://en.wikipedia.org/wiki/Uniform_Resource_Identifier. (I wrote the majority of it, and got peer review from the URI WG a while back). - Read http://en.wikipedia.org/wiki/Percent_encoding. (I wrote most of this too). - Familiarize yourself with STD 66. (i.e., don't trust anything I wrote ;)) Especially note its differences from RFC 2396 (summarized in an appendix). - Seek peer review for any changes that you attribute to changing standards. In my experience implementing a general-purpose URI processing library (http://cvs.4suite.org/viewcvs/4Suite/Ft/Lib/Uri.py?view=markup ), there were times when I thought the standard was saying a bit more than it really was, especially when it came to percent-encoding, which has several somewhat-conflicting conventions and standards governing it. I tried to cover these in the Wikipedia article. - Anticipate real-world use cases. If you go down the road of doing what the standards recommend (be aware of "should" vs "must" and whether it's directed at URI producers or consumers), you might lose sight of the fact that there's a reason, for example, people use encodings other than the recommended UTF-8 as the basis for percent-encoding. Similarly, expectations surrounding the behavior of 'file' URIs and path-portions thereof are sometimes less than optimal in the real world. If you're designing an API, be flexible, and seek review for any compatibilities you intend to introduce. - Be aware of the fact that people might have different expectations when they use different string types (unicode, str) in URI processing, and different levels of awareness of the levels of abstraction at which URI processing operates. It can be difficult to uniformly handle unicode and str. And then there's IRIs (RFC 3987)... For additional background, you might also check the python-dev discussion of urllib in Sep 2004, urlparse in Nov 2005, and the competing uriparse.py proposals (Apr, Jun 2006). Mike From l.mastrodomenico at gmail.com Sun Mar 25 05:52:35 2007 From: l.mastrodomenico at gmail.com (Lino Mastrodomenico) Date: Sun, 25 Mar 2007 04:52:35 +0100 Subject: [Python-Dev] SoC proposal: multimedia library Message-ID: Hello everyone, I would like to participate as a student in google Summer of Code and I'm interested in feedback on a multimedia library for Python. The library I propose should have the following features: * the capability to extract and decompress video and audio from a few common multimedia file format; * and, vice versa, it can create a compressed video file from a series of still images and from uncompressed audio; * it should have an interface as simple and pythonic as possible; * it must be cross-platform, easy to extend and not be limited to a single file container format or to some specific audio or video compression format; * at least a subset of the supported formats should be available without any external dependency. Why? I think this can be an useful addition to the standard library: the goal isn't to reimplement a full fledged media player like mplayer or VLC, but to offer, e.g., a very simple way to add multimedia animations to a Tkinter application. Or, for an application that generates animations, a simple way to save them in a common file format that can be played by existing media players. How? I know that this may sound like a project way too big for SoC, but I think it may be feasible: I plan to split it in two parts. A low level part responsible for splitting encoded audio and video data from multimedia files (demuxing) and putting it together in new multimedia files (muxing). I plan to initially only support AVI and MOV/Quicktime files because they are relatively simple and well documented (and I have already written Python programs for manipulating these two formats, so I know them pretty well). New container formats can be added later and, if the library interface meets the requirements, existing application will be able to use them without any modification. The second part of the library is a collection of codecs for compressing and decompressing audio and video streams. I don't plan to invest too much time here: the idea is to write only a few ctypes wrappers around existing libraries for common formats (e.g. MPEG4/XVID and Theora for video and MP3 and Vorbis for audio) and/or use existing third-part Python bindings. Again, new codecs can be added later. Similar libraries. There are already a number of multimedia libraries both inside and outside of the standard library, but AFAIK they all have limitations: * a lot of them are specific for a single container format or codec (e.g. aifc, wave, oggpy, python-vorbis...); * they aren't cross-platform (e.g. videoreader). Is there any interest in a library of this kind (inside or outside of the stdlib)? Suggestions? Criticism? -- Lino Mastrodomenico E-mail: l.mastrodomenico at gmail.com From tjreedy at udel.edu Sun Mar 25 06:49:04 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 25 Mar 2007 00:49:04 -0400 Subject: [Python-Dev] SoC proposal: multimedia library References: Message-ID: "Lino Mastrodomenico" wrote in message news:cc93256f0703242052m2072e9f2ve90ea6632076bd1 at mail.gmail.com... | Is there any interest in a library of this kind (inside or outside of | the stdlib)? | Suggestions? Criticism? There is a Python wrapping of the cross-platform Simple Directmedia Library (C) that is a major part of PyGame. Are you familiar with both and how does your proposal improve on the current situation. tjr From facundo at taniquetil.com.ar Sun Mar 25 06:57:22 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Sun, 25 Mar 2007 04:57:22 +0000 (UTC) Subject: [Python-Dev] Final (final?) update to patch about socket timeout... References: Message-ID: Facundo Batista wrote: >> Tests failed because of this commit *only* in "alpha Tru64 5.1 trunk" buildbot. > > Also it fails in "g4 osx.4 trunk". In all the other platforms it works > ok. As usual, human error. Now I used the already present threading custom testing architecture in test_socket.py, and fixed an issue in httplib.py, and everything is ok. Well, not a big ok, some buildbots fails because other causes, and even the "alpha Tru64" fails in test_socket, but it was failing *before * my check ins, so I'm sure I didn't break anything. This process is finished. I'll continue to work in the other libraries. Sorry for the mess. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From jcarlson at uci.edu Sun Mar 25 10:37:52 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun, 25 Mar 2007 01:37:52 -0700 Subject: [Python-Dev] SoC proposal: multimedia library In-Reply-To: References: Message-ID: <20070325002407.FCE4.JCARLSON@uci.edu> "Lino Mastrodomenico" wrote: > Is there any interest in a library of this kind (inside or outside of > the stdlib)? For decoding, not many packages can currently match VLC. It has wrappers for most major GUI toolkits, and seems to be easily accessable via ctypes. There are also bindings for various media players in wxPython, though they are strictly play only - no screenshots, etc. For encoding, MediaCoder exists, but seems to be Windows only. I would personally suggest limiting support and going for ffmpeg (via ctypes would be quickest, though a SWIG wrapping could make it more convenient). Is it desireable? I don't know. Make it available and we will likely see a slimmed down iMovie clone in a couple months. Is the wrapping of VLC and ffmpeg sufficient for a SoC project? I don't know - I've never wrapped a 3rd party library before. - Josiah From l.mastrodomenico at gmail.com Sun Mar 25 12:22:27 2007 From: l.mastrodomenico at gmail.com (Lino Mastrodomenico) Date: Sun, 25 Mar 2007 12:22:27 +0200 Subject: [Python-Dev] SoC proposal: multimedia library In-Reply-To: <20070325002407.FCE4.JCARLSON@uci.edu> References: <20070325002407.FCE4.JCARLSON@uci.edu> Message-ID: 2007/3/25, Terry Reedy : > There is a Python wrapping of the cross-platform Simple Directmedia Library > (C) that is a major part of PyGame. Are you familiar with both and how > does your proposal improve on the current situation. Yes I know both Pygame and SMPEG (the SDL MPEG library used by Pygame). The differences with my proposal are: * SMPEG is not a general purpose library: it only supports MPEG-1 video in a MPEG system stream container (and the corresponding audio); * it only supports decoding, not encoding; * the Pygame wrapper is strictly tied to Pygame itself and direct playback; e.g.: AFAIK there is no way to access the uncompressed audio track of a MPEG file, you can only "play" a movie; * it's CPython specific: my plan is to use only Python and ctypes, to make it easily portable to PyPy. The nearest existing library to what I propose that I was able to find is PyMedia (http://www.pymedia.org/ ). But it still isn't what I want: their interface is too low level and not exactly pythonic and their implementation is written in C and C++ (a very old fork of the ffmpeg library). 2007/3/25, Josiah Carlson : > For decoding, not many packages can currently match VLC. It has wrappers > for most major GUI toolkits, and seems to be easily accessable via > ctypes. There are also bindings for various media players in wxPython, > though they are strictly play only - no screenshots, etc. Yes, but I don't want to implement yet another media player, my proposal is for a library that supports both encoding and decoding and doesn't force the use of a specific output layer. E.g. you should be able to get the uncompressed data of a video frame and pass it to Tkinter.Image, or PIL (the Python Imaging Library), or do fancy things to it with numpy... > For encoding, MediaCoder exists, but seems to be Windows only. Well, technically it works under Linux too, with Wine. ;-) But, yes, I want something a bit more cross-platform than that. > I would > personally suggest limiting support and going for ffmpeg (via ctypes > would be quickest, though a SWIG wrapping could make it more convenient). Yes, I thought about simply writing a wrapper around ffmpeg, but I think it would be preferable to write at least the (de)muxing part of the library in Python since this should guarantee that at least a subset of the functionality (e.g. an AVI file with MJPEG video and u-law sound) is always available without any dependecy outside of the standard library and of libraries that are pretty much always available on Linux/MacOSX/Windows. Of course the CPU intensive work of encoding and decoding audio and video will be done by existing C libraries. As I said before I have a slight preference for using ctypes instead of the CPython APIs because this means that the library will (probably) be usable unmodified by PyPy, but I can change my mind if the community prefers something else. -- Lino Mastrodomenico E-mail: l.mastrodomenico at gmail.com From facundo at taniquetil.com.ar Sun Mar 25 22:45:37 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Sun, 25 Mar 2007 20:45:37 +0000 (UTC) Subject: [Python-Dev] HTTP responses and errors Message-ID: urllib2.py, after receiving an HTTP response, decides if it was an error and raises an Exception, or it just returns the info. For example, you make ``urllib2.urlopen("http://www.google.com")``. If you receive 200, it's ok; if you receive 500, you get an exception raised. How it decides? Function HTTPErrorProcessor, line 490, actually says: class HTTPErrorProcessor(BaseHandler): ... if code not in (200, 206): # it prepares an error response ... Why only 200 and 206? A coworker of mine found this (he was receiving 202, "Accepted"). In RFC 2616 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html) it says about codes "2xx"... This class of status code indicates that the client's request was successfully received, understood, and accepted. I know it's no difficult to work this around (you have to catch all the exceptions, and check for the code), but I was wondering the reasoning of this. IMHO, "2xx" should not raise an exception. If you also think it's a bug, I can fix it. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From python-dev at aerojockey.com Fri Mar 23 06:53:55 2007 From: python-dev at aerojockey.com (Carl Banks) Date: Fri, 23 Mar 2007 01:53:55 -0400 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: <4603051E.5080409@ee.byu.edu> References: <46027466.8010206@aerojockey.com> <4603051E.5080409@ee.byu.edu> Message-ID: <46036B73.7030006@aerojockey.com> (cc'ing back to Python-dev; the original reply was intended for it by I had an email malfunction.) Travis Oliphant wrote: >Carl Banks wrote: >> 3. Allow getbuffer to return an array of "derefence offsets", one for >> each dimension. For a given dimension i, if derefoff[i] is >> nonnegative, it's assumed that the current position (base pointer + >> indexing so far) is a pointer to a subarray, and derefoff[i] is the >> offest in that array where the current position goes for the next >> dimension. If derefoff[i] is negative, there is no dereferencing. >> Here is an example of how it'd work: > > > This sounds interesting, but I'm not sure I totally see it. I probably > need a picture to figure out what you are proposing. I'll get on it sometime. For now I hope an example will do. > The derefoff > sounds like some-kind of offset. Is that enough? Why not just make > derefoff[i] == 0 instead of negative? I may have misunderstood something. I had thought the values exported by getbuffer could change as the view narrowed, but I'm not sure if it's the case now. I'll assume it isn't for now, because it simplifies things and demonstrates the concept better. Let's start from the beginning. First, change the prototype to this: typedef PyObject *(*getbufferproc)(PyObject *obj, void **buf, Py_ssize_t *len, int *writeable, char **format, int *ndims, Py_ssize_t **shape, Py_ssize_t **strides, int **isptr) "isptr" is a flag indicating whether, for a certain dimension, the positision we've strided to so far is a pointer that should be followed before proceeding with the rest of the strides. Now here's what a general "get_item_pointer" function would look like, given a set of indices: void* get_item_pointer(int ndim, void* buf, Py_ssize_t* strides, Py_ssize_t* derefoff, Py_ssize_t *indices) { char* pointer = (char*)buf; int i; for (i = 0; i < ndim; i++) { pointer += strides[i]*indices[i]; if (isptr[i]) { pointer = *(char**)pointer; } } return (void*)pointer; } > I don't fully understand the PIL example you gave. Yeah. How about more details. Here is a hypothetical image data object structure: struct rgba { unsigned char r, g, b, a; }; struct ImageObject { PyObject_HEAD; ... struct rgba** lines; Py_ssize_t height; Py_ssize_t width; Py_ssize_t shape_array[2]; Py_ssize_t stride_array[2]; Py_ssize_t view_count; }; "lines" points to malloced 1-D array of (struct rgba*). Each pointer in THAT block points to a seperately malloced array of (struct rgba). Got that? In order to access, say, the red value of the pixel at x=30, y=50, you'd use "lines[50][30].r". So what does ImageObject's getbuffer do? Leaving error checking out: PyObject* getbuffer(PyObject *self, void **buf, Py_ssize_t *len, int *writeable, char **format, int *ndims, Py_ssize_t **shape, Py_ssize_t **strides, int **isptr) { static int _isptr[2] = { 1, 0 }; *buf = self->lines; *len = self->height*self->width; *writable = 1; *ndims = 2; self->shape_array[0] = height; self->shape_array[1] = width; *shape = &self->shape_array; self->stride_array[0] = sizeof(struct rgba*); /* yep */ self->stride_array[1] = sizeof(struct rgba); *strides = &self->stride_array; *isptr = _isptr; self->view_count ++; /* create and return view object here, but for what? */ } There are three essential differences from a regular, contiguous array. 1. buf is set to point at the array of pointers, not directly to the data. 2. The isptr thing. isptr[0] is true to indicate that the first dimension is an array of pointers, not the actual data. 3. stride[0] is sizeof(struct rgba*), not self->width*sizeof(struct rgba) like it would be for a contiguous array. This is because your first stride is through an array of pointers, not the data itself. So let's examine what "get_item_pointer" above will do given these values. Once again, we're looking for the pixel at x=30, y=50. First, we set pointer to buf, that is, self->lines. Then we take the first stride: we add index[0]+strides[0], that is, 50*4=200, to poitner. pointer now equals &self->lines[50]. Now, we check isptr[0]. We see that it is true. Thus, the position we've strided to is, in fact, a pointer to a subarray where the actual data is. So we follow it: pointer = *pointer. pointer now equals self->lines[50] which equals &self->lines[50][0]. Next dimension. We take the second stride: we add index[1]+strides[1], that is, 30*4=120, to pointer. pointer now equals &self->lines[50][30]. Now, we check isptr[1]. It's false. No dereferencing this step. We're done. Return pointer. >> By the way, has anyone signed up to modify the standard library >> modules? I could do those when the protocol is finalized. And if >> you're implementing the new buffer protocol in 2.6 (while deprecating >> but not removing the old protocol, I presume), will the modules also >> be updated for 2.6? >> > Nobody has signed up for anything. I'm willing for anyone to help. > Many of the standard library modules will need to be modified. And > yes, I do want to implement the new protocol for 2.6 (adding it to the > current one). Updating the modules for 2.6 would not be high priority > (except the struct module), but is a desirable. Ok, then, consider me available for it. > Thanks for the ideas. Ok, I have two questions, now. First, I'm not sure why getbuffer needs to return a view object. I expect most views of data to be created separately--for instance, a view of an image is likely to be created in Python using something like this: imgview = ImageView(image,(left,right),(top,bottom)) I'd expect the ImageView object would call getbuffer and use the data returned in buf, len, writable, etc., and would have no need for a type-specific view object. Furthermore, I would expect in many cases different views are desirable, and some cases where the viewer is unknown to the exporter. And, if it does have to return a view for some reason, why bother returning buf, len, and friends in the function? Just return those values in the view object. Second question: what happens if a view wants to re-export the buffer? Do the views of the buffer ever change? Example, say you create a transposed view of a Numpy array. Now you want a slice of the transposed array. What does the transposed view's getbuffer export? Naively, I'd expect the "strides" and "shape" array to have rearranged indices, but it looks like you might be trying to get rid of this complexity. The reason I ask is: if things like "buf" and "strides" and "shape" could change when a buffer is re-exported, then it can complicate things for PIL-like buffers. (How would you account for offsets in a dimension that's in a subarray?) Carl Banks From orsenthil at users.sourceforge.net Mon Mar 26 18:59:44 2007 From: orsenthil at users.sourceforge.net (Senthil) Date: Mon, 26 Mar 2007 22:29:44 +0530 Subject: [Python-Dev] Fwd: RFC - GoogleSOC proposal -cleanupurllib In-Reply-To: <20070324103704.FCD5.JCARLSON@uci.edu> References: <7c42eba10703231735i30c14d19u52a28322208d715e@mail.gmail.com> <7c42eba10703240217s413b1e70md953906d5188065e@mail.gmail.com> <20070324103704.FCD5.JCARLSON@uci.edu> Message-ID: <7c42eba10703260959g6ea2052ej910af0e0eeafe0d6@mail.gmail.com> On 3/24/07, Josiah Carlson wrote: > > If I were to summarize your proposal, it would be "merge, clean up, and > add features to urllib, urllib2, and urlparse". Sounds reasonable for > Python 3, but remember that you will need to update any standard library > module that currently uses urllib, urllib2, and urlparse to make sure > that they pass the test suite, probably add tests to the test suite for > your new module, and include documentation (I can't remember if any of > that was in your proposal). Thanks Josiah, I have included the tasks of updating other standard library modules that use urllib, urllib2, and urlparse, including tests and documentation in my proposal. That was a very helpful feedback. I have submitted my proposal in the web app as well. http://puggy.symonds.net/~senthil/summer_code_urllib.txt Thanks, Senthil > "Senthil Kumaran" wrote: > > Resending this; as i think my sf.net email alias got blocked by the python-dev. > > > > > > ---------- Forwarded message ---------- > > From: Senthil > > Date: Mar 24, 2007 6:05 AM > > Subject: RFC - GoogleSOC proposal -cleanupurllib > > To: python-dev at python.org > > > > > > Hi All, > > I have written a proposal to cleanup urllib as part of Google SoC. I > > am attaching the file 'soc1' with this email. Requesting you to go > > through the proposal and provide any feedback which I can incorporate > > in my submission. > > -- O.R.Senthil Kumaran http://phoe6.livejournal.com From oliphant.travis at ieee.org Mon Mar 26 18:49:31 2007 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 26 Mar 2007 09:49:31 -0700 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: <46036B73.7030006@aerojockey.com> References: <46027466.8010206@aerojockey.com> <4603051E.5080409@ee.byu.edu> <46036B73.7030006@aerojockey.com> Message-ID: <4607F99B.20800@ieee.org> Carl Banks wrote: > We're done. Return pointer. Thank you for this detailed example. I will have to parse it in more depth but I think I can see what you are suggesting. > > First, I'm not sure why getbuffer needs to return a view object. The view object in your case would just be the ImageObject. The reason I was thinking the function should return "something" is to provide more flexibility in what a view object actually is. I've also been going back and forth between explicitly passing all this information around or placing it in an actual view-object. In some sense, a view object is a NumPy array in my mind. But, with the addition of isptr we are actually expanding the memory abstraction of the view object beyond an explicit NumPy array. In the most common case, I envisioned the view object would just be the object itself in which case it doesn't actually have to be returned. While returning the view object would allow unspecified flexibilty in the future, it really adds nothing to the current vision. We could add a view object separately as an abstract API on top of the buffer interface. > > > Second question: what happens if a view wants to re-export the buffer? > Do the views of the buffer ever change? Example, say you create a > transposed view of a Numpy array. Now you want a slice of the > transposed array. What does the transposed view's getbuffer export? Basically, you could not alter the internal representation of the object while views which depended on those values were around. In NumPy, a transposed array actually creates a new NumPy object that refers to the same data but has its own shape and strides arrays. With the new buffer protocol, the NumPy array would not be able to alter it's shape/strides/or reallocate its data areas while views were being held by other objects. With the shape and strides information, the format information, and the data buffer itself, there are actually several pieces of memory that may need to be protected because they may be shared with other objects. This makes me wonder if releasebuffer should contain an argument which states whether or not to release the memory, the shape and strides information, the format information, or all of it. Having such a thing as a view object would actually be nice because it could hold on to a particular view of data with a given set of shape and strides (whose memory it owns itself) and then the exporting object would be free to change it's shape/strides information as long as the data did not change. > > The reason I ask is: if things like "buf" and "strides" and "shape" > could change when a buffer is re-exported, then it can complicate things > for PIL-like buffers. (How would you account for offsets in a dimension > that's in a subarray?) I'm not sure what you mean, offsets are handled by changing the starting location of the pointer to the buffer. -Travis From orsenthil at users.sourceforge.net Mon Mar 26 20:00:00 2007 From: orsenthil at users.sourceforge.net (Senthil) Date: Mon, 26 Mar 2007 23:30:00 +0530 Subject: [Python-Dev] Fwd: RFC - GoogleSOC proposal -cleanupurllib In-Reply-To: <200703242148.l2OLm4J1085642@chilled.skew.org> References: <7c42eba10703240217s413b1e70md953906d5188065e@mail.gmail.com> <200703242148.l2OLm4J1085642@chilled.skew.org> Message-ID: <7c42eba10703261100y72bb4f86i97334ab2f094316a@mail.gmail.com> On 3/25/07, Mike Brown wrote: > From your proposal: > > > 2) In all modules, Follow the new RFC 2396 in favour of RFC 1738 and RFC 1808. > > The "new" RFC 2396 was superseded by STD 66 (RFC 3986) two years ago. Your > failure to notice this development doesn't bode well :) j/k, although it does > undermine confidence somewhat. I apologize, Mike. I over-looked it and perhaps did not do sufficient research on the RFCs. I read through the RFC3986 and RFC3987, and have changed the proposal on changes urllib to be compliant with these current RFCs. > In any case, I have a few suggestions: Thank you for your suggestions. I went through your discussions on urllib utf issues and I think a careful study and discussions with people is much neccessary before any implementation thought. 4Suite.org references to RFC3986 comtible methods are helpful. With the comments incorporated, I have submited the proposal: http://puggy.symonds.net/~senthil/summer_code_urllib.txt Thanks, O.R.Senthil Kumaran http://phoe6.livejournal.com From pete at shinners.org Mon Mar 26 23:03:25 2007 From: pete at shinners.org (Pete Shinners) Date: Mon, 26 Mar 2007 21:03:25 +0000 (UTC) Subject: [Python-Dev] SoC proposal: multimedia library References: Message-ID: Lino Mastrodomenico gmail.com> writes: > I would like to participate as a student in google Summer of Code and > I'm interested in feedback on a multimedia library for Python. > > The library I propose should have the following features: > * the capability to extract and decompress video and audio from a > few common multimedia file format; > * and, vice versa, it can create a compressed video file from a > series of still images and from uncompressed audio; > * it should have an interface as simple and pythonic as possible; My main question is what is the image and sound container passed back to Python? This single thing along would be worth a SoC if it could be implemented across all libraries. Will your image objects be transferrable to PIL, Pygame, PyOpengl, Numpy, Pythonmagick, Pycairo, wxPython, etc? It would be best if this could avoid the "fromstring/tostring" mojo that always requires and extra copy of the data for each transfer. If the core numpy arrays could ever get added to the Python standard library, this problem would be mostly solved. I believe there was a SoC for Python last year for this exact array problem. Your proposal sounds dangerously close to implementing some file demuxing by yourself. Do not dare touch any of the file bits in your own library. This proposal should only be for getting data from existing decompression libraries into some general Python container. From pythondev at aerojockey.com Tue Mar 27 00:49:50 2007 From: pythondev at aerojockey.com (Carl Banks) Date: Mon, 26 Mar 2007 18:49:50 -0400 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: <4607F99B.20800@ieee.org> References: <46027466.8010206@aerojockey.com> <4603051E.5080409@ee.byu.edu> <46036B73.7030006@aerojockey.com> <4607F99B.20800@ieee.org> Message-ID: <46084E0E.3030104@aerojockey.com> Travis Oliphant wrote: > Carl Banks wrote: >> We're done. Return pointer. > > Thank you for this detailed example. I will have to parse it in more > depth but I think I can see what you are suggesting. > >> First, I'm not sure why getbuffer needs to return a view object. > > The view object in your case would just be the ImageObject. ITSM that we are using the word "view" very differently. Consider this example: A = zeros((100,100)) B = A.transpose() In this scenario, A would be the exporter object, I think we both would call it that. When I use the word "view", I'm referring to B. However, you seem to be referring to the object returned by A.getbuffer, right? What term have you been using to refer to B? Obviously, it would help the discussion if we could get our terminology straight. (Frankly, I don't agree with your usage; it doesn't agree with other uses of the word "view". For example, consider the proposed Python 3000 dictionary views: D = dict() V = D.items() Here, V is the view, and it's analogous to B in the above example.) I'd suggest the object returned by A.getbuffer should be called the "buffer provider" or something like that. For the sake of discussion, I'm going to avoid the word "view" altogether. I'll call A the exporter, as before. B I'll refer to as the requestor. The object returned by A.getbuffer is the provider. > The reason > I was thinking the function should return "something" is to provide more > flexibility in what a view object actually is. > > I've also been going back and forth between explicitly passing all this > information around or placing it in an actual view-object. In some > sense, a view object is a NumPy array in my mind. But, with the > addition of isptr we are actually expanding the memory abstraction of > the view object beyond an explicit NumPy array. > > In the most common case, I envisioned the view object would just be the > object itself in which case it doesn't actually have to be returned. > While returning the view object would allow unspecified flexibilty in > the future, it really adds nothing to the current vision. > > We could add a view object separately as an abstract API on top of the > buffer interface. Having thought quite a bit about it, and having written several abortive replies, I now understand it and see the importance of it. getbuffer returns the object that you are to call releasebuffer on. It may or may not be the same object as exporter. Makes sense, is easy to explain. It's easy to see possible use cases for returning a different object. A hypothetical future incarnation of NumPy might shift the responsibility of managing buffers from NumPy array object to a hidden raw buffer object. In this scenario, the NumPy object is the exporter, but the raw buffer object the provider. Considering this use case, it's clear that getbuffer should return the shape and stride data independently of the provider. The raw buffer object wouldn't have that information; all it does is store a pointer and keep a reference count. Shape and stride is defined by the exporter. >> Second question: what happens if a view wants to re-export the buffer? >> Do the views of the buffer ever change? Example, say you create a >> transposed view of a Numpy array. Now you want a slice of the >> transposed array. What does the transposed view's getbuffer export? > > Basically, you could not alter the internal representation of the object > while views which depended on those values were around. > > In NumPy, a transposed array actually creates a new NumPy object that > refers to the same data but has its own shape and strides arrays. > > With the new buffer protocol, the NumPy array would not be able to alter > it's shape/strides/or reallocate its data areas while views were being > held by other objects. But requestors could alter their own copies of the data, no? Back to the transpose example: B itself obviously can't use the same "strides" array as A uses. It would have to create its own strides, right? So, what if someone takes a slice out of B? When calling B.getbuffer, does it get B's strides, or A's? I think it should get B's. After all, if you're taking a slice of B, don't you care about the slicing relative to B's axes? I can't really think of a use case for exporting A's stride data when you take a slice of B, and it doesn't seem to simplify memory management, because B has to make it's own copies anyways. > With the shape and strides information, the format information, and the > data buffer itself, there are actually several pieces of memory that may > need to be protected because they may be shared with other objects. > This makes me wonder if releasebuffer should contain an argument which > states whether or not to release the memory, the shape and strides > information, the format information, or all of it. Here's what I think: the lock should only apply to the buffer itself, and not to shape and stride data at all. If the requestor wants to keep its own copies of the data, it would have to malloc its own storage for it. I expect that this would be very rare. As for the provider; I think that's between it the exporter. If the exporter and provider know about each other, they shouldn't have any problems managing memory together. > Having such a thing as a view object would actually be nice because it > could hold on to a particular view of data with a given set of shape and > strides (whose memory it owns itself) and then the exporting object > would be free to change it's shape/strides information as long as the > data did not change. What I don't undestand is why it's important for the provider to retain this data. The requestor only needs the information when it's created; it will calculate its own versions of the data, and will not need the originals again, so no need to the provider to keep them around. Indeed, in the use case I described of the raw buffer object, the provider doesn't even know about the exporter's shape and strides. >> The reason I ask is: if things like "buf" and "strides" and "shape" >> could change when a buffer is re-exported, then it can complicate things >> for PIL-like buffers. (How would you account for offsets in a dimension >> that's in a subarray?) > > I'm not sure what you mean, offsets are handled by changing the starting > location of the pointer to the buffer. Yes, well, the single buffer problem isn't as well solved as I originally thought, so maybe it's best to focus on that first. But to anwser your question: you can't just change the starting location because there's hundreds of buffers. You'd either have to change the starting location of each one of them, which is highly undesirable, or to factor in an offset somehow. (This was, in fact, the point of the "derefoff" term in my original suggestion.) Anyways, despite the miscommunications so far, I now have a very good idea of what's going on. We definitely need to get terms straight. I agree that getbuffer should return an object. I think we need to think harder about the case when requestors re-export the buffer. (Maybe it's time to whip up some experimental objects?) Carl Banks From greg.ewing at canterbury.ac.nz Tue Mar 27 01:09:41 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 27 Mar 2007 11:09:41 +1200 Subject: [Python-Dev] SoC proposal: multimedia library In-Reply-To: References: Message-ID: <460852B5.9040009@canterbury.ac.nz> Pete Shinners wrote: > Will your image objects be transferrable to PIL, Pygame, PyOpengl, Numpy, > Pythonmagick, Pycairo, wxPython, etc? It would be best if this could avoid the > "fromstring/tostring" mojo that always requires and extra copy of the data for > each transfer. This would be an excellent test case for the enhanced buffer protocol that's currently being worked on. I wouldn't worry about this too much initially, though. Just get some kind of library-independent image/sound object working, and a buffer interface can be added later once the details are worked out. -- Greg From greg.ewing at canterbury.ac.nz Tue Mar 27 01:37:49 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 27 Mar 2007 11:37:49 +1200 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: <46084E0E.3030104@aerojockey.com> References: <46027466.8010206@aerojockey.com> <4603051E.5080409@ee.byu.edu> <46036B73.7030006@aerojockey.com> <4607F99B.20800@ieee.org> <46084E0E.3030104@aerojockey.com> Message-ID: <4608594D.9050205@canterbury.ac.nz> Carl Banks wrote: > It's easy to see possible use cases for returning a different object. A > hypothetical future incarnation of NumPy might shift the > responsibility of managing buffers from NumPy array object to a hidden > raw buffer object. In this scenario, the NumPy object is the exporter, > but the raw buffer object the provider. But since the NumPy object has to know about the provider, it can simply pass the release call on to it if appropriate. I don't see how this case necessitates making the release call on a different object. I'm -1 on involving any other objects or returning object references from the buffer interface, unless someone can come up with a use case which actually *requires* this (as opposed to it just being something which might be "nice to have"). The buffer interface should be Blazingly Fast(tm), and messing with PyObject*s is not the way to get that. > > This makes me wonder if releasebuffer should contain an argument which > > states whether or not to release the memory, the shape and strides > > information, the format information, or all of it. > > Here's what I think: the lock should only apply to the buffer itself, > and not to shape and stride data at all. If the requestor wants to keep > its own copies of the data, it would have to malloc its own storage for > it. I expect that this would be very rare. Seems to me the lock should apply to *everything* returned by getbuffer. If the requestor is going to iterate over the data, and there are multiple dimensions, surely it's going to want to refer to the shape and stride info at regular intervals while it's doing that. Requiring it to make its own copy would be a burden. -- Greg From oliphant.travis at ieee.org Tue Mar 27 02:12:57 2007 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 26 Mar 2007 17:12:57 -0700 Subject: [Python-Dev] An updated extended buffer PEP Message-ID: <46086189.60108@ieee.org> Hi Carl and Greg, Here is my updated PEP which incorporates several parts of the discussions we have been having. -Travis From oliphant.travis at ieee.org Tue Mar 27 02:17:17 2007 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 26 Mar 2007 17:17:17 -0700 Subject: [Python-Dev] An updated extended buffer PEP In-Reply-To: <46086189.60108@ieee.org> References: <46086189.60108@ieee.org> Message-ID: <4608628D.80200@ieee.org> Travis Oliphant wrote: > Hi Carl and Greg, > > Here is my updated PEP which incorporates several parts of the > discussions we have been having. > And here is the actual link: http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/pep_buffer.txt > -Travis > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org > From andrew-pythondev at puzzling.org Tue Mar 27 02:29:27 2007 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Tue, 27 Mar 2007 10:29:27 +1000 Subject: [Python-Dev] SoC proposal: multimedia library In-Reply-To: References: Message-ID: <20070327002927.GA9273@steerpike.home.puzzling.org> Lino Mastrodomenico wrote: > Hello everyone, > > I would like to participate as a student in google Summer of Code and > I'm interested in feedback on a multimedia library for Python. > > The library I propose should have the following features: > * the capability to extract and decompress video and audio from a > few common multimedia file format; > * and, vice versa, it can create a compressed video file from a > series of still images and from uncompressed audio; > * it should have an interface as simple and pythonic as possible; > * it must be cross-platform, easy to extend and not be limited to > a single file container format or to some specific audio or video > compression format; > * at least a subset of the supported formats should be available > without any external dependency. Except I guess for the last point, don't the Python bindings for GStreamer already provide this? -Andrew. From l.mastrodomenico at gmail.com Tue Mar 27 03:03:25 2007 From: l.mastrodomenico at gmail.com (Lino Mastrodomenico) Date: Tue, 27 Mar 2007 03:03:25 +0200 Subject: [Python-Dev] Standard Image and Sound classes (was: SoC proposal: multimedia library) Message-ID: 2007/3/26, Pete Shinners : > My main question is what is the image and sound container passed back to Python? > This single thing along would be worth a SoC if it could be implemented across > all libraries. > > Will your image objects be transferrable to PIL, Pygame, PyOpengl, Numpy, > Pythonmagick, Pycairo, wxPython, etc? It would be best if this could avoid the > "fromstring/tostring" mojo that always requires and extra copy of the data for > each transfer. I agree. I withdrew my original "multimedia library" idea and submitted a proposal for the creation of two standard Image and Sound classes. If the PSF accepts my proposal and if Google grants a student slot for me, I will try to implement an interface for the two classes that can work well for everyone and be easily usable from both the Python and the C side. I guess this will require a PEP, if this is the case I will volunteer to write it. And, most important, I will create patches for the appropriate standard library modules and for a lot of external libraries, to try to make the use of the two new classes really interoperable. Finally, as Greg suggested, adding the new buffer interface to the two classes can be a really good idea. I'm following with interest that discussion. Keeping fingers crossed... -- Lino Mastrodomenico E-mail: l.mastrodomenico at gmail.com From pythondev at aerojockey.com Tue Mar 27 03:49:01 2007 From: pythondev at aerojockey.com (Carl Banks) Date: Mon, 26 Mar 2007 21:49:01 -0400 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: <46085952.2010508@ee.byu.edu> References: <46027466.8010206@aerojockey.com> <4603051E.5080409@ee.byu.edu> <46036B73.7030006@aerojockey.com> <4607F99B.20800@ieee.org> <46084E0E.3030104@aerojockey.com> <46085952.2010508@ee.byu.edu> Message-ID: <4608780D.4090407@aerojockey.com> Travis Oliphant wrote: > Carl Banks wrote: > >> Tr >> ITSM that we are using the word "view" very differently. Consider >> this example: >> >> A = zeros((100,100)) >> B = A.transpose() > > > You are thinking of NumPy's particular use case. I'm thinking of a > generic use case. So, yes I'm using the word view in two different > contexts. > > In this scenario, NumPy does not even use the buffer interface. It > knows how to transpose it's own objects and does so by creating a new > NumPy object (with it's own shape and strides space) with a data buffer > pointed to by "A". I realized that as soon as I tried a simple Python demonstration of it. So it's a poor example. But I hope it's obvious how it would generalize to a different type. >>> Having such a thing as a view object would actually be nice because >>> it could hold on to a particular view of data with a given set of >>> shape and strides (whose memory it owns itself) and then the >>> exporting object would be free to change it's shape/strides >>> information as long as the data did not change. >> >> >> What I don't undestand is why it's important for the provider to >> retain this data. The requestor only needs the information when it's >> created; it will calculate its own versions of the data, and will not >> need the originals again, so no need to the provider to keep them around. > > That is certainly a policy we could enforce (and pretty much what I've > been thinking). We just need to make it explicit that the shape and > strides provided is only guaranteed up until a GIL release (i.e. > arbitrary Python code could change these memory areas both their content > and location) and so if you need them later, make your own copies. > > If this were the policy, then NumPy could simply pass pointers to its > stored shape and strides arrays when the buffer interface is called but > then not worry about re-allocating these arrays before the "buffer" lock > is released. Another object could hold on to the memory area of the > NumPy array but would have to store shape and strides information if it > wanted to keep it. > NumPy could also just pass a pointer to the char * representation of the > format (which in NumPy would be stored in the dtype object) and would > not have to worry about the dtype being re-assigned later. Bingo! This is my preference. >>>> The reason I ask is: if things like "buf" and "strides" and "shape" >>>> could change when a buffer is re-exported, then it can complicate >>>> things for PIL-like buffers. (How would you account for offsets in >>>> a dimension that's in a subarray?) >>> >>> >>> I'm not sure what you mean, offsets are handled by changing the >>> starting location of the pointer to the buffer. >> >> >> >> But to anwser your question: you can't just change the starting >> location because there's hundreds of buffers. You'd either have to >> change the starting location of each one of them, which is highly >> undesirable, or to factor in an offset somehow. (This was, in fact, >> the point of the "derefoff" term in my original suggestion.) > > > I get better what your derefoff was doing now. I was missing the > de-referencing that was going on. Couldn't you still just store a > pointer to the start of the array. In other words, isn't your **isptr > suggestion sufficient? It seems to be. No. The problem arises when slicing. In a single buffer, you would adjust the base pointer to point at the element [0,0] of the slice. But you can't do that with multiple buffers. Instead, you have to add an offset after deferencing the pointer to the subarray. Hence my derefoff proposal. It dereferenced the pointer, then added an offset to get you to the 0 position in that dimension. >> Anyways, despite the miscommunications so far, I now have a very good >> idea of what's going on. We definitely need to get terms straight. I >> agree that getbuffer should return an object. I think we need to >> think harder about the case when requestors re-export the buffer. >> (Maybe it's time to whip up some experimental objects?) > > I'm still not clear what you are concerned about. If an object > consumes the buffer interface and then wants to be able to later export > it to another, then from our discussion about the shape/strides and > format information, it would have to maintain it's own copies of these > things, because it could not rely on the original provider (or exporter) > to keep them around once the GIL is released. Right. So, if someone calls getbuffer, it would send its own copies of the buffer information, and not the original exporter's. The values returned by getbuffer can vary for a given buffer, depending on the exporter. Which means the data returned by getbuffer could reflect slicing. Which means the isptr array is not sufficient for the PIL-style multiple buffers. > This is the reason we would have to be very clear about the guaranteed > persistance of the shape/strides and format memory whose pointers are > returned through the proposed buffer interface. > > Thanks for the discussion. It is nice to have someone to talk with > about these things. A conversation always results in a better > implementation. I just hope it wasn't all due to terminological misunderstading. Carl Banks From n.s.buttar at gmail.com Tue Mar 27 06:55:36 2007 From: n.s.buttar at gmail.com (Navtej Singh) Date: Tue, 27 Mar 2007 10:25:36 +0530 Subject: [Python-Dev] SoC proposal: multimedia library In-Reply-To: <20070325002407.FCE4.JCARLSON@uci.edu> References: <20070325002407.FCE4.JCARLSON@uci.edu> Message-ID: <1090e4100703262155l3339435egdd5baa223c847df6@mail.gmail.com> > > I would > personally suggest limiting support and going for ffmpeg (via ctypes > would be quickest, though a SWIG wrapping could make it more convenient). > Python wrapper for FFmpeg using Pyrex is available on google code ( http://code.google.com/p/pyffmpeg/) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070327/1e4ef77f/attachment.html From greg.ewing at canterbury.ac.nz Tue Mar 27 08:33:11 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 27 Mar 2007 18:33:11 +1200 Subject: [Python-Dev] An updated extended buffer PEP In-Reply-To: <4608628D.80200@ieee.org> References: <46086189.60108@ieee.org> <4608628D.80200@ieee.org> Message-ID: <4608BAA7.3080503@canterbury.ac.nz> Travis Oliphant wrote: > > Here is my updated PEP which incorporates several parts of the > > discussions we have been having. It looks pretty good. However, I'm still having trouble seeing what use it is returning a different object from getbuffer. There seems to be no rationale set out for this in the PEP. Can you give me a concrete example of a case where it would be necessary? Also it appears that you're returning a borrowed reference, so if the provider object is not the same as the main object, this would seem to require the main object to keep references to all the provider objects that it has handed out, until releasebuffer has been called on them. This seems very odd to me. -- Greg From greg.ewing at canterbury.ac.nz Tue Mar 27 08:43:42 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 27 Mar 2007 18:43:42 +1200 Subject: [Python-Dev] An updated extended buffer PEP In-Reply-To: <4608628D.80200@ieee.org> References: <46086189.60108@ieee.org> <4608628D.80200@ieee.org> Message-ID: <4608BD1E.3040407@canterbury.ac.nz> Here's another idea, to allow multiple views of the same buffer with different shape/stride info to coexist, but without extra provider objects or refcount weirdness. Also it avoids using calls with a brazillion arguments. struct bufferinfo { void **buf; Py_ssize_t *len; int *writeable; char **format; int *ndims; Py_ssize_t **shape; Py_ssize_t **strides; int **isptr; }; int (*getbuffer)(PyObject *obj, struct bufferinfo *info); int (*releasebuffer)(PyObject *obj, struct bufferinfo *info); If the object has constant shape/stride info, it just fills in the info struct with pointers to its own memory, and does nothing when releasebuffer is called (other than unlocking its buffer). If its shape/stride info can change, it mallocs memory for them and copies them into the info struct. When releasebuffer is called, it frees this memory. It is the responsibility of the consumer to ensure that the base object remains alive until releasebuffer has been called on the info struct (to avoid leaking any memory that has been malloced for shapes/strides). -- Greg From pythondev at aerojockey.com Tue Mar 27 13:56:48 2007 From: pythondev at aerojockey.com (Carl Banks) Date: Tue, 27 Mar 2007 07:56:48 -0400 Subject: [Python-Dev] An updated extended buffer PEP In-Reply-To: <4608628D.80200@ieee.org> References: <46086189.60108@ieee.org> <4608628D.80200@ieee.org> Message-ID: <46090680.2090102@aerojockey.com> Travis Oliphant wrote: > Travis Oliphant wrote: >> Hi Carl and Greg, >> >> Here is my updated PEP which incorporates several parts of the >> discussions we have been having. > > And here is the actual link: > > http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/pep_buffer.txt What's the purpose of void** segments in PyObject_GetBuffer? It seems like it's leftover from an older incarnation? I'd hope after more recent discussion, we'll end up simplifying releasebuffer. It seems like it'd be a nightmare to keep track of what you've released. Finally, the isptr thing. It's just not sufficient. Frankly, I'm having doubts whether it's a good idea to support multibuffer at all. Sure, it brings generality, but I'm thinking its too hard to explain and too hard to get one's head around, and will lead to lots of misunderstanding and bugginess. OTOH, it really doen't burden anyone except those who want to export multi-buffered arrays, and we only have one shot to do it. I just hope it doesn't confuse everyone so much that no one bothers. Here's how I would update the isptr thing. I've changed "derefoff" to "subbufferoffsets" to describe it better. typedef PyObject *(*getbufferproc)(PyObject *obj, void **buf, Py_ssize_t *len, int *writeable, char **format, int *ndims, Py_ssize_t **shape, Py_ssize_t **strides, Py_ssize_t **subbufferoffsets); subbufferoffsets Used to export information about multibuffer arrays. It is an address of a ``Py_ssize_t *`` variable that will be set to point at an array of ``Py_ssize_t`` of length ``*ndims``. [I don't even want to try a verbal description.] To demonstrate how subbufferoffsets works, here is am example of a function that returns a pointer to an element of ANY N-dimensional array, single- or multi-buffered. void* get_item_pointer(int ndim, void* buf, Py_ssize_t* strides, Py_ssize_t* subarrayoffs, Py_ssize_t *indices) { char* pointer = (char*)buf; int i; for (i = 0; i < ndim; i++) { pointer += strides[i]*indices[i]; if (subarraysoffs[i] >= 0) { pointer = *(char**)pointer + subarraysoffs[i]; } } return (void*)pointer; } For single buffers, subbufferoffsets is negative in every dimension and it reduces to normal single-buffer indexing. For multi-buffers, subbufferoffsets indicates when to dereference the pointer and switch to the new buffer, and gives the offset into the buffer to start at. In most cases, the subbufferoffset would be zero (indicating it should start at the beginning of the new buffer), but can be a positive number if the following dimension has been sliced, and thus the 0th entry in that dimension would not be at the beginning of the new buffer. Other than that, looks good. :) Carl Banks From anthony at interlink.com.au Tue Mar 27 14:57:18 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue, 27 Mar 2007 22:57:18 +1000 Subject: [Python-Dev] Standard Image and Sound classes (was: SoC proposal: multimedia library) In-Reply-To: References: Message-ID: <200703272257.19434.anthony@interlink.com.au> On Tuesday 27 March 2007 11:03, Lino Mastrodomenico wrote: > 2007/3/26, Pete Shinners : > > My main question is what is the image and sound container > > passed back to Python? This single thing along would be worth a > > SoC if it could be implemented across all libraries. > > > > Will your image objects be transferrable to PIL, Pygame, > > PyOpengl, Numpy, Pythonmagick, Pycairo, wxPython, etc? It would > > be best if this could avoid the "fromstring/tostring" mojo that > > always requires and extra copy of the data for each transfer. > > I agree. I withdrew my original "multimedia library" idea and > submitted a proposal for the creation of two standard Image and > Sound classes. Ideally you'd hook this into the standard library's existing sound file handling modules. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From martin at v.loewis.de Tue Mar 27 16:48:02 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 27 Mar 2007 16:48:02 +0200 Subject: [Python-Dev] HTTP responses and errors In-Reply-To: References: Message-ID: <46092EA2.3030804@v.loewis.de> > Why only 200 and 206? This kind of question can often be answered through the revision history. If you do 'svn annotate', you see that the line testing for 206 was last changed in r36262. Comparing that to the previous revision, you see that it before said if r.status == 200: and that amk changed it with the log message [Bug #912845] urllib2 only checks for a 200 return code, but 206 is also legal if a Range: header was supplied. (Actually, should the first 'if' statement be modified to allow any 2xx status code?) Going to bugs.python.org/912845, you see that the current form was proposed by Ahmed F. (oneofone), apparently because it crashed for him with 206. In 2006, heresiarch ask the same question that you are now asking and that amk asked before. Going further back, you see that HTTPErrorProcessor (along with the handling of 200) as added by jhylton in 34909, which in turn originates from #852995, where jjlee introduced the handlers in order to support cookie handling. Looking at the change, you see that it is just refactoring; the special-casing of 200 was present before. In earlier versions, the error handling was done using this block: 14267 jhylton if code == 200: 14267 jhylton return addinfourl(fp, hdrs, req.get_full_url()) 14267 jhylton else: 14267 jhylton return self.parent.error('http', req, fp, code, msg, hdrs) You then find that 14267 is the initial revision, checked in with the comment # EXPERIMENTAL # # An extensible library for opening URLs using a variety protocols. # Intended as a replacement for urllib. So it seems that it only tests for 200 and 206 because the experiments never produced a need for anything else. Regards, Martin From facundo at taniquetil.com.ar Tue Mar 27 18:12:06 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Tue, 27 Mar 2007 16:12:06 +0000 (UTC) Subject: [Python-Dev] HTTP responses and errors References: <46092EA2.3030804@v.loewis.de> Message-ID: Martin v. L?wis wrote: >> Why only 200 and 206? > This kind of question can often be answered through the revision > history. If you do 'svn annotate', you see that the line testing > ... > So it seems that it only tests for 200 and 206 because the experiments > never produced a need for anything else. Thanks for this detailed explanation, I learned a lot of how to "discover" the history of a piece of code (didn't know about "annotate"). Regarding the codes themselves: As the tests for 200 and 206 came from just needing them, I think there's no reason to not include the rest of 200. So, in the base that the RFC says that "2xx" codes means that the request was succeded, I think we shouldn't raise an Exception. Right now, it's a bug. Do you think it's safe to fix this or will break much code? Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From phd at phd.pp.ru Tue Mar 27 18:17:36 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Tue, 27 Mar 2007 20:17:36 +0400 Subject: [Python-Dev] HTTP responses and errors In-Reply-To: References: <46092EA2.3030804@v.loewis.de> Message-ID: <20070327161736.GB22481@phd.pp.ru> On Tue, Mar 27, 2007 at 04:12:06PM +0000, Facundo Batista wrote: > (didn't know about "annotate"). It is also known under the name "blame"! ;) Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From martin at v.loewis.de Tue Mar 27 19:12:39 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 27 Mar 2007 19:12:39 +0200 Subject: [Python-Dev] HTTP responses and errors In-Reply-To: References: <46092EA2.3030804@v.loewis.de> Message-ID: <46095087.9050002@v.loewis.de> > Right now, it's a bug. Do you think it's safe to fix this or will break > much code? Who am I to judge whether a fix will break much code? Personally, I think it should treat all 2xx responses as success. Callers can then still check the response code themselves if they need to. Regards, Martin From martin at v.loewis.de Tue Mar 27 19:14:35 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 27 Mar 2007 19:14:35 +0200 Subject: [Python-Dev] HTTP responses and errors In-Reply-To: <20070327161736.GB22481@phd.pp.ru> References: <46092EA2.3030804@v.loewis.de> <20070327161736.GB22481@phd.pp.ru> Message-ID: <460950FB.5070108@v.loewis.de> Oleg Broytmann schrieb: > On Tue, Mar 27, 2007 at 04:12:06PM +0000, Facundo Batista wrote: >> (didn't know about "annotate"). > > It is also known under the name "blame"! ;) Or "praise", depending on your mood :-) Regards, Martin From phd at phd.pp.ru Tue Mar 27 19:17:56 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Tue, 27 Mar 2007 21:17:56 +0400 Subject: [Python-Dev] HTTP responses and errors In-Reply-To: <460950FB.5070108@v.loewis.de> References: <46092EA2.3030804@v.loewis.de> <20070327161736.GB22481@phd.pp.ru> <460950FB.5070108@v.loewis.de> Message-ID: <20070327171756.GC22481@phd.pp.ru> On Tue, Mar 27, 2007 at 07:14:35PM +0200, "Martin v. L?wis" wrote: > Oleg Broytmann schrieb: > > On Tue, Mar 27, 2007 at 04:12:06PM +0000, Facundo Batista wrote: > >> (didn't know about "annotate"). > > > > It is also known under the name "blame"! ;) > > Or "praise", depending on your mood :-) But "blame" is its official primary name! > svn praise blame (praise, annotate, ann): Output the content of specified files or URLs with revision and author information in-line. usage: blame TARGET[@REV]... See? (-: Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From facundo at taniquetil.com.ar Tue Mar 27 19:34:39 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Tue, 27 Mar 2007 17:34:39 +0000 (UTC) Subject: [Python-Dev] HTTP responses and errors References: <46092EA2.3030804@v.loewis.de> <46095087.9050002@v.loewis.de> Message-ID: Martin v. L?wis wrote: > Who am I to judge whether a fix will break much code? Personally, I Sorry, this was an error. I thought "you" as in plural (in spanish there're two different words for third person of plural and singular), and wrote it as is; now, re-reading the parragraph, it's confusing. So, you-people-in-the-list, do you think fix this will be a problem? > think it should treat all 2xx responses as success. Callers can > then still check the response code themselves if they need to. The same I think. If nobody has a conflic with this decission, I'll fix this. Thank you! :) -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From oliphant.travis at ieee.org Tue Mar 27 21:30:51 2007 From: oliphant.travis at ieee.org (Travis E. Oliphant) Date: Tue, 27 Mar 2007 13:30:51 -0600 Subject: [Python-Dev] An updated extended buffer PEP In-Reply-To: <4608BD1E.3040407@canterbury.ac.nz> References: <46086189.60108@ieee.org> <4608628D.80200@ieee.org> <4608BD1E.3040407@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Here's another idea, to allow multiple views of the same > buffer with different shape/stride info to coexist, but > without extra provider objects or refcount weirdness. > Also it avoids using calls with a brazillion arguments. > > struct bufferinfo { > void **buf; > Py_ssize_t *len; > int *writeable; > char **format; > int *ndims; > Py_ssize_t **shape; > Py_ssize_t **strides; > int **isptr; > }; > > int (*getbuffer)(PyObject *obj, struct bufferinfo *info); > > int (*releasebuffer)(PyObject *obj, struct bufferinfo *info); > This is not much different from my original "view" object. Stick a PyObject_HEAD at the start of this bufferinfo and you have it. Memory management was the big reason I wanted to do something like this. I don't see why a PyObject_HEAD would make anything significantly slower. Then we could use Python's memory management very easily to create and destroy these things. This bufferinfo object would become the "provider" I was talking about. > If the object has constant shape/stride info, it just fills > in the info struct with pointers to its own memory, and does > nothing when releasebuffer is called (other than unlocking > its buffer). > > If its shape/stride info can change, it mallocs memory for > them and copies them into the info struct. When releasebuffer > is called, it frees this memory. > > It is the responsibility of the consumer to ensure that the > base object remains alive until releasebuffer has been called > on the info struct (to avoid leaking any memory that has > been malloced for shapes/strides). This is a reasonable design choice. I actually prefer to place all the buffer information in a single object rather than the multiple argument design because it scales better and is easier to explain and understand. -Travis From oliphant.travis at ieee.org Tue Mar 27 21:39:16 2007 From: oliphant.travis at ieee.org (Travis E. Oliphant) Date: Tue, 27 Mar 2007 13:39:16 -0600 Subject: [Python-Dev] An updated extended buffer PEP In-Reply-To: <46090680.2090102@aerojockey.com> References: <46086189.60108@ieee.org> <4608628D.80200@ieee.org> <46090680.2090102@aerojockey.com> Message-ID: Carl Banks wrote: > Travis Oliphant wrote: >> Travis Oliphant wrote: >>> Hi Carl and Greg, >>> >>> Here is my updated PEP which incorporates several parts of the >>> discussions we have been having. >> And here is the actual link: >> >> http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/pep_buffer.txt > > > What's the purpose of void** segments in PyObject_GetBuffer? It seems > like it's leftover from an older incarnation? > Yeah, I forgot to change that location. > I'd hope after more recent discussion, we'll end up simplifying > releasebuffer. It seems like it'd be a nightmare to keep track of what > you've released. Yeah, I agree. I think I'm leaning toward the bufferinfo structure which allows the exporter to copy memory for things that it wants to be free to change while the buffer is exported. > > > Finally, the isptr thing. It's just not sufficient. Frankly, I'm > having doubts whether it's a good idea to support multibuffer at all. > Sure, it brings generality, but I'm thinking its too hard to explain and > too hard to get one's head around, and will lead to lots of > misunderstanding and bugginess. OTOH, it really doen't burden anyone > except those who want to export multi-buffered arrays, and we only have > one shot to do it. I just hope it doesn't confuse everyone so much that > no one bothers. People used to have doubts about explaining strides in NumPy as well. I sure would have hated to see them eliminate the possiblity because of those doubts. I think the addition you discuss is not difficult once you get a hold of it. I also understand now why subbufferoffsets is needed. I was thinking that for slices you would just re-create a whole other array of pointers to contain that addition. But, that is really not advisable. It makes sense when you are talking about a single pointer variable (like in NumPy) but it doesn't when you have an array of pointers. Providing the example about how to extract the pointer from the returned information goes a long way towards clearing up any remaining confusion. Your ImageObject example is also helpful. I really like the addition and think it is clear enough and supports a lot of use cases with very little effort. > > Here's how I would update the isptr thing. I've changed "derefoff" to > "subbufferoffsets" to describe it better. > > > typedef PyObject *(*getbufferproc)(PyObject *obj, void **buf, > Py_ssize_t *len, int *writeable, > char **format, int *ndims, > Py_ssize_t **shape, > Py_ssize_t **strides, > Py_ssize_t **subbufferoffsets); > > > subbufferoffsets > > Used to export information about multibuffer arrays. It is an > address of a ``Py_ssize_t *`` variable that will be set to point at > an array of ``Py_ssize_t`` of length ``*ndims``. > > [I don't even want to try a verbal description.] > > To demonstrate how subbufferoffsets works, here is am example of a > function that returns a pointer to an element of ANY N-dimensional > array, single- or multi-buffered. > > void* get_item_pointer(int ndim, void* buf, Py_ssize_t* strides, > Py_ssize_t* subarrayoffs, Py_ssize_t *indices) { > char* pointer = (char*)buf; > int i; > for (i = 0; i < ndim; i++) { > pointer += strides[i]*indices[i]; > if (subarraysoffs[i] >= 0) { > pointer = *(char**)pointer + subarraysoffs[i]; > } > } > return (void*)pointer; > } > > For single buffers, subbufferoffsets is negative in every dimension > and it reduces to normal single-buffer indexing. What about just having subbufferoffsets be NULL in this case? i.e. you don't need it. If some of the dimensions did not need dereferencing then they would be negative (how about we just say -1 to be explicit)? > For multi-buffers, > subbufferoffsets indicates when to dereference the pointer and switch > to the new buffer, and gives the offset into the buffer to start at. > In most cases, the subbufferoffset would be zero (indicating it should > start at the beginning of the new buffer), but can be a positive > number if the following dimension has been sliced, and thus the 0th > entry in that dimension would not be at the beginning of the new > buffer. > > > > Other than that, looks good. :) > I think we are getting closer. What do you think about Greg's idea of basically making the provider the bufferinfo structure and having the exporter handle copying memory over for shape and strides if it wants to be able to change those before the lock is released. -Travis From aahz at pythoncraft.com Tue Mar 27 21:46:39 2007 From: aahz at pythoncraft.com (Aahz) Date: Tue, 27 Mar 2007 12:46:39 -0700 Subject: [Python-Dev] HTTP responses and errors In-Reply-To: References: <46092EA2.3030804@v.loewis.de> <46095087.9050002@v.loewis.de> Message-ID: <20070327194639.GA27381@panix.com> On Tue, Mar 27, 2007, Facundo Batista wrote: > > Sorry, this was an error. I thought "you" as in plural (in spanish > there're two different words for third person of plural and singular), > and wrote it as is; now, re-reading the parragraph, it's confusing. > > So, you-people-in-the-list, do you think fix this will be a problem? The proper English word for plural "you" is "y'all". ;-) Except for "all y'all". Isn't English fun? -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ Need a book? Use your library! From arigo at tunes.org Tue Mar 27 23:52:07 2007 From: arigo at tunes.org (Armin Rigo) Date: Tue, 27 Mar 2007 23:52:07 +0200 Subject: [Python-Dev] PyPy 1.0: JIT compilers for free and more Message-ID: <20070327215207.GA14315@code0.codespeak.net> Hi all, Sorry for the spamming. I hope this will be of interest to some of you. Armin ========================================== PyPy 1.0: JIT compilers for free and more ========================================== Welcome to the PyPy 1.0 release - a milestone integrating the results of four years of research, engineering, management and sprinting efforts, concluding the 28 months phase of EU co-funding! Although still not mature enough for general use, PyPy 1.0 materializes for the first time the full extent of our original vision: - A flexible Python interpreter, written in "RPython": - Mostly unaware of threading, memory and lower-level target platform aspects. - Showcasing advanced interpreter features and prototypes. - Passing core CPython regression tests, translatable to C, LLVM and .NET. - An advanced framework to translate such interpreters and programs: - That performs whole type-inference on RPython programs. - Can weave in threading, memory and target platform aspects. - Has low level (C, LLVM) and high level (CLI, Java, JavaScript) backends. - A **Just-In-Time Compiler generator** able to **automatically** enhance the low level versions of our Python interpreter, leading to run-time machine code that runs algorithmic examples at speeds typical of JITs! Previous releases, particularly the 0.99.0 release from February, already highlighted features of our Python implementation and the abilities of our translation approach but the **new JIT generator** clearly marks a major research result and gives weight to our vision that one can generate efficient interpreter implementations, starting from a description in a high level language. We have prepared several entry points to help you get started: * The main entry point for JIT documentation and status: http://codespeak.net/pypy/dist/pypy/doc/jit.html * The main documentation and getting-started PyPy entry point: http://codespeak.net/pypy/dist/pypy/doc/index.html * Our online "play1" demos showcasing various Python interpreters, features (and a new way to program AJAX applications): http://play1.codespeak.net/ * Our detailed and in-depth Reports about various aspects of the project: http://codespeak.net/pypy/dist/pypy/doc/index-report.html In the next few months we are going to discuss the goals and form of the next stage of development - now more than ever depending on your feedback and contributions - and we hope you appreciate PyPy 1.0 as an interesting basis for greater things to come, as much as we do ourselves! have fun, the PyPy release team, Samuele Pedroni, Armin Rigo, Holger Krekel, Michael Hudson, Carl Friedrich Bolz, Antonio Cuni, Anders Chrigstroem, Guido Wesdorp Maciej Fijalkowski, Alexandre Fayolle and many others: http://codespeak.net/pypy/dist/pypy/doc/contributor.html What is PyPy? ================================ Technically, PyPy is both a Python interpreter implementation and an advanced compiler, or more precisely a framework for implementing dynamic languages and generating virtual machines for them. The framework allows for alternative frontends and for alternative backends, currently C, LLVM and .NET. For our main target "C", we can can "mix in" different garbage collectors and threading models, including micro-threads aka "Stackless". The inherent complexity that arises from this ambitious approach is mostly kept away from the Python interpreter implementation, our main frontend. PyPy is now also a Just-In-Time compiler generator. The translation framework contains the now-integrated JIT generation technology. This depends only on a few hints added to the interpreter source and should be able to cope with the changes to the interpreter and be generally applicable to other interpreters written using the framework. Socially, PyPy is a collaborative effort of many individuals working together in a distributed and sprint-driven way since 2003. PyPy would not have gotten as far as it has without the coding, feedback and general support from numerous people. Formally, many of the current developers were involved in executing an EU contract with the goal of exploring and researching new approaches to language and compiler development and software engineering. This contract's duration is about to end this month (March 2007) and we are working and preparing the according final review which is scheduled for May 2007. For the future, we are in the process of setting up structures to help maintain conceptual integrity of the project and to discuss and deal with funding opportunities related to further PyPy sprinting and developments. See here for results of the discussion so far: http://codespeak.net/pipermail/pypy-dev/2007q1/003577.html 1.0.0 Feature highlights ============================== Here is a summary list of key features included in PyPy 1.0: - The Just-In-Time compiler generator, now capable of generating the first JIT compiler versions of our Python interpreter: http://codespeak.net/pypy/dist/pypy/doc/jit.html - More Python interpreter optimizations (a CALL_METHOD bytecode, a method cache, rope-based strings), now running benchmarks at around half of CPython's speed (without the JIT): http://codespeak.net/pypy/dist/pypy/doc/interpreter-optimizations.html - The Python interpreter can be translated to .NET and enables interactions with the CLR libraries: http://codespeak.net/pypy/dist/pypy/doc/cli-backend.html http://codespeak.net/pypy/dist/pypy/doc/clr-module.html - Aspect Oriented Programming facilities (based on mutating the Abstract Syntax Tree): http://codespeak.net/pypy/dist/pypy/doc/aspect_oriented_programming.html http://codespeak.net/pypy/extradoc/eu-report/D10.1_Aspect_Oriented_Programming_in_PyPy-2007-03-22.pdf - The JavaScript backend has evolved to a point where it can be used to write AJAX web applications with it. This is still an experimental technique, though. For demo applications which also showcase various generated Python and PROLOG interpreters, see: http://play1.codespeak.net/ - Proxying object spaces and features of our Python interpreter: - Tainting: a 270-line proxy object space tracking and boxing sensitive information within an application. - Transparent proxies: allow the customization of both application and builtin objects from application level code. Now featuring an initial support module (tputil.py) for working with transparent proxies. For a detailed description and discussion of high level backends and Python interpreter features, please see our extensive "D12" report: http://codespeak.net/pypy/extradoc/eu-report/D12.1_H-L-Backends_and_Feature_Prototypes-2007-03-22.pdf Funding partners and organisations ===================================================== PyPy development and activities happen as an open source project and with the support of a consortium partially funded by a 28 month European Union IST research grant for the period from December 2004 to March 2007. The full partners of that consortium are: Heinrich-Heine University (Germany), Open End (Sweden) merlinux GmbH (Germany), tismerysoft GmbH (Germany) Logilab Paris (France), DFKI GmbH (Germany) ChangeMaker (Sweden), Impara (Germany) From aahz at pythoncraft.com Wed Mar 28 00:53:56 2007 From: aahz at pythoncraft.com (Aahz) Date: Tue, 27 Mar 2007 15:53:56 -0700 Subject: [Python-Dev] PyPy 1.0: JIT compilers for free and more In-Reply-To: <20070327215207.GA14315@code0.codespeak.net> References: <20070327215207.GA14315@code0.codespeak.net> Message-ID: <20070327225356.GA27244@panix.com> On Tue, Mar 27, 2007, Armin Rigo wrote: > > Sorry for the spamming. I hope this will be of interest to some of you. This is not spamming, this is wonderful news! Congratulations! -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ Need a book? Use your library! From oliphant.travis at ieee.org Wed Mar 28 01:59:43 2007 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 27 Mar 2007 17:59:43 -0600 Subject: [Python-Dev] An updated extended buffer PEP In-Reply-To: References: <46086189.60108@ieee.org> Message-ID: <4609AFEF.7090905@ieee.org> Lisandro Dalcin wrote: > On 3/26/07, Travis Oliphant wrote: >> Here is my updated PEP which incorporates several parts of the >> discussions we have been having. > > Travis, it looks really good, below my comments I hope you don't mind me replying to python-dev. > > 1- Is it hard to EXTEND PyBufferProcs in order to be able to use all > this machinery in Py 2.X series, not having to wait until Py3k? No, I don't think it will be hard. I just wanted to focus on Py3k since it is going to happen before Python 2.6 and I wanted it discussed in that world. > > 2- Its not clear for me if this PEP will enable object types defined > in the Python side to export buffer info. This is a feature I really > like in numpy, and simplifies my life a lot when I need to export > memory for C/C++ object wrapped with the help of tools like SWIG. This PEP does not address that. You will have to rely on the objects themselves for any such information. > > 3- Why not to constraint the returned 'view' object to be of a > specific type defined in the C side (and perhaps available in the > Python side)? This 'view' object could maintain a reference to the > base object containing the data, could call releasebuffer using the > base object when the view object is decref'ed, and can have a flag > field for think like OWN_MEMORY, OWN_SHAPE, etc in order to properly > manage memory deallocation. Does all this make sense? Yes, that was my original thinking and we are kind of coming back to it after several iterations. Perhaps, though we can stick with an object-less buffer interface but have this "view object" as an expanded buffer object. -Travis From greg.ewing at canterbury.ac.nz Wed Mar 28 02:13:18 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 28 Mar 2007 12:13:18 +1200 Subject: [Python-Dev] An updated extended buffer PEP In-Reply-To: References: <46086189.60108@ieee.org> <4608628D.80200@ieee.org> <4608BD1E.3040407@canterbury.ac.nz> Message-ID: <4609B31E.6040806@canterbury.ac.nz> Travis E. Oliphant wrote: > Greg Ewing wrote: >> struct bufferinfo { >> ... >> }; >> >> int (*getbuffer)(PyObject *obj, struct bufferinfo *info); >> int (*releasebuffer)(PyObject *obj, struct bufferinfo *info); > This is not much different from my original "view" object. Stick a > PyObject_HEAD at the start of this bufferinfo and you have it. The important difference is that it *doesn't* have PyObject_HEAD at the start of it. :-) > I don't see why a PyObject_HEAD would make anything significantly > slower. Then we could use Python's memory management very easily to > create and destroy these things. In the case where the shape/stride info is constant, and the caller is able to allocate the struct bufferinfo on the stack, my proposal requires no memory allocations at all. That's got to be faster than allocating and freeing a Python object. When it is necessary to allocate memory for the shape/stride, some mallocs and frees (or Python equivalents) are going to be needed either way. I don't see how using a Python object makes this any easier. -- Greg From greg.ewing at canterbury.ac.nz Wed Mar 28 02:16:34 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 28 Mar 2007 12:16:34 +1200 Subject: [Python-Dev] An updated extended buffer PEP In-Reply-To: <4609AFEF.7090905@ieee.org> References: <46086189.60108@ieee.org> <4609AFEF.7090905@ieee.org> Message-ID: <4609B3E2.30603@canterbury.ac.nz> Travis Oliphant wrote: > Perhaps, though we can stick with an > object-less buffer interface but have this "view object" as an expanded > buffer object. I like this idea. -- Greg From andrewm at object-craft.com.au Wed Mar 28 02:44:55 2007 From: andrewm at object-craft.com.au (Andrew McNamara) Date: Wed, 28 Mar 2007 10:44:55 +1000 Subject: [Python-Dev] [Fwd: PEP 0305 (small problem with the CSV reader)] In-Reply-To: <4609B159.1030108@object-craft.com.au> References: <4609B159.1030108@object-craft.com.au> Message-ID: <20070328004455.758305CC6DB@longblack.object-craft.com.au> >First of all, let me say thank you for the CSV module. Thanks. >I've been using it and today is the first time I hit a minor bump in the road. >What happened is I opened this file with genome annotations with a >long field and the error "field larger than field limit" showed up. >From what I can see it is in the "static int parse_add_char(ReaderObj >*self, char c)" function. >This function uses the static long field_limit = 128 * 1024; /* max >parsed field size */ >I'm not sure if this is supposed to be recomputed or if there is >something I need to do to change it, but for right now it just says my >row is bigger than 131,072 and stops. >I don't think Python 2.5 has any such string length limitations and >this shouldn't be. This limit was added back in January 2005 to provide some protection against the situation where the parser is returning fields directly from a file, and the file contains a mismatched quote character: this would otherwise result in the entire file being unexpectedly read into memory. You can change the limit with the csv.field_size_limit() method. As you note, it defaults to 128K, but you can set it to anything up to (2**31)-1 or 2147483647 (about 2 billion). BTW, I've taken the liberty of CC'ing this to the python-dev list, so the motivation for this feature is recorded - it caused me some head scratching, and I added it. -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/ From l.mastrodomenico at gmail.com Wed Mar 28 03:39:20 2007 From: l.mastrodomenico at gmail.com (Lino Mastrodomenico) Date: Wed, 28 Mar 2007 03:39:20 +0200 Subject: [Python-Dev] Standard Image and Sound classes (was: SoC proposal: multimedia library) In-Reply-To: <200703272257.19434.anthony@interlink.com.au> References: <200703272257.19434.anthony@interlink.com.au> Message-ID: 2007/3/27, Anthony Baxter : > On Tuesday 27 March 2007 11:03, Lino Mastrodomenico wrote: > > I agree. I withdrew my original "multimedia library" idea and > > submitted a proposal for the creation of two standard Image and > > Sound classes. > > Ideally you'd hook this into the standard library's existing sound > file handling modules. Yes, in my SoC application I listed the following stdlib modules (for both sound and images): Tkinter, Tix, wave, ossaudiodev, winsound, aifc, audioop, sunau, sunaudiodev and imageop. There are also a few Mac modules that can probably use the new classes: videoreader, PixMapWrapper and some Carbon modules. IRIX goodness: al, imgfile, jpeg and gl, but I can't test these. BTW, will they be still here in Python 3.0? -- Lino Mastrodomenico E-mail: l.mastrodomenico at gmail.com From l.mastrodomenico at gmail.com Wed Mar 28 04:02:04 2007 From: l.mastrodomenico at gmail.com (Lino Mastrodomenico) Date: Wed, 28 Mar 2007 04:02:04 +0200 Subject: [Python-Dev] Standard Image and Sound classes (was: SoC proposal: multimedia library) In-Reply-To: References: <200703272257.19434.anthony@interlink.com.au> Message-ID: 2007/3/28, Lino Mastrodomenico : > IRIX goodness: al, imgfile, jpeg and gl, but I can't test these. BTW, > will they be still here in Python 3.0? Ok, I found the answer: PEP 3108. Sorry for the noise. -- Lino Mastrodomenico E-mail: l.mastrodomenico at gmail.com From steve at shrogers.com Wed Mar 28 05:03:50 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Tue, 27 Mar 2007 21:03:50 -0600 Subject: [Python-Dev] PyPy 1.0: JIT compilers for free and more In-Reply-To: <20070327225356.GA27244@panix.com> References: <20070327215207.GA14315@code0.codespeak.net> <20070327225356.GA27244@panix.com> Message-ID: <4609DB16.8020209@shrogers.com> Aahz wrote: > On Tue, Mar 27, 2007, Armin Rigo wrote: >> Sorry for the spamming. I hope this will be of interest to some of you. > > This is not spamming, this is wonderful news! Congratulations! Second the congrats! From facundo at taniquetil.com.ar Wed Mar 28 05:48:46 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Wed, 28 Mar 2007 03:48:46 +0000 (UTC) Subject: [Python-Dev] [ 1688393 ] sock.recvfrom(-24) crashes Message-ID: I applied the patch in this bug to the trunk. As it's a bug, and a very nasty one (it causes an ugly crash), please consider backporting it to 2.5.x. If you apply this to 2.5.x, just close the bug. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From oliphant.travis at ieee.org Wed Mar 28 06:32:26 2007 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 27 Mar 2007 22:32:26 -0600 Subject: [Python-Dev] An updated extended buffer PEP In-Reply-To: <4609D559.30207@aerojockey.com> References: <46086189.60108@ieee.org> <4608628D.80200@ieee.org> <46090680.2090102@aerojockey.com> <4609D559.30207@aerojockey.com> Message-ID: <4609EFDA.4050605@ieee.org> Carl Banks wrote: > Travis E. Oliphant wrote: >> I think we are getting closer. What do you think about Greg's idea >> of basically making the provider the bufferinfo structure and having >> the exporter handle copying memory over for shape and strides if it >> wants to be able to change those before the lock is released. > > It seems like it's just a different way to return the data. You could > do it by setting values through pointers, or do it by returning a > structure. Which way you choose is a minor detail in my opinion. I'd > probably favor returning the information in a structure. > > I would consider adding two fields to the structure: > > size_t structsize; /* size of the structure */ Why is this necessary? can't you get that by sizeof(bufferinfo)? > PyObject* releaser; /* the object you need to call releasebuffer on */ Is this so that another object could be used to manage releases if desired? -Travis From oliphant.travis at ieee.org Wed Mar 28 07:58:18 2007 From: oliphant.travis at ieee.org (Travis E. Oliphant) Date: Tue, 27 Mar 2007 23:58:18 -0600 Subject: [Python-Dev] The latest extended buffer PEP In-Reply-To: <46086189.60108@ieee.org> References: <46086189.60108@ieee.org> Message-ID: <460A03FA.4050902@ieee.org> The latest update is here. Carl and Greg, can I add your names to the PEP author list? I think we are very close. I'd like to start working on the implmentation. The modifications to the struct module is probably where I'll start. I really like the possibilities this will open up for sharing of video, images, audio, databases, between different objects. Algorithms could be written that are object agnostic and work for any object exporting the buffer interface. Are we ready for a pronouncement? -Travis From oliphant.travis at ieee.org Wed Mar 28 07:58:18 2007 From: oliphant.travis at ieee.org (Travis E. Oliphant) Date: Tue, 27 Mar 2007 23:58:18 -0600 Subject: [Python-Dev] The latest extended buffer PEP In-Reply-To: <46086189.60108@ieee.org> References: <46086189.60108@ieee.org> Message-ID: <460A03FA.4050902@ieee.org> The latest update is here. Carl and Greg, can I add your names to the PEP author list? I think we are very close. I'd like to start working on the implmentation. The modifications to the struct module is probably where I'll start. I really like the possibilities this will open up for sharing of video, images, audio, databases, between different objects. Algorithms could be written that are object agnostic and work for any object exporting the buffer interface. Are we ready for a pronouncement? -Travis From steve at holdenweb.com Wed Mar 28 10:38:18 2007 From: steve at holdenweb.com (Steve Holden) Date: Wed, 28 Mar 2007 04:38:18 -0400 Subject: [Python-Dev] HTTP responses and errors In-Reply-To: <20070327194639.GA27381@panix.com> References: <46092EA2.3030804@v.loewis.de> <46095087.9050002@v.loewis.de> <20070327194639.GA27381@panix.com> Message-ID: Aahz wrote: > On Tue, Mar 27, 2007, Facundo Batista wrote: >> Sorry, this was an error. I thought "you" as in plural (in spanish >> there're two different words for third person of plural and singular), >> and wrote it as is; now, re-reading the parragraph, it's confusing. >> >> So, you-people-in-the-list, do you think fix this will be a problem? > > The proper English word for plural "you" is "y'all". ;-) Except for > "all y'all". Isn't English fun? That's not English, it's 'Mer'can. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Recent Ramblings http://holdenweb.blogspot.com From pythondev at aerojockey.com Wed Mar 28 14:29:00 2007 From: pythondev at aerojockey.com (Carl Banks) Date: Wed, 28 Mar 2007 08:29:00 -0400 Subject: [Python-Dev] An updated extended buffer PEP In-Reply-To: <4609EFDA.4050605@ieee.org> References: <46086189.60108@ieee.org> <4608628D.80200@ieee.org> <46090680.2090102@aerojockey.com> <4609D559.30207@aerojockey.com> <4609EFDA.4050605@ieee.org> Message-ID: <460A5F8C.7090504@aerojockey.com> Travis Oliphant wrote: > Carl Banks wrote: >> Travis E. Oliphant wrote: >>> I think we are getting closer. What do you think about Greg's idea >>> of basically making the provider the bufferinfo structure and having >>> the exporter handle copying memory over for shape and strides if it >>> wants to be able to change those before the lock is released. >> It seems like it's just a different way to return the data. You could >> do it by setting values through pointers, or do it by returning a >> structure. Which way you choose is a minor detail in my opinion. I'd >> probably favor returning the information in a structure. >> >> I would consider adding two fields to the structure: >> >> size_t structsize; /* size of the structure */ > Why is this necessary? can't you get that by sizeof(bufferinfo)? In case you want to add something later. Though if you did that, it would be a different major release, meaning you'd have to rebuild anyway. They rashly add fields to the PyTypeObject in the same way. :) So never mind. >> PyObject* releaser; /* the object you need to call releasebuffer on */ > Is this so that another object could be used to manage releases if desired? Yes, that was a use case I saw for a different "view" object. I don't think it's crucially important to have it, but for exporting objects that delegate management of the buffer to another object, then it would be very helpful if the exporter could tell consumers that the other object is managing the buffer. Suppose A is an exporting object, but it uses a hidden object R to manage the buffer memory. Thus you have A referring to R, like this: A -> R Now object B takes a view of A. If we don't have this field, then B will have to hold a reference to A, like this: B -> A -> R A would be responsible for keeping track of views, and A could not be garbage collected until B disappears. If we do have this field, then A could tell be B to hold a reference to R instead: B -> R A -> R A is no longer obliged to keep track of views, and it can be garbage collected even if B still exists. Here's a concrete example of where it would be useful: consider a ByteBufferSlice object. Basically, the object represents a shared-memory slice of a 1-D array of bytes (for example, Python 3000 bytes object, or an mmap object). Now, if the ByteBufferSlice object could tell the consumer that someone else is managing the buffer, then it wouldn't have to keep track of views, thus simplifying things. P.S. In thinking about this, it occurred to me that there should be a way to lock the buffer without requesting details. ByteBufferSlice would already know the details of the buffer, but it would need to increment the original buffer's lock count. Thus, I propose new fuction: typedef int (*lockbufferproc)(PyObject* self); Carl Banks From pythondev at aerojockey.com Wed Mar 28 15:24:29 2007 From: pythondev at aerojockey.com (Carl Banks) Date: Wed, 28 Mar 2007 09:24:29 -0400 Subject: [Python-Dev] An updated extended buffer PEP In-Reply-To: <460A5F8C.7090504@aerojockey.com> References: <46086189.60108@ieee.org> <4608628D.80200@ieee.org> <46090680.2090102@aerojockey.com> <4609D559.30207@aerojockey.com> <4609EFDA.4050605@ieee.org> <460A5F8C.7090504@aerojockey.com> Message-ID: <460A6C8D.6020909@aerojockey.com> Carl Banks wrote: > Here's a concrete example of where it would be useful: consider a > ByteBufferSlice object. Basically, the object represents a > shared-memory slice of a 1-D array of bytes (for example, Python 3000 > bytes object, or an mmap object). > > Now, if the ByteBufferSlice object could tell the consumer that someone > else is managing the buffer, then it wouldn't have to keep track of > views, thus simplifying things. > > P.S. In thinking about this, it occurred to me that there should be a > way to lock the buffer without requesting details. ByteBufferSlice > would already know the details of the buffer, but it would need to > increment the original buffer's lock count. Thus, I propose new fuction: > > typedef int (*lockbufferproc)(PyObject* self); And, because real examples are better than philosophical speculations, here's a skeleton implementation of the ByteBufferSlice array, sans boilerplate and error checking, and with some educated guessing about future details: typedef struct { PyObject_HEAD PyObject* releaser; unsigned char* buf; Py_ssize_t length; } ByteBufferSliceObject; PyObject* ByteBufferSlice_new(PyObject* bufobj, Py_ssize_t start, Py_ssize_t end) { ByteBufferSliceObject* self; BufferInfoObject* bufinfo; self = (ByteBufferSliceObject*)type->tp_alloc(type, 0); bufinfo = PyObject_GetBuffer(bufobj); self->releaser = bufinfo->releaser; self->buf = bufinfo->buf + start; self->length = end-start; /* look how soon we're done with this information */ Py_DECREF(bufinfo); return self; } PyObject* ByteBufferSlice_dealloc(PyObject* self) { PyObject_ReleaseBuffer(self->releaser); self->ob_type->tp_free((PyObject*)self); } PyObject* ByteBufferSlice_getbuffer(PyObject* self, int flags) { BufferInfoObject* bufinfo; static Py_ssize_t stridesarray[] = { 1 }; bufinfo = BufferInfo_New(); bufinfo->releaser = self->releaser; bufinfo->writable = 1; bufinfo->buf = self->buf; bufinfo->length = self->length; bufinfo->ndims = 1; bufinfo->strides = stridesarray; bufinfo->size = &self->length; bufinfo->subbufoffsets = NULL; /* Before we go, increase the original buffer's lock count */ PyObject_LockBuffer(self->releaser); return bufinfo; } /* don't define releasebuffer or lockbuffer */ /* only objects that manage buffers themselves would define these */ /* Now look how easy this is */ /* Everything works out if ByteBufferSlice reexports the buffer */ PyObject* ByteBufferSlice_getslice(PyObject* self, Py_ssize_t start, Py_ssize_t end) { return ByteBufferSlice_new(self,start,end); } The implementation of this is very straightforward, and it's easy to see why and how "bufinfo->release" works, and why it'd be useful. It's almost like there's two protocols here: a buffer exporter protocol (getbuffer) and a buffer manager protocol (lockbuffer and releasebuffer). Some objects would support only exporter protocol; others both. Carl Banks From skip at pobox.com Wed Mar 28 19:27:50 2007 From: skip at pobox.com (skip at pobox.com) Date: Wed, 28 Mar 2007 12:27:50 -0500 Subject: [Python-Dev] Has anyone been in touch with Fred Drake? Message-ID: <17930.42390.715817.20782@montanaro.dyndns.org> I've been seeing bounces for Fred Drake's Comcast email for a couple days. Fred, are you listening? If not, does someone else have a non-Comcast email link to Fred? (I assume his acm.org address is just an alias which might well point to his Comcast mailbox.) Thx, Skip From facundo at taniquetil.com.ar Wed Mar 28 23:14:17 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Wed, 28 Mar 2007 21:14:17 +0000 (UTC) Subject: [Python-Dev] About SSL tests Message-ID: There's this bug (#451607) about the needing of tests for socket SSL... Last interesting update in the tracker is five years ago, and since a lot of work has been done in test_socket_ssl.py (Brett, Neal, Tim, George Brandl). Do you think is useful to leave this bug opened? Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From martin at martinthomas.net Wed Mar 28 23:17:06 2007 From: martin at martinthomas.net (Martin Thomas) Date: Wed, 28 Mar 2007 16:17:06 -0500 Subject: [Python-Dev] Has anyone been in touch with Fred Drake? In-Reply-To: <17930.42390.715817.20782@montanaro.dyndns.org> References: <17930.42390.715817.20782@montanaro.dyndns.org> Message-ID: <1175116626.9677.34.camel@tigger> Down here in Texas, Comcast subscribers were recently moved to Roadrunner.. changing email addresses from, for example, thingy at comcast.net to thingy at tx.rr.com. Other parts of the country were also affected. Is it possible that Fred has been moved also? What part of the country is he in? //Martin On Wed, 2007-03-28 at 12:27 -0500, skip at pobox.com wrote: > I've been seeing bounces for Fred Drake's Comcast email for a couple days. > Fred, are you listening? If not, does someone else have a non-Comcast email > link to Fred? (I assume his acm.org address is just an alias which might > well point to his Comcast mailbox.) > > Thx, > > Skip > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/martin%40martinthomas.net From fdrake at acm.org Wed Mar 28 23:42:02 2007 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed, 28 Mar 2007 17:42:02 -0400 Subject: [Python-Dev] Has anyone been in touch with Fred Drake? In-Reply-To: <1175116626.9677.34.camel@tigger> References: <17930.42390.715817.20782@montanaro.dyndns.org> <1175116626.9677.34.camel@tigger> Message-ID: <200703281742.03098.fdrake@acm.org> On Wednesday 28 March 2007 17:17, Martin Thomas wrote: > Down here in Texas, Comcast subscribers were recently moved to > Roadrunner.. changing email addresses from, for example, > thingy at comcast.net to thingy at tx.rr.com. Other parts of the country were > also affected. Is it possible that Fred has been moved also? What part > of the country is he in? Not to worry; I've been found. (And not in Texas, as it happens.) Comcast has a helpful feature where they save any spam you get, and count it against you. I think I've fixed the settings so they won't continue pushing that mis-feature on me. :-/ I appear to be receiving my Python lists just fine again. -Fred -- Fred L. Drake, Jr. From greg.ewing at canterbury.ac.nz Thu Mar 29 00:41:37 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 29 Mar 2007 10:41:37 +1200 Subject: [Python-Dev] An updated extended buffer PEP In-Reply-To: <460A6C8D.6020909@aerojockey.com> References: <46086189.60108@ieee.org> <4608628D.80200@ieee.org> <46090680.2090102@aerojockey.com> <4609D559.30207@aerojockey.com> <4609EFDA.4050605@ieee.org> <460A5F8C.7090504@aerojockey.com> <460A6C8D.6020909@aerojockey.com> Message-ID: <460AEF21.2040900@canterbury.ac.nz> Carl Banks wrote: > /* don't define releasebuffer or lockbuffer */ > /* only objects that manage buffers themselves would define these */ That's an advantage, but it's a pretty small one -- the releasebuffer implementation would be very simple in this case. I'm bothered that the releaser field makes the protocol asymmetrical and thus harder to reason about. It would cost me more mental effort to convince myself that a releasebuffer implementation wasn't needed in any particular case than it would to write the one-line implementation otherwise required. -- Greg From greg.ewing at canterbury.ac.nz Thu Mar 29 01:28:25 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 29 Mar 2007 11:28:25 +1200 Subject: [Python-Dev] An updated extended buffer PEP In-Reply-To: <460A5F8C.7090504@aerojockey.com> References: <46086189.60108@ieee.org> <4608628D.80200@ieee.org> <46090680.2090102@aerojockey.com> <4609D559.30207@aerojockey.com> <4609EFDA.4050605@ieee.org> <460A5F8C.7090504@aerojockey.com> Message-ID: <460AFA19.2090100@canterbury.ac.nz> Carl Banks wrote: > Now object B takes a view of A. If we don't have this field, then B > will have to hold a reference to A, like this: > > B -> A -> R > > A would be responsible for keeping track of views, A isn't keeping track of views, it's keeping track of the single object R, which it has to keep a reference to anyway. > and A could not be > garbage collected until B disappears. I'm not convinced that this would be a serious problem. An object that's using a different object to manage the buffer is probably quite small, so it doesn't matter much if it stays around. > Here's a concrete example of where it would be useful: consider a > ByteBufferSlice object. Basically, the object represents a > shared-memory slice of a 1-D array of bytes (for example, Python 3000 > bytes object, or an mmap object). And this would be a very small object, not worth the trouble of caring whether it stays around a bit longer than needed, IMO. > P.S. In thinking about this, it occurred to me that there should be a > way to lock the buffer without requesting details. Perhaps you could do this by calling getbuffer with NULL for the bufferinfo pointer, and similarly call releasebuffer with NULL to unlock it. -- Greg From brett at python.org Thu Mar 29 01:38:45 2007 From: brett at python.org (Brett Cannon) Date: Wed, 28 Mar 2007 16:38:45 -0700 Subject: [Python-Dev] About SSL tests In-Reply-To: References: Message-ID: On 3/28/07, Facundo Batista wrote: > There's this bug (#451607) about the needing of tests for socket SSL... > > Last interesting update in the tracker is five years ago, and since a > lot of work has been done in test_socket_ssl.py (Brett, Neal, Tim, > George Brandl). > > Do you think is useful to leave this bug opened? Having a bug left open because a module needs more test is not really needed. It's rather obvious when a module needs more tests. =) I say close it. I just wish we had a more reliable web site to connect to for SSL tests. -Brett From exarkun at divmod.com Thu Mar 29 01:53:39 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Wed, 28 Mar 2007 18:53:39 -0500 Subject: [Python-Dev] About SSL tests In-Reply-To: Message-ID: <20070328235339.18920.1745524514.divmod.quotient.6350@ohm> On Wed, 28 Mar 2007 16:38:45 -0700, Brett Cannon wrote: >On 3/28/07, Facundo Batista wrote: >> There's this bug (#451607) about the needing of tests for socket SSL... >> >> Last interesting update in the tracker is five years ago, and since a >> lot of work has been done in test_socket_ssl.py (Brett, Neal, Tim, >> George Brandl). >> >> Do you think is useful to leave this bug opened? > >Having a bug left open because a module needs more test is not really >needed. It's rather obvious when a module needs more tests. =) > >I say close it. I just wish we had a more reliable web site to >connect to for SSL tests. How about something even better? Take a look at "openssl s_server". This is still a pretty terrible way to test the SSL functionality, but it's loads better than connecting to a site on the public internet. Jean-Paul From facundo at taniquetil.com.ar Thu Mar 29 02:22:23 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Thu, 29 Mar 2007 00:22:23 +0000 (UTC) Subject: [Python-Dev] About SSL tests References: <20070328235339.18920.1745524514.divmod.quotient.6350@ohm> Message-ID: Jean-Paul Calderone wrote: > Take a look at "openssl s_server". This is still a pretty terrible way > to test the SSL functionality, but it's loads better than connecting to > a site on the public internet. How would you deal with the deployment and maintenance of this server in all buildbot's machines? Or we just can ask to see if we have the server available, and then run the tests if yes? Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From exarkun at divmod.com Thu Mar 29 02:40:57 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Wed, 28 Mar 2007 19:40:57 -0500 Subject: [Python-Dev] About SSL tests In-Reply-To: Message-ID: <20070329004057.18920.331373336.divmod.quotient.6356@ohm> On Thu, 29 Mar 2007 00:22:23 +0000 (UTC), Facundo Batista wrote: >Jean-Paul Calderone wrote: > >> Take a look at "openssl s_server". This is still a pretty terrible way >> to test the SSL functionality, but it's loads better than connecting to >> a site on the public internet. > >How would you deal with the deployment and maintenance of this server in >all buildbot's machines? > >Or we just can ask to see if we have the server available, and then run >the tests if yes? > If the openssl binary is available, when the test starts, launch it in a child process, talk to it for the test, then kill it when the test is done. Jean-Paul From greg.ewing at canterbury.ac.nz Thu Mar 29 11:37:40 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 29 Mar 2007 21:37:40 +1200 Subject: [Python-Dev] Distutils and -framework on MacOSX Message-ID: <460B88E4.4060204@canterbury.ac.nz> Has anyone found a way of persuading distutils to pass MacOSX -framework options properly when linking? Using extra_link_args doesn't work, because they need to be at the beginning of the command line to work properly, but extra_link_args gets put at the end. And extra_compile_args is only used when compiling, not linking. -- Greg From ronaldoussoren at mac.com Thu Mar 29 11:55:54 2007 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 29 Mar 2007 11:55:54 +0200 Subject: [Python-Dev] Distutils and -framework on MacOSX In-Reply-To: <460B88E4.4060204@canterbury.ac.nz> References: <460B88E4.4060204@canterbury.ac.nz> Message-ID: <32AD8D7B-249A-4940-A316-8B4228FACAB0@mac.com> On 29 Mar, 2007, at 11:37, Greg Ewing wrote: > Has anyone found a way of persuading distutils to > pass MacOSX -framework options properly when linking? > > Using extra_link_args doesn't work, because they need > to be at the beginning of the command line to work > properly, but extra_link_args gets put at the end. > And extra_compile_args is only used when compiling, > not linking. What's wrong with adding -framework flags to the end? I do this all the time and have yet to run into problems. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3562 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20070329/3cf41e37/attachment.bin From jimjjewett at gmail.com Thu Mar 29 18:02:27 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 29 Mar 2007 12:02:27 -0400 Subject: [Python-Dev] A simplified extended buffer PEP Message-ID: Greg Ewing wrote: > Carl Banks wrote: > > /* don't define releasebuffer or lockbuffer */ > > /* only objects that manage buffers themselves would define these */ > That's an advantage, but it's a pretty small one ... > the releaser field makes the protocol asymmetrical > and thus harder to reason about. Instead of "releaser", how about an "owner" field, that points to a PyObject? PyObject_GetBuffer will create a new reference to owner, and unlock/release is just a DECREF on that same object. (Or does redirecting the DECREF target like this look too magical?) If a buffer exporter want to keep things simple, it can just set "owner" to self. If it has fancier needs (such as mutating the buffer without mutating the view), then it can create a manager to do whatever it wants, and set that as the owner. Note that the PyBufferProcs structure isn't needed any more -- from a consumer/requestor's perspective, the unlock/release is just DECREFing the owner. Looking at http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/pep_buffer.txt (Other than utility functions like PyObject_SizeFromFormat,) I think the entire protocol can then collapse to the bufferinfo struct and either // might set an error and return NULL struct bufferinfo *PyObject_GetBuffer(PyObject *obj, int flags) or // Replace the *view elements with the actual buffers metadata // (return = 0) ==> success // (return > 0) ==> modified (perhaps rejecting the RO argument)? // (return < 0) ==> failure int PyObject_GetBuffer(PyObject *obj, struct bufferinfo *view) -jJ From draghuram at gmail.com Thu Mar 29 20:45:07 2007 From: draghuram at gmail.com (Raghuram Devarakonda) Date: Thu, 29 Mar 2007 14:45:07 -0400 Subject: [Python-Dev] Should TemporaryFile() return a wrapper like NamedTemporaryFile()? Message-ID: <2c51ecee0703291145g6a3442beu84704f41e5b54b94@mail.gmail.com> Hi, I am looking into the issue http://python.org/sf/1615275 (suggested by Neal) and added a comment there. I would appreciate it if some one can check it out and comment on it. Basically, I am wondering if TemporaryFile() should return a wrapper instance (with "file" member) just like NamedTemporaryFile(). It shouldn't effect any existing code (unless I am missing something) but should help in cross-platform code. Thanks, Raghu. From collinw at gmail.com Thu Mar 29 21:50:14 2007 From: collinw at gmail.com (Collin Winter) Date: Thu, 29 Mar 2007 14:50:14 -0500 Subject: [Python-Dev] Possible bugs in test_struct Message-ID: <43aa6ff70703291250u1d576258xf21b809201343405@mail.gmail.com> While converting test_struct to use unittest, I came across these two issues: 1) r51119 by Bob Ippolito added a test case for "SF bug 1530559: struct.pack raises TypeError where it used to convert." (Handy diff at http://svn.python.org/view/python/trunk/Lib/test/test_struct.py?rev=51119&r1=46679&r2=51119). As far as I can tell, this test doesn't work, thanks to the lines check_float_coerce = with_warning_restore(deprecated_err) def test_1530559(): for endian in ('', '>', '<'): for fmt in ('B', 'H', 'I', 'L', 'b', 'h', 'i', 'l'): check_float_coerce(endian + fmt, 1.0) check_float_coerce(endian + fmt, 1.5) The problem is that the calls to check_float_coerce() attempt to call the string produced by "endian + fmt", so a TypeError will always be raised. Fixing the test to do what was actually intended causes a TestFailed error to be raised with the message "did not raise error for float coerce". I'm confused as to why this is a test failure, when it seems like "rais[ing an] error for float coerce" is exactly what we're trying to avoid. Anyone know what's going on here? 2) test_705836() features the code big = (1 << 25) - 1 big = math.ldexp(big, 127 - 24) try: packed = struct.pack(">f", big) except OverflowError: pass else: TestFailed("expected OverflowError") Actually raising the TestFailed exception (as opposed to just instantiating it) causes a test failure. Is this a bug in struct or a bad test? Thanks, Collin Winter From draghuram at gmail.com Thu Mar 29 23:28:38 2007 From: draghuram at gmail.com (Raghuram Devarakonda) Date: Thu, 29 Mar 2007 16:28:38 -0500 Subject: [Python-Dev] Should TemporaryFile() return a wrapper like NamedTemporaryFile()? In-Reply-To: <2c51ecee0703291145g6a3442beu84704f41e5b54b94@mail.gmail.com> References: <2c51ecee0703291145g6a3442beu84704f41e5b54b94@mail.gmail.com> Message-ID: <2c51ecee0703291428g776dbac6ob8b9d4c67f6d3fce@mail.gmail.com> On 3/29/07, Raghuram Devarakonda wrote: > Hi, > > I am looking into the issue http://python.org/sf/1615275 (suggested by > Neal) and added a comment there. I would appreciate it if some one can > check it out and comment on it. Basically, I am wondering if > TemporaryFile() should return a wrapper instance (with "file" member) > just like NamedTemporaryFile(). It shouldn't effect any existing code > (unless I am missing something) but should help in cross-platform On second thoughts, this change will break existing usages such as array.tofile(TemporaryFile()). Raghu From fdrake at acm.org Thu Mar 29 23:40:23 2007 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu, 29 Mar 2007 17:40:23 -0400 Subject: [Python-Dev] Italic text in the manual In-Reply-To: <45F012D9.5040309@v.loewis.de> References: <740c3aec0703080520r3237a403rba0046f575c0b5bd@mail.gmail.com> <45F012D9.5040309@v.loewis.de> Message-ID: <200703291740.23431.fdrake@acm.org> On Thursday 08 March 2007 08:42, Martin v. L?wis wrote: > Certainly not. In today's copy (8.3.07, 13:30 GMT), this starts > between 18.17 and 18.17.1. However, looking at the tex, I cannot > find anything suspicious. texcheck complains about a missing ), > which I added, but it only was a problem of the text, not of the > markup. This doesn't seem to be a problem any more, so I'm going to presume Martin fixed it. ;-) -Fred -- Fred L. Drake, Jr. From collinw at gmail.com Thu Mar 29 23:48:19 2007 From: collinw at gmail.com (Collin Winter) Date: Thu, 29 Mar 2007 16:48:19 -0500 Subject: [Python-Dev] Italic text in the manual In-Reply-To: <200703291740.23431.fdrake@acm.org> References: <740c3aec0703080520r3237a403rba0046f575c0b5bd@mail.gmail.com> <45F012D9.5040309@v.loewis.de> <200703291740.23431.fdrake@acm.org> Message-ID: <43aa6ff70703291448v37c70244jee8d77bbc9654e1c@mail.gmail.com> On 3/29/07, Fred L. Drake, Jr. wrote: > On Thursday 08 March 2007 08:42, Martin v. L?wis wrote: > > Certainly not. In today's copy (8.3.07, 13:30 GMT), this starts > > between 18.17 and 18.17.1. However, looking at the tex, I cannot > > find anything suspicious. texcheck complains about a missing ), > > which I added, but it only was a problem of the text, not of the > > markup. > > This doesn't seem to be a problem any more, so I'm going to presume Martin > fixed it. ;-) The docs for atexit in py3k [1] are mostly (though not all) in italics; I can't figure out why, and I'd appreciate if anyone with stronger latex-foo could take a look. Thanks, Collin Winter [1] - http://docs.python.org/dev/3.x/lib/module-atexit.html From g.brandl at gmx.net Thu Mar 29 23:54:09 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 29 Mar 2007 23:54:09 +0200 Subject: [Python-Dev] Italic text in the manual In-Reply-To: <43aa6ff70703291448v37c70244jee8d77bbc9654e1c@mail.gmail.com> References: <740c3aec0703080520r3237a403rba0046f575c0b5bd@mail.gmail.com> <45F012D9.5040309@v.loewis.de> <200703291740.23431.fdrake@acm.org> <43aa6ff70703291448v37c70244jee8d77bbc9654e1c@mail.gmail.com> Message-ID: Collin Winter schrieb: > On 3/29/07, Fred L. Drake, Jr. wrote: >> On Thursday 08 March 2007 08:42, Martin v. L?wis wrote: >> > Certainly not. In today's copy (8.3.07, 13:30 GMT), this starts >> > between 18.17 and 18.17.1. However, looking at the tex, I cannot >> > find anything suspicious. texcheck complains about a missing ), >> > which I added, but it only was a problem of the text, not of the >> > markup. >> >> This doesn't seem to be a problem any more, so I'm going to presume Martin >> fixed it. ;-) > > The docs for atexit in py3k [1] are mostly (though not all) in > italics; I can't figure out why, and I'd appreciate if anyone with > stronger latex-foo could take a look. This is still the same error as in the trunk; the fix hasn't been forward-ported yet. Georg From mike.klaas at gmail.com Fri Mar 30 00:16:02 2007 From: mike.klaas at gmail.com (Mike Klaas) Date: Thu, 29 Mar 2007 15:16:02 -0700 Subject: [Python-Dev] Italic text in the manual In-Reply-To: References: <740c3aec0703080520r3237a403rba0046f575c0b5bd@mail.gmail.com> <45F012D9.5040309@v.loewis.de> <200703291740.23431.fdrake@acm.org> <43aa6ff70703291448v37c70244jee8d77bbc9654e1c@mail.gmail.com> Message-ID: <3d2ce8cb0703291516r48ff7698h792eb659f7cdcd0f@mail.gmail.com> On 3/29/07, Georg Brandl wrote: > Collin Winter schrieb: > > The docs for atexit in py3k [1] are mostly (though not all) in > > italics; I can't figure out why, and I'd appreciate if anyone with > > stronger latex-foo could take a look. > > This is still the same error as in the trunk; the fix hasn't been forward-ported > yet. In the interest of expanding the general level of latex fu, the difference stems from \em version \emph: \em switches the current font to emphasized in the current scope, whereas \emph is a command that formats its argument emphasizedly. Thus {\em ... } and \emph{...} achieve similar results, but \em{ ... } switches the top-level font, which is followed by a grouping {} (which likley does nothing). -Mike From fdrake at acm.org Fri Mar 30 01:14:17 2007 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu, 29 Mar 2007 19:14:17 -0400 Subject: [Python-Dev] Italic text in the manual In-Reply-To: <43aa6ff70703291448v37c70244jee8d77bbc9654e1c@mail.gmail.com> References: <740c3aec0703080520r3237a403rba0046f575c0b5bd@mail.gmail.com> <200703291740.23431.fdrake@acm.org> <43aa6ff70703291448v37c70244jee8d77bbc9654e1c@mail.gmail.com> Message-ID: <200703291914.18023.fdrake@acm.org> On Thursday 29 March 2007 17:48, Collin Winter wrote: > The docs for atexit in py3k [1] are mostly (though not all) in > italics; I can't figure out why, and I'd appreciate if anyone with > stronger latex-foo could take a look. This is now fixed in Py3K, and there are no further occurrances of \em on the trunk of in Py3K. The online build will catch up when the automated build runs again. -Fred -- Fred L. Drake, Jr. From nnorwitz at gmail.com Fri Mar 30 05:18:42 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 29 Mar 2007 20:18:42 -0700 Subject: [Python-Dev] Get 2.5 changes in now, branch will be frozen soon Message-ID: This is a reminder that the 2.5 branch will be frozen early next week. If there are changes you want to get into 2.5.1, they should be checked in within a few days. Be conservative! There will be a 2.5.2, it's better to wait than to have to make a new release for one rushed feature. If you don't believe, just wait until Anthony shows up at your doorstep. :-) n From greg.ewing at canterbury.ac.nz Fri Mar 30 07:27:04 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 30 Mar 2007 17:27:04 +1200 Subject: [Python-Dev] Distutils and -framework on MacOSX In-Reply-To: <32AD8D7B-249A-4940-A316-8B4228FACAB0@mac.com> References: <460B88E4.4060204@canterbury.ac.nz> <32AD8D7B-249A-4940-A316-8B4228FACAB0@mac.com> Message-ID: <460C9FA8.7060307@canterbury.ac.nz> Ronald Oussoren wrote: > What's wrong with adding -framework flags to the end? I do this all the > time and have yet to run into problems. I don't know what's wrong. Sometimes it works for me too, but often it doesn't, and when it doesn't, putting them at the front seems to fix it. Apple's man page for ld says that placement of the -framework options is important... but doesn't say anything about where they're supposed to be. Not very helpful. I'd be very grateful if anyone could shed some light on this. It causes me no end of hassles whenever I try to compile an extension. -- Greg From sgeiger at ncee.net Fri Mar 30 08:28:01 2007 From: sgeiger at ncee.net (Shane Geiger) Date: Fri, 30 Mar 2007 01:28:01 -0500 Subject: [Python-Dev] p2p vpn in python? Message-ID: <460CADF1.5020704@ncee.net> Is anyone in the Python world considering writing a P2P VPN application in Python (a la Hamachi)? -- Shane Geiger IT Director National Council on Economic Education sgeiger at ncee.net | 402-438-8958 | http://www.ncee.net Leading the Campaign for Economic and Financial Literacy -------------- next part -------------- A non-text attachment was scrubbed... Name: sgeiger.vcf Type: text/x-vcard Size: 297 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20070330/5fe086a4/attachment.vcf From skip at pobox.com Fri Mar 30 16:41:07 2007 From: skip at pobox.com (skip at pobox.com) Date: Fri, 30 Mar 2007 09:41:07 -0500 Subject: [Python-Dev] Get 2.5 changes in now, branch will be frozen soon In-Reply-To: References: Message-ID: <17933.8579.460120.148673@montanaro.dyndns.org> Neal> If you don't believe, just wait until Anthony shows up at your Neal> doorstep. :-) And later sends you a bill for the airline tickets. ;-) Skip From nnorwitz at gmail.com Fri Mar 30 20:16:30 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Fri, 30 Mar 2007 11:16:30 -0700 Subject: [Python-Dev] Get 2.5 changes in now, branch will be frozen soon In-Reply-To: <17933.8579.460120.148673@montanaro.dyndns.org> References: <17933.8579.460120.148673@montanaro.dyndns.org> Message-ID: On 3/30/07, skip at pobox.com wrote: > Neal> If you don't believe, just wait until Anthony shows up at your > Neal> doorstep. :-) > > And later sends you a bill for the airline tickets. ;-) No, no. He just pays with the tickets from your bank account or credit card. Don't ask me how he does it. I really don't want to know. :-) n -- PS. And of course he flies first class all the way. From aahz at pythoncraft.com Fri Mar 30 22:03:09 2007 From: aahz at pythoncraft.com (Aahz) Date: Fri, 30 Mar 2007 13:03:09 -0700 Subject: [Python-Dev] p2p vpn in python? In-Reply-To: <460CADF1.5020704@ncee.net> References: <460CADF1.5020704@ncee.net> Message-ID: <20070330200309.GA11490@panix.com> On Fri, Mar 30, 2007, Shane Geiger wrote: > > Is anyone in the Python world considering writing a P2P VPN application > in Python (a la Hamachi)? You would probably do better to ask on comp.lang.python -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ Need a book? Use your library! From tjreedy at udel.edu Fri Mar 30 23:23:35 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 30 Mar 2007 17:23:35 -0400 Subject: [Python-Dev] p2p vpn in python? References: <460CADF1.5020704@ncee.net> Message-ID: "Shane Geiger" wrote in message news:460CADF1.5020704 at ncee.net... | Is anyone in the Python world considering writing a P2P VPN application | in Python (a la Hamachi)? Ask this on comp.lang.python ( == general Python mailing list == gmane.comp.python.general). This list is for discussion of the next Python release, not applications. From facundo at taniquetil.com.ar Sat Mar 31 02:38:37 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Sat, 31 Mar 2007 00:38:37 +0000 (UTC) Subject: [Python-Dev] Get 2.5 changes in now, branch will be frozen soon References: Message-ID: Neal Norwitz wrote: > This is a reminder that the 2.5 branch will be frozen early next week. > If there are changes you want to get into 2.5.1, they should be > checked in within a few days. Be conservative! There will be a There's this bug (which includes the patch): #1688393. It's ok, solves a bug. I even commited it to the trunk (with test cases and all). I'm not working in the 2.5 branch, so if anybody makes me the favor... ;) -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From facundo at taniquetil.com.ar Sat Mar 31 02:56:23 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Sat, 31 Mar 2007 00:56:23 +0000 (UTC) Subject: [Python-Dev] About SSL tests References: <20070329004057.18920.331373336.divmod.quotient.6356@ohm> Message-ID: Jean-Paul Calderone wrote: > If the openssl binary is available, when the test starts, launch it in > a child process, talk to it for the test, then kill it when the test is > done. Ok. I'll try to do something like this. I'm assigning the bug to myself. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From apt.shansen at gmail.com Sat Mar 31 11:04:28 2007 From: apt.shansen at gmail.com (Stephen Hansen) Date: Sat, 31 Mar 2007 02:04:28 -0700 Subject: [Python-Dev] Get 2.5 changes in now, branch will be frozen soon In-Reply-To: References: Message-ID: <7a9c25c20703310204gcd9e061m19971a743133fc38@mail.gmail.com> I'm sure everyone remembers the big ol' honking discussion on the change to os.splitext; it sorta fizzled after Guido asked if people would accept a pronouncement on the subject. I'm not anyone in the Python world, but felt strongly enough on the particular subject to submit a patch (and later revisions based upon the evolving conversation), but I think this should get resolved one way or the other. If the existing "fix" goes in for 2.5.1, then I'm going to withdraw said patch since I think it would end up doing more harm then good to silently change the semantics of how the function treats and defines extensions in one version only to do so again in another. The options, from how I see it, are: * The patch that Martin committed to fix the behavior such that splitext('.cshrc') returned ('.cshrc', '') remains. This would silently alter the behavior of a function whose tests have held the existing behavior correct; but it would make the behavior more logical in many situations. * The change gets simply reverted for now to return to the previous behavior, to perhaps be addressed later. This would revert to the status quo and its definition of 'extension' when it comes to files that have a leading dot, which may not be long-term desirable but which would at least give time for a plan and/or decision on the issue without a silent behavior adjustment. * The introduction of a keyword parameter to determine if the function should treat that initial dot as the start of an extension or a hidden filename. This would (IMHO) provide a migration path from the existing behavior which is a bit odd to the new behavior which makes more sense, while still allowing people to use whichever they choose to be appropriate for their domain. The question is also open on the issue of warnings; there was some sentiment that opposed a warning in this case (or in general) With the latter, another question is what the default is now; and will it change in the future. I just wanted to offer a gentle prod to see if a decision can be made; if any decision requires an adjustment to patches, tests and documentation, I'm willing to do them. Whatever the decision ends up being. On 3/29/07, Neal Norwitz wrote: > > This is a reminder that the 2.5 branch will be frozen early next week. > If there are changes you want to get into 2.5.1, they should be > checked in within a few days. Be conservative! There will be a > 2.5.2, it's better to wait than to have to make a new release for one > rushed feature. If you don't believe, just wait until Anthony shows > up at your doorstep. :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070331/9a5b6bda/attachment.htm From jcarlson at uci.edu Sat Mar 31 19:36:20 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 31 Mar 2007 10:36:20 -0700 Subject: [Python-Dev] Get 2.5 changes in now, branch will be frozen soon In-Reply-To: <7a9c25c20703310204gcd9e061m19971a743133fc38@mail.gmail.com> References: <7a9c25c20703310204gcd9e061m19971a743133fc38@mail.gmail.com> Message-ID: <20070331103448.0498.JCARLSON@uci.edu> "Stephen Hansen" wrote: > I'm sure everyone remembers the big ol' honking discussion on the change to > os.splitext; it sorta fizzled after Guido asked if people would accept a > pronouncement on the subject. I'm not anyone in the Python world, but felt > strongly enough on the particular subject to submit a patch (and later > revisions based upon the evolving conversation), but I think this should get > resolved one way or the other. If the existing "fix" goes in for 2.5.1, then > I'm going to withdraw said patch since I think it would end up doing more > harm then good to silently change the semantics of how the function treats > and defines extensions in one version only to do so again in another. Anthony Baxter said that the patch wasn't making it into 2.5.1, and since he is the release manager, his word is just about as final as Guido's (at least regarding the releases he does). - Josiah From apt.shansen at gmail.com Sat Mar 31 19:40:56 2007 From: apt.shansen at gmail.com (Stephen Hansen) Date: Sat, 31 Mar 2007 10:40:56 -0700 Subject: [Python-Dev] Get 2.5 changes in now, branch will be frozen soon In-Reply-To: <20070331103448.0498.JCARLSON@uci.edu> References: <7a9c25c20703310204gcd9e061m19971a743133fc38@mail.gmail.com> <20070331103448.0498.JCARLSON@uci.edu> Message-ID: <7a9c25c20703311040p39651445h41c03150f58bfd7e@mail.gmail.com> > Anthony Baxter said that the patch wasn't making it into 2.5.1, and > since he is the release manager, his word is just about as final as > Guido's (at least regarding the releases he does). Ah, oops! Work got busy, and I must have missed that in the Endless Threads. Nevermind then. :) --S -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070331/7f3e9259/attachment.html From sgeiger at ncee.net Sat Mar 31 22:24:07 2007 From: sgeiger at ncee.net (Shane Geiger) Date: Sat, 31 Mar 2007 15:24:07 -0500 Subject: [Python-Dev] proposed which.py replacement Message-ID: <460EC367.8090007@ncee.net> Trent Mick has a module called which.py that might make a nice platform-independent replacement for python2.5/Tools/scripts/which.py. http://www.trentm.com/projects/which/ Why which.py? |which.py| is a small GNU-which replacement. It has the following features: * it is portable (Windows, Linux, Mac OS X, Un*x); * it understands PATHEXT and "App Paths" registration on Windows (i.e. it will find everything that |start| does from the command shell); * it can print all matches on the PATH; * it can note "near misses" on the PATH (e.g. files that match but may not, say, have execute permissions); and * it can be used as a Python module. I also would be happy to have this be a replacement for the |which.py| in the Python CVS tree at |dist/src/Tools/scripts/which.py| which is Unix-specific and not usable as a module; and perhaps for inclusion in the stdlib. -- Shane Geiger IT Director National Council on Economic Education sgeiger at ncee.net | 402-438-8958 | http://www.ncee.net Leading the Campaign for Economic and Financial Literacy -------------- next part -------------- A non-text attachment was scrubbed... Name: sgeiger.vcf Type: text/x-vcard Size: 297 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20070331/934f73cd/attachment.vcf From guido at python.org Sat Mar 31 22:39:37 2007 From: guido at python.org (Guido van Rossum) Date: Sat, 31 Mar 2007 13:39:37 -0700 Subject: [Python-Dev] proposed which.py replacement In-Reply-To: <460EC367.8090007@ncee.net> References: <460EC367.8090007@ncee.net> Message-ID: If you ask me, having it hosted by Trent is probably more helpful for its popularity than putting it in the Python source distro; the Tools directory is mostly a poorly-maintained collection of trivia I wrote many years ago that is now quietly gathering dust. (Not all of it, of course; there's some useful stuff there that I *didn't* write, which ended up there because it is either *used* by the distro (e.g. the compiler package support) or because the author needed a channel that guaranteed open source status (e.g. world and pynche). But Trent's which.py doesn't seem to fall in either category.) --Guido On 3/31/07, Shane Geiger wrote: > > Trent Mick has a module called which.py that might make a nice > platform-independent replacement for python2.5/Tools/scripts/which.py. > > > > > http://www.trentm.com/projects/which/ > > > Why which.py? > > |which.py| is a small GNU-which replacement. It has the following features: > > * it is portable (Windows, Linux, Mac OS X, Un*x); > * it understands PATHEXT and "App Paths" registration on Windows > (i.e. it will find everything that |start| does from the command > shell); > * it can print all matches on the PATH; > * it can note "near misses" on the PATH (e.g. files that match but > may not, say, have execute permissions); and > * it can be used as a Python module. > > I also would be happy to have this be a replacement for the |which.py| > in the Python CVS tree at |dist/src/Tools/scripts/which.py| which is > Unix-specific and not usable as a module; and perhaps for inclusion in > the stdlib. > > > -- > Shane Geiger > IT Director > National Council on Economic Education > sgeiger at ncee.net | 402-438-8958 | http://www.ncee.net > > Leading the Campaign for Economic and Financial Literacy > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From python at rcn.com Sat Mar 31 21:35:14 2007 From: python at rcn.com (Raymond Hettinger) Date: Sat, 31 Mar 2007 12:35:14 -0700 Subject: [Python-Dev] Get 2.5 changes in now, branch will be frozen soon References: <7a9c25c20703310204gcd9e061m19971a743133fc38@mail.gmail.com> Message-ID: <007901c77408$19f697e0$6601a8c0@RaymondLaptop1> [Stephen Hansen= > I just wanted to offer a gentle prod to see if a decision can be made; > if any decision requires an adjustment to patches, tests and documentation, > I'm willing to do them. We should get a pronouncement on this or else whatever goes out in Py2.5.1 becomes the de-facto decision and becomes much harder to change. Raymond From oliphant at ee.byu.edu Tue Mar 27 01:48:53 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 26 Mar 2007 23:48:53 -0000 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: <4608594D.9050205@canterbury.ac.nz> References: <46027466.8010206@aerojockey.com> <4603051E.5080409@ee.byu.edu> <46036B73.7030006@aerojockey.com> <4607F99B.20800@ieee.org> <46084E0E.3030104@aerojockey.com> <4608594D.9050205@canterbury.ac.nz> Message-ID: <46085BE0.5080605@ee.byu.edu> Greg Ewing wrote: > But since the NumPy object has to know about the provider, > it can simply pass the release call on to it if appropriate. > I don't see how this case necessitates making the release call > on a different object. > > I'm -1 on involving any other objects or returning object > references from the buffer interface, unless someone can > come up with a use case which actually *requires* this > (as opposed to it just being something which might be > "nice to have"). The buffer interface should be Blazingly > Fast(tm), and messing with PyObject*s is not the way to > get that. The current proposal would be fast but would be more flexible for objects that don't have a memory representation that can be shared unless they create their own "sharing object" that perhaps copies the data into a contiguous chunk first. Objects which have memory which can be shared perfectly through the interface would simply pass themselves as the return value (after incrementing their "extant buffers" count by one). > > Seems to me the lock should apply to *everything* returned > by getbuffer. If the requestor is going to iterate over the > data, and there are multiple dimensions, surely it's going to > want to refer to the shape and stride info at regular intervals > while it's doing that. Requiring it to make its own copy > would be a burden. There are two use cases that seem to be under discussion. 1) When you want to apply an algorithm to an arbitrary object that exposes the buffer interface 2) When you want to create an object that shares memory with another object exposing the buffer interface. These two use cases have slightly different needs. What I want to avoid is forcing the exporting object to be unable to change its shape and strides just because an object is using the memory for use case #2. I think the solution that states the shape and strides information are only guaranteed valid until the GIL is released is sufficent. Alternatively, one could release the shape and strides and format separately from the memory with a flag as a second argument to releasebuffer. -Travis > > -- > Greg > > From oliphant at ee.byu.edu Tue Mar 27 01:52:03 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 26 Mar 2007 23:52:03 -0000 Subject: [Python-Dev] Extended Buffer Interface/Protocol In-Reply-To: <46084E0E.3030104@aerojockey.com> References: <46027466.8010206@aerojockey.com> <4603051E.5080409@ee.byu.edu> <46036B73.7030006@aerojockey.com> <4607F99B.20800@ieee.org> <46084E0E.3030104@aerojockey.com> Message-ID: <46085952.2010508@ee.byu.edu> Carl Banks wrote: > Tr > ITSM that we are using the word "view" very differently. Consider > this example: > > A = zeros((100,100)) > B = A.transpose() You are thinking of NumPy's particular use case. I'm thinking of a generic use case. So, yes I'm using the word view in two different contexts. In this scenario, NumPy does not even use the buffer interface. It knows how to transpose it's own objects and does so by creating a new NumPy object (with it's own shape and strides space) with a data buffer pointed to by "A". Yes, I use the word "view" for this NumPy usage, but only in the context of NumPy. In the PEP, I've been using the word "view" quite a bit more generically. So, I don't think this is a good example because A.transpose() will never call getbuffer of the A object (it will instead use the known structure of NumPy directly). So, let's talk about the generic situation instead of the NumPy specific one. > > I'd suggest the object returned by A.getbuffer should be called the > "buffer provider" or something like that. I don't care what we call it. I've been using the word "view" because of the obvious analogy to my use of view in NumPy. When I had envisioned returning an actual object very similar to a NumPy array from the buffer interface it made a lot of sense to call it a view. Now, I'm fine to call it "buffer provider" > > For the sake of discussion, I'm going to avoid the word "view" > altogether. I'll call A the exporter, as before. B I'll refer to as > the requestor. The object returned by A.getbuffer is the provider. Fine. Let's use that terminology since it is new and not cluttered by other uses in other contexts. > Having thought quite a bit about it, and having written several > abortive replies, I now understand it and see the importance of it. > getbuffer returns the object that you are to call releasebuffer on. > It may or may not be the same object as exporter. Makes sense, is > easy to explain. Yes, that's exactly all I had considered it to be. Only now, I'm wondering if we need to explicitly release a lock on the shape, strides, and format information as well as the buffer location information. > > It's easy to see possible use cases for returning a different object. > A hypothetical future incarnation of NumPy might shift the > responsibility of managing buffers from NumPy array object to a hidden > raw buffer object. In this scenario, the NumPy object is the > exporter, but the raw buffer object the provider. > > Considering this use case, it's clear that getbuffer should return the > shape and stride data independently of the provider. The raw buffer > object wouldn't have that information; all it does is store a pointer > and keep a reference count. Shape and stride is defined by the exporter. So, who manages the memory to the shape and strides and isptr arrays? When a provider is created do these need to be created so that the shape and strides arrays are never deallocated when in use. The situation I'm considering is if you have a NumPy array of shape (2,3,3) which you then obtain a provider of (presumably from another package) and it retains a lock on the memory for a while. Should it also retain a lock on the shape and strides array? Can the NumPy array re-assign the shape and strides while the provider has still not been released? I would like to say yes, which means that the provider must supply it's own copy of shape and strides arrays. This could be the policy. Namely, that the provider must supply the memory for the shape, strides, and format arrays which is guaranteed for as long as a lock is held. In the case of NumPy, that provider could create it's own copy of the shape and strides arrays (or do it when the shape and strides arrays are re-assigned). > >>> Second question: what happens if a view wants to re-export the >>> buffer? Do the views of the buffer ever change? Example, say you >>> create a transposed view of a Numpy array. Now you want a slice of >>> the transposed array. What does the transposed view's getbuffer >>> export? >> >> >> Basically, you could not alter the internal representation of the >> object while views which depended on those values were around. >> >> In NumPy, a transposed array actually creates a new NumPy object that >> refers to the same data but has its own shape and strides arrays. >> >> With the new buffer protocol, the NumPy array would not be able to >> alter it's shape/strides/or reallocate its data areas while views >> were being held by other objects. > > > But requestors could alter their own copies of the data, no? Back to > the transpose example: B itself obviously can't use the same "strides" > array as A uses. It would have to create its own strides, right? I don't like this example because B does have it's own strides because it is a complete NumPy array. I think we are talking about the same thing and that is "who manages the memory" for the shape and strides (and format). I think the easiest solution is to say that the "provider" does. It will manage that memory until the lock is released. How the exporter object handles that is up to the exporter. > > So, what if someone takes a slice out of B? When calling B.getbuffer, > does it get B's strides, or A's? > > I think it should get B's. After all, if you're taking a slice of B, > don't you care about the slicing relative to B's axes? I can't really > think of a use case for exporting A's stride data when you take a > slice of B, and it doesn't seem to simplify memory management, because > B has to make it's own copies anyways. We already have an implementation in NumPy for memory sharing and in that implementation each NumPy array controls it's own shape and strides and only the data-location is shared. I don't like the terminology of "re-exporting" because it is unnecessarily confusing to me. Naturally an object can both consume and export the buffer interface. If it does this, then it will want to make it's own copies of the shape and strides arrays so as not to rely on the "provider" for these. > > Here's what I think: the lock should only apply to the buffer itself, > and not to shape and stride data at all. If the requestor wants to > keep its own copies of the data, it would have to malloc its own > storage for it. I expect that this would be very rare. What I'm worried about is un-necessarily preventing an exporter from changing its shape and strides because a consumer is holding a lock to it's memory (but wouldn't need a lock to it's shape and strides because it made it's own copy). How does the consumer signal that situation to the provider? So, we can't avoid thinking about a lock on the shape and stride data. We can make a policy that the provider always owns the memory for the shape and stride and will not alter it until the lock is released. Or we can say that the shape and stride information could change before the lock on the memory is released if the GIL is released (so make your own copies if you need to keep track of shape and stride). Or we could have a separate lock on shape/stride. But, we have to make some policy. > > As for the provider; I think that's between it the exporter. If the > exporter and provider know about each other, they shouldn't have any > problems managing memory together. Sure, what the provider and exporter do is up to the exporting object. > > >> Having such a thing as a view object would actually be nice because >> it could hold on to a particular view of data with a given set of >> shape and strides (whose memory it owns itself) and then the >> exporting object would be free to change it's shape/strides >> information as long as the data did not change. > > > What I don't undestand is why it's important for the provider to > retain this data. The requestor only needs the information when it's > created; it will calculate its own versions of the data, and will not > need the originals again, so no need to the provider to keep them around. That is certainly a policy we could enforce (and pretty much what I've been thinking). We just need to make it explicit that the shape and strides provided is only guaranteed up until a GIL release (i.e. arbitrary Python code could change these memory areas both their content and location) and so if you need them later, make your own copies. If this were the policy, then NumPy could simply pass pointers to its stored shape and strides arrays when the buffer interface is called but then not worry about re-allocating these arrays before the "buffer" lock is released. Another object could hold on to the memory area of the NumPy array but would have to store shape and strides information if it wanted to keep it. NumPy could also just pass a pointer to the char * representation of the format (which in NumPy would be stored in the dtype object) and would not have to worry about the dtype being re-assigned later. > >>> The reason I ask is: if things like "buf" and "strides" and "shape" >>> could change when a buffer is re-exported, then it can complicate >>> things for PIL-like buffers. (How would you account for offsets in >>> a dimension that's in a subarray?) >> >> >> I'm not sure what you mean, offsets are handled by changing the >> starting location of the pointer to the buffer. > > > > But to anwser your question: you can't just change the starting > location because there's hundreds of buffers. You'd either have to > change the starting location of each one of them, which is highly > undesirable, or to factor in an offset somehow. (This was, in fact, > the point of the "derefoff" term in my original suggestion.) I get better what your derefoff was doing now. I was missing the de-referencing that was going on. Couldn't you still just store a pointer to the start of the array. In other words, isn't your **isptr suggestion sufficient? It seems to be. > > > Anyways, despite the miscommunications so far, I now have a very good > idea of what's going on. We definitely need to get terms straight. I > agree that getbuffer should return an object. I think we need to > think harder about the case when requestors re-export the buffer. > (Maybe it's time to whip up some experimental objects?) I'm still not clear what you are concerned about. If an object consumes the buffer interface and then wants to be able to later export it to another, then from our discussion about the shape/strides and format information, it would have to maintain it's own copies of these things, because it could not rely on the original provider (or exporter) to keep them around once the GIL is released. This is the reason we would have to be very clear about the guaranteed persistance of the shape/strides and format memory whose pointers are returned through the proposed buffer interface. Thanks for the discussion. It is nice to have someone to talk with about these things. A conversation always results in a better implementation. -Travis From status at bugs.python.org Tue Mar 27 07:24:11 2007 From: status at bugs.python.org (Tracker) Date: Tue, 27 Mar 2007 05:24:11 -0000 Subject: [Python-Dev] Summary of Tracker Issues Message-ID: <20070327052409.20A6A782AD@psf.localdomain> ACTIVITY SUMMARY (03/20/07 - 03/27/07) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1646 open ( +2) / 8583 closed ( +1) / 10229 total ( +3) Average duration of open issues: 747 days. Median duration of open issues: 695 days. Open Issues Breakdown open 1646 ( +2) pending 0 ( +0) Issues Created Or Reopened (3) ______________________________ Buy Tramadol online 03/25/07 http://bugs.python.org/issue1023 created conor2k Testing bugs.python.org domain 03/26/07 CLOSED http://bugs.python.org/issue1024 created izak Test issue 03/26/07 http://bugs.python.org/issue1025 created loewis Issues Now Closed (2) _____________________ Personal loans. Bad credit personal loans. Unsecured personal l 1 days http://bugs.python.org/issue1002 dubois Testing bugs.python.org domain 0 days http://bugs.python.org/issue1024 izak -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070327/98aa1f2f/attachment-0001.htm