From martin at v.loewis.de Mon Sep 1 07:04:00 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Mon Sep 1 02:04:01 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: References: Message-ID: "Tim Peters" writes: > > This is much worse, then: How can it possibly know what formats the C > > library expects in the current locale? What if the C library insists that > > a thousands-separator is used when the locale has one? etc. > > I'm not sure that's a realistic objection. Ok. Are you then, overall, in favour of taking the proposed approach? It is not thread-safe, but only so if somebody calls setlocale in a different thread, and that is known not to be thread-safe - so I could live with that limitation. It is just that the patch does not "feel" right, given that there must be "native" locale-inaware parsing of floating point constants somewhere on each platform (atleast on those that support C++98). Regards, Martin From martin at v.loewis.de Mon Sep 1 07:36:16 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Mon Sep 1 02:36:17 2003 Subject: [Python-Dev] Be Honest about LC_NUMERIC [REPOST] In-Reply-To: <20030814022128.GN3095@async.com.br> References: <20030814022128.GN3095@async.com.br> Message-ID: Christian Reis writes: > So, in an attempt to garner comments (now that we have 2.3 off the > chopping block) I'm reposting my PEP proposal (with minor updates). I can agree with the declared problem of the PEP, and the rationale for fixing it. Tim also convinced me that the approach taken to solve it is, technically, acceptable. So I only list issues where I disagree. > This change should also solve the aforementioned thread-safety > problems. It does not, and I think the PEP should point out that it doesn't. > This problem was initially reported as a problem in the GTK+ > libraries [5]; since then it has been correctly diagnosed as an > inconsistency in Python's implementation. However, in a fortunate > coincidence, the glib library implements a number of > LC_NUMERIC-agnostic functions (for an example, see [6]) for reasons > similar to those presented in this paper. One of my early concerns (and I still have this concern) is that the contributors here appear to take the position "We have this fine code developed elsewhere, it seems to work, so we copy it. We don't actually have to understand this code". I would feel more comfortable if the code was written from scratch for usage in Python, with just the ideas borrowed from glib. Proper attribution of contributors and licensing are just one aspect, we really need the submitter of the code fully understand it, and be capable of reacting to problems quickly. That said, I don't actually require that the code is written from scratch. Instead, a detailed elaboration of how precisely the implementation is approached, in the PEP, would be good. The PEP should also point out deficiencies of the approach taken, e.g. the issue of spelling NaN, inf, etc. If it can be determined not to be an issue in real life (i.e. for all interesting platforms), this should be documented as well. Regards, Martin From james at daa.com.au Mon Sep 1 15:59:32 2003 From: james at daa.com.au (James Henstridge) Date: Mon Sep 1 02:59:40 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: References: Message-ID: <3F52EE54.6090800@daa.com.au> On 31/08/2003 9:25 AM, Tim Peters wrote: >[James Henstridge] > > >>>As Christian said, there is code in glib (not to be confused with >>>glibc: the GNU C library) that could act as a basis for locale >>>independent float conversion functions in Python. >>> >>> > >[Martin v. L?wis] > > >>I very much doubt that this statement is true. Are you sure this code >>supports all the platforms where Python runs? E.g. what about the >>three (!) different floating point formats on a VAX? >> >> > >Well, you should look at the patch: it doesn't know anything about internal >fp formats -- all conversions are performed in the end by calling the >platform C's strtod() or snprintf(). What it does do is: > >1. For string to double, preprocess the input string to change it to > use current-locale spelling before calling the platform C strtod(). > >2. For double to string, postprocess the result of the platform C > snprintf() to replace current-locale spelling with a "standard" > spelling. > >So this is much more string-munging code than it is floating-point code. >Indeed, there doesn't appear to be a single floating-point operation in the >entire patch (apart from calls to platform string<->float functions). > >OTOH, despite the claims, it doesn't look threadsafe to me: there's no >guarantee, e.g., that the idea of current locale g_ascii_strtod() obtains >from localeconv() at its start is still in effect by the time >g_ascii_strtod() gets around to calling strtod(). > This is true. However, in practice we found it fixed a number of thread safety issues in programs. Your average localised package usually switches to the user's preferred locale on startup, so that it can display strings and messages, and occasionally wants to read/write numbers in a locale independent format (usually when saving/loading files). The most common way of doing this is the setlocale/strtod/setlocale combo, which has thread safety problems and possible reentrancy problems if done wrong. The method used by g_ascii_strotod() removes the need to switch locale when parsing the float, which means that an application using it may only need to call setlocale() once on startup and never again. This seems to be the best way to use setlocale w.r.t. thread safety. The existing locale handling in Python shares this property, but makes it difficult for external libraries to format and parse floats in the locale's representation. From what I can see, leaving LC_NUMERIC set to the locale value rather than "C" leads to better interoperability. > So at best it solves part >of one relevant problem here (other relevant problems include that platform >C libraries disagree about how to spell infinities, NaNs and signed zeroes; >about how many digits to use for an exponent; and about how to round results >(for example, > > > >>>>"%.1f" % 2.25 >>>> >>>> >'2.3' > > > >on Windows, but most (not all!) flavors of Unix produce the IEEE-754 >to-nearest/even rounded '2.2' instead)). > >It's easy to write portable, perfectly-rounding string<->double conversion >routines without calling any platform functions. The rub is that "fast" >goes out the window then, unless you give up at least one of {portable, >accurate}. > > It would be great for Python to have consistent float parsing/formatting on every platform in the future. Making sure that every place where Python wants to parse or format a float in a locale independent fashion go through a single set of functions should make it easier to drop in a new set of routines in the future. However, getting rid of the LC_NUMERIC=C requirement would have real benefits today. James. -- Email: james@daa.com.au WWW: http://www.daa.com.au/~james/ From martin at v.loewis.de Mon Sep 1 08:15:08 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Mon Sep 1 03:15:10 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: <3F52EE54.6090800@daa.com.au> References: <3F52EE54.6090800@daa.com.au> Message-ID: James Henstridge writes: > The existing locale handling in Python shares this property, but makes > it difficult for external libraries to format and parse floats in the > locale's representation. From what I can see, leaving LC_NUMERIC set > to the locale value rather than "C" leads to better interoperability. I think everybody agrees that allowing non-C LC_NUMERIC settings in the C library is very desirable. My concerns are about the specific approach taken to implement that change. Or, actually, with an entire class of approaches: namely those that involve complex algorithms (i.e. which include a for-statement :) to implement that feature. Regards, Martin From guido at python.org Mon Sep 1 02:34:36 2003 From: guido at python.org (Guido van Rossum) Date: Mon Sep 1 04:35:34 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: Your message of "01 Sep 2003 08:03:21 +0200." References: Message-ID: <200309010834.h818Yac22250@12-236-84-31.client.attbi.com> > It is just that the patch does not "feel" right, given that there must > be "native" locale-inaware parsing of floating point constants > somewhere on each platform (atleast on those that support C++98). FWIW, I have the same feeling, but the idea of having to support our own version of such code is even more uncomfortable. Maybe at least we can detect platforms for which we know there is a native conversion in the library, and not use the hack on those? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Sep 1 02:36:32 2003 From: guido at python.org (Guido van Rossum) Date: Mon Sep 1 04:37:16 2003 Subject: [Python-Dev] Be Honest about LC_NUMERIC [REPOST] In-Reply-To: Your message of "01 Sep 2003 08:35:43 +0200." References: <20030814022128.GN3095@async.com.br> Message-ID: <200309010836.h818aWF22262@12-236-84-31.client.attbi.com> > That said, I don't actually require that the code is written from > scratch. Instead, a detailed elaboration of how precisely the > implementation is approached, in the PEP, would be good. > > The PEP should also point out deficiencies of the approach taken, > e.g. the issue of spelling NaN, inf, etc. If it can be determined not > to be an issue in real life (i.e. for all interesting platforms), this > should be documented as well. I think that anything that might still be needed years after the code is adopted should be in comments in the code, not in a PEP. --Guido van Rossum (home page: http://www.python.org/~guido/) From gjc at inescporto.pt Mon Sep 1 12:07:43 2003 From: gjc at inescporto.pt (Gustavo J. A. M. Carneiro) Date: Mon Sep 1 07:07:44 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: <200309010834.h818Yac22250@12-236-84-31.client.attbi.com> References: <200309010834.h818Yac22250@12-236-84-31.client.attbi.com> Message-ID: <1062414058.21848.4.camel@spectrum.inescn.pt> A Seg, 2003-09-01 ?s 09:34, Guido van Rossum escreveu: > > It is just that the patch does not "feel" right, given that there must > > be "native" locale-inaware parsing of floating point constants > > somewhere on each platform (atleast on those that support C++98). > > FWIW, I have the same feeling, but the idea of having to support > our own version of such code is even more uncomfortable. > > Maybe at least we can detect platforms for which we know there is a > native conversion in the library, and not use the hack on those? In case people haven't noticed, the second version of the patch, submitted following the first patch, already does that. It adds a configure.in check for glibc's strtod_l() and uses that instead of glib code whenever it's available. Only on non-glibc systems is the glib code compiled in. Regards. -- Gustavo Jo?o Alves Marques Carneiro From guido at python.org Mon Sep 1 11:05:35 2003 From: guido at python.org (Guido van Rossum) Date: Mon Sep 1 13:07:09 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: Your message of "01 Sep 2003 12:00:58 BST." <1062414058.21848.4.camel@spectrum.inescn.pt> References: <200309010834.h818Yac22250@12-236-84-31.client.attbi.com> <1062414058.21848.4.camel@spectrum.inescn.pt> Message-ID: <200309011705.h81H5Zf22978@12-236-84-31.client.attbi.com> > A Seg, 2003-09-01 ās 09:34, Guido van Rossum escreveu: > > FWIW, I have the same feeling, but the idea of having to support > > our own version of such code is even more uncomfortable. > > > > Maybe at least we can detect platforms for which we know there is a > > native conversion in the library, and not use the hack on those? > > In case people haven't noticed, the second version of the patch, > submitted following the first patch, already does that. It adds a > configure.in check for glibc's strtod_l() and uses that instead of glib > code whenever it's available. Only on non-glibc systems is the glib > code compiled in. This doesn't address the first sentence of mine quoted above: I'm uncomfortable with having this code being part of Python, given that we have no expertise to maintain it long-term (nor should we need such expertise, IMO). Here's yet another idea (which probably has flaws as well): instead of substituting the locale's decimal separator, rewrite strings like "3.14" as "314e-2" and rewrite strings like "3.14e5" as "314e3", then pass to strtod(), which assigns the same meaning to such strings in all locales. This removes the question of what decimal separator is used by the locale completely, and thus removes the last bit of thread-unsafety from the code. However, I don't know if underflow can cause the result to be different, e.g. perhaps 1.23eX could be computed but 123e(X-2) could not??? (Sounds pretty unlikely on the face of it since I'd expect any decent conversion algorithm to pretty much break its input down into a string of digits and an exponent, but I've never actually studied such algorithms in detail.) --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Mon Sep 1 15:30:23 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Sep 1 14:30:58 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: Message-ID: [Martin] > Ok. Are you then, overall, in favour of taking the proposed approach? It solves part of one problem; I'd rather solve all of it, but can't volunteer time to do that. > It is not thread-safe, but only so if somebody calls setlocale in a > different thread, and that is known not to be thread-safe - so I could > live with that limitation. There's no way of using C's locale gimmicks that's threadsafe, short of all callers agreeing to follow a beyond-standard-C exclusion protocol -- which is the same as saying "no way" in reality. So that's part of one problem no patch of this ilk *can* solve. It's not that the patch doesn't try hard enough, it's that this approach is inherently inadequate to solve all of this particular problem. > It is just that the patch does not "feel" right, given that there must > be "native" locale-inaware parsing of floating point constants > somewhere on each platform (atleast on those that support C++98). I haven't found one on Windows (doesn't mean it doesn't exist, does mean it's apparently well hidden if it does exist). > ... > One of my early concerns (and I still have this concern) is that the > contributors here appear to take the position "We have this fine code > developed elsewhere, it seems to work, so we copy it. We don't > actually have to understand this code". I would feel more comfortable > if the code was written from scratch for usage in Python, with just > the ideas borrowed from glib. Proper attribution of contributors and > licensing are just one aspect, we really need the submitter of the > code fully understand it, and be capable of reacting to problems > quickly. The patch is certainly more code than is needed to solve the part of the problem it does solve. For example, things like typedef char gchar; typedef short gshort; typedef long glong; typedef int gint; introduce silly synonyms ("silly" == typing gshort instead of short does nothing except introduce possibilities for confusion); there are many definitions like #define g_ascii_isupper(c) \ ((g_ascii_table[(guchar) (c)] & G_ASCII_UPPER) != 0) that are never referenced; the code caters to C99's hexadecimal float literals but Python doesn't; and so on. If someone who understood Python internals read my earlier two-sentence description of how the patch works, they could write something that works equally well for Python's purposes with a fraction of the code introduced by the patch. > ... > The PEP should also point out deficiencies of the approach taken, > e.g. the issue of spelling NaN, inf, etc. If it can be determined not > to be an issue in real life (i.e. for all interesting platforms), this > should be documented as well. Well, the patch doesn't even pretend to address other issues with portability of float literals. They routinely come up on c.l.py, so of course users bump into them; when someone is motivated enough to file a bug report, I shuffle it off to PEP 42, under the "non-accidental 754 support" heading (which covers many fp issues beyond just literals, of course). [James Henstridge] > ... > Your average localised package usually switches to the user's > preferred locale on startup, so that it can display strings and > messages, and occasionally wants to read/write numbers in a locale > independent format (usually when saving/loading files). The most > common way of doing this is the setlocale/strtod/setlocale combo, > which has thread safety problems and possible reentrancy problems if > done wrong. I became acutely aware of the problems here due to the spambayes project, part of which embeds Python in Outlook 2000/2002. Outlook routinely runs more than a dozen threads, and by observation changes locale "frequently". None of that is documented, Python has no influence over when or why Outlook decides to switch locale, and neither can Python exclude Outlook's other threads when the Outlook thread Python is running in becomes active. Mark Hammond solved our problems there by forcing locale back to "C" every chance he gets; that's an anti-social and probabilistic approach, but appears to be the best spambayes can do today. Having spambayes grow its own float<->string code doesn't help, because the worst problem spambayes had is that Python's marshal format uses ASCII strings to store float literals in .pyc files, so that Python itself can (and does) load insane float values out of .pyc files if LC_NUMERIC isn't "C" at the time a .pyc file gets imported. The only thing that could truly solve spambayes's problems here is for Python to use a thoroughly thread-safe string->float routine, where "thoroughly" includes not caring whether other threads switch locale in mid-stream. An irony is that Microsoft's *native* locale gimmicks are thread-safe (each Win32 thread has its own idea of Win32 locale); why Outlook is even mucking with C's thread-braindead notion of locale is a mystery. In short, I can't be enthusiastic about the patch because it doesn't solve the only relevant locale problem I've actually run into. I understand that it may well solve many I haven't run into. OTOH, the specific problem I'm acutely worried about would be better addressed by changing the way Python marhals float values. [Guido] > Maybe at least we can detect platforms for which we know there is a > native conversion in the library, and not use the hack on those? I rarely find that piles of conditionalized code are more comprehensible or reliable; they usually result in mysterious x-platform differences, and become messier over time as we stumble into more platform library bugs, quirks, and limitations. > ... > Here's yet another idea (which probably has flaws as well): instead of > substituting the locale's decimal separator, rewrite strings like > "3.14" as "314e-2" and rewrite strings like "3.14e5" as "314e3", then > pass to strtod(), which assigns the same meaning to such strings in > all locales. This is a harder transformation than s/./locale_decimal_point/. It does address the thread-safety issue. Numerically it's flaky, as only a perfectly-rounding string->float routine can guarantee to return bit-for-bit identical results given equivalent (viewed as infinite precision) decimal representations as inputs, and few platform string->float routines do perfect rounding. > This removes the question of what decimal separator is used by the > locale completely, and thus removes the last bit of thread-unsafety > from the code. However, I don't know if underflow can cause the result > to be different, e.g. perhaps 1.23eX could be computed but 123e(X-2) > could not??? (Sounds pretty unlikely on the face of it since I'd expect > any decent conversion algorithm to pretty much break its input down into > a string of digits and an exponent, but I've never actually studied > such algorithms in detail.) Each library is likely fail in its own unique ways. Here's a cute one: """ base = 1.2345678901234567 digits = "12345678901234567" for exponent in range(-16, -15000, -1): string = digits + "0" * (-16 - exponent) string += "e%d" % exponent derived = float(string) assert base == derived, (string, derived) """ On Windows, this first fails at exponent -5202, where float(string) delivers a result a factor of 10 too large. I was surprised it did that well! Under Cygwin Python 2.2.3, it consumed > 14 minutes of CPU time, but never failed. I believe they're using a derivative of David Gay's excruciatingly complex IEEE-754 perfect-rounding string<->float routines (which would explain both why it didn't fail and why it consumed enormous CPU time; the code is excruciatingly complex because it does perfect rounding quickly for "normal" inputs, via a large variety of delicate speed tricks; when those tricks don't apply, it has to simulate unbounded-precision arithmetic to guarantee perfect rounding). From kiko at async.com.br Mon Sep 1 17:44:33 2003 From: kiko at async.com.br (Christian Reis) Date: Mon Sep 1 15:45:13 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: References: Message-ID: <20030901194433.GK1917@async.com.br> On Mon, Sep 01, 2003 at 02:30:23PM -0400, Tim Peters wrote: > > One of my early concerns (and I still have this concern) is that the > > contributors here appear to take the position "We have this fine code > > developed elsewhere, it seems to work, so we copy it. We don't > > actually have to understand this code". I would feel more comfortable > > if the code was written from scratch for usage in Python, with just > > the ideas borrowed from glib. Proper attribution of contributors and > > licensing are just one aspect, we really need the submitter of the > > code fully understand it, and be capable of reacting to problems > > quickly. > > The patch is certainly more code than is needed to solve the part of the > problem it does solve. For example, things like > > typedef char gchar; > typedef short gshort; > typedef long glong; > typedef int gint; > > introduce silly synonyms ("silly" == typing gshort instead of short does > nothing except introduce possibilities for confusion); there are many > definitions like > > #define g_ascii_isupper(c) \ > ((g_ascii_table[(guchar) (c)] & G_ASCII_UPPER) != 0) > > that are never referenced; the code caters to C99's hexadecimal float > literals but Python doesn't; and so on. If someone who understood Python > internals read my earlier two-sentence description of how the patch works, > they could write something that works equally well for Python's purposes > with a fraction of the code introduced by the patch. I would certainly concede this point -- Gustavo's patch is a proof-of-concept implementation. I do believe that the glib code is a good starting point for an implementation, and the author has submitted a written agreement, so the next step would be obtaining approval of the general approach and then diving in to clean up and minimize the code as much as possible. The issue is whether a cleaned up patch is the way python-dev would like this to go, or if another, perhaps orthogonal, approach -- such as the conversion Guido has proposed -- would be preferrable. > > The PEP should also point out deficiencies of the approach taken, > > e.g. the issue of spelling NaN, inf, etc. If it can be determined not > > to be an issue in real life (i.e. for all interesting platforms), this > > should be documented as well. > > Well, the patch doesn't even pretend to address other issues with > portability of float literals. They routinely come up on c.l.py, so of Are these representations (NaN, infinity, etc) LC_NUMERIC-dependent? Or, more generally, locale-dependent? As for the thread-safety issues, it's true, changing locale multiple times in runtime is bound to confuse the conversion code (and consequently the interpreter). One way around this would be a complete reimplementation of the relevant functions, but here's betting that that alternative would be shunned even more intensely. Take care, -- Christian Reis, Senior Engineer, Async Open Source, Brazil. http://async.com.br/~kiko/ | [+55 16] 261 2331 | NMFL From martin at v.loewis.de Mon Sep 1 22:36:59 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Mon Sep 1 17:37:01 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: <200309011705.h81H5Zf22978@12-236-84-31.client.attbi.com> References: <200309010834.h818Yac22250@12-236-84-31.client.attbi.com> <1062414058.21848.4.camel@spectrum.inescn.pt> <200309011705.h81H5Zf22978@12-236-84-31.client.attbi.com> Message-ID: Guido van Rossum writes: > Here's yet another idea (which probably has flaws as well): instead of > substituting the locale's decimal separator, rewrite strings like > "3.14" as "314e-2" and rewrite strings like "3.14e5" as "314e3", then > pass to strtod(), which assigns the same meaning to such strings in > all locales. Unfortunately, that doesn't work for sprintf, for which you still need to replace the radixchar. Regards, Martin From gjc at inescporto.pt Tue Sep 2 00:08:11 2003 From: gjc at inescporto.pt (Gustavo J A M Carneiro) Date: Mon Sep 1 18:08:11 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: <20030901194433.GK1917@async.com.br> References: <20030901194433.GK1917@async.com.br> Message-ID: <1062454091.3272.11.camel@emperor.homelinux.net> A Seg, 2003-09-01 ?s 20:44, Christian Reis escreveu: > On Mon, Sep 01, 2003 at 02:30:23PM -0400, Tim Peters wrote: > > > One of my early concerns (and I still have this concern) is that the > > > contributors here appear to take the position "We have this fine code > > > developed elsewhere, it seems to work, so we copy it. We don't > > > actually have to understand this code". I would feel more comfortable > > > if the code was written from scratch for usage in Python, with just > > > the ideas borrowed from glib. Proper attribution of contributors and > > > licensing are just one aspect, we really need the submitter of the > > > code fully understand it, and be capable of reacting to problems > > > quickly. > > > > The patch is certainly more code than is needed to solve the part of the > > problem it does solve. For example, things like > > > > typedef char gchar; > > typedef short gshort; > > typedef long glong; > > typedef int gint; > > > > introduce silly synonyms ("silly" == typing gshort instead of short does > > nothing except introduce possibilities for confusion); there are many > > definitions like > > > > #define g_ascii_isupper(c) \ > > ((g_ascii_table[(guchar) (c)] & G_ASCII_UPPER) != 0) > > > > that are never referenced; the code caters to C99's hexadecimal float > > literals but Python doesn't; and so on. If someone who understood Python > > internals read my earlier two-sentence description of how the patch works, > > they could write something that works equally well for Python's purposes > > with a fraction of the code introduced by the patch. > > I would certainly concede this point -- Gustavo's patch is a > proof-of-concept implementation. I do believe that the glib code is a > good starting point for an implementation, and the author has submitted > a written agreement, so the next step would be obtaining approval of the > general approach and then diving in to clean up and minimize the code as > much as possible. Actually, this code presentation is intentional, for the following two reasons: 1- I didn't want to accidentally introduce any bug, so I tried to copy-paste the code with as little changes as possible; 2- In the current form, if glib developers find any bug in this code, we can easily merge the changes back into python. Perhaps I was wrong to have done it this way... Anyway, replacing the g* types is trivial with any decent text editor. Regards. -- Gustavo J. A. M. Carneiro From martin at v.loewis.de Mon Sep 1 23:19:09 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Mon Sep 1 18:19:10 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: <1062454091.3272.11.camel@emperor.homelinux.net> References: <20030901194433.GK1917@async.com.br> <1062454091.3272.11.camel@emperor.homelinux.net> Message-ID: Gustavo J A M Carneiro writes: > Perhaps I was wrong to have done it this way... Anyway, replacing the > g* types is trivial with any decent text editor. It's more than that. We want you to fully understand the patch, and to spot errors in it even before the glib developers find them. We want you to provide a minimalistic patch, that just implements the required functionality, and nothing else. We want the patch to be maintainable due to it being easy to read and follow, instead of being maintainable due to the fact that it is identical with some code elsewhere in the world. Regards, Martin From sholden at holdenweb.com Mon Sep 1 20:14:54 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Sep 1 19:17:20 2003 Subject: [Python-Dev] readline.clear_history() In-Reply-To: <2mn0duxc2b.fsf@starship.python.net> Message-ID: [Michael Hudson] > Guido van Rossum writes: > > > Probably. I'm personally not so stringent for minor stuff like this > > (hey, I added bool() to 2.2.1 :-), but it seems others are > > increasingly resisting even minor feature additions in > micro releases > > (except in the email package :-). > > The problem with adding things like this is that it makes it harder to > port code from *2.3.1 to 2.3* and at least in principle it would be > nice if code that works on 2.3.1 works on 2.3 unless it tickles a > specific bug that has been fixed. > > And, come Panther, there are going to be rather a lot of Python 2.3 > installs out there. > Just as a matter of interest, do you have any idea how many Mac OSX licensees there are? regards -- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ From guido at python.org Mon Sep 1 18:03:29 2003 From: guido at python.org (Guido van Rossum) Date: Mon Sep 1 20:04:54 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: Your message of "Mon, 01 Sep 2003 14:30:23 EDT." References: Message-ID: <200309020003.h8203TD23456@12-236-84-31.client.attbi.com> [Tim] > In short, I can't be enthusiastic about the patch because it doesn't > solve the only relevant locale problem I've actually run into. I > understand that it may well solve many I haven't run into. At this point in your life, Tim, is there any patch you could be truly enthusiastic about? :-) I'm asking because I'd like to see the specific problem that started this thread solved, if necessary using a compromise that means the solution isn't perfect. I'm even willing to take a step back in the status quo, given that the status quo isn't perfect anyway, and that compromises mean something has to give. *Maybe* the right solution is that we have to accept a hard-to-understand overcomplicated piece of code that we don't know how to maintain (but for which the author asserts that we won't have to do much maintenance in the foreseeable future). But *maybe* there's a simpler solution. > OTOH, the specific problem I'm acutely worried about would be better > addressed by changing the way Python marhals float values. So solve it. The approach used by binary pickles seems entirely reasonable. All we need to do is change the .pyc magic number. (There's undoubtedly user code in the world that would break because it requires interoperability between Python versions. So let the marshal module grow a way to specify the format.) > [Guido] > > Maybe at least we can detect platforms for which we know there is a > > native conversion in the library, and not use the hack on those? > > I rarely find that piles of conditionalized code are more > comprehensible or reliable; they usually result in mysterious > x-platform differences, and become messier over time as we stumble > into more platform library bugs, quirks, and limitations. Fair enough. So *if* we decide to use the donated conversion code, we should start by using it unconditionally. I predict that at some point in the future we'll find a platform whose quirks are not handled by the donated code, and where it's simpler to use a correct native equivalent than to try to fix the donated code; but I expect that point to be pretty far in the future, *or* the platform to be pretty far from the main stream. > > ... > > Here's yet another idea (which probably has flaws as well): instead of > > substituting the locale's decimal separator, rewrite strings like > > "3.14" as "314e-2" and rewrite strings like "3.14e5" as "314e3", then > > pass to strtod(), which assigns the same meaning to such strings in > > all locales. > > This is a harder transformation than s/./locale_decimal_point/. It does > address the thread-safety issue. Numerically it's flaky, as only a > perfectly-rounding string->float routine can guarantee to return bit-for-bit > identical results given equivalent (viewed as infinite precision) decimal > representations as inputs, and few platform string->float routines do > perfect rounding. > > > This removes the question of what decimal separator is used by the > > locale completely, and thus removes the last bit of thread-unsafety > > from the code. However, I don't know if underflow can cause the result > > to be different, e.g. perhaps 1.23eX could be computed but 123e(X-2) > > could not??? (Sounds pretty unlikely on the face of it since I'd expect > > any decent conversion algorithm to pretty much break its input down into > > a string of digits and an exponent, but I've never actually studied > > such algorithms in detail.) > > Each library is likely fail in its own unique ways. Here's a cute one: > > """ > base = 1.2345678901234567 > > digits = "12345678901234567" > > for exponent in range(-16, -15000, -1): > string = digits + "0" * (-16 - exponent) > string += "e%d" % exponent > derived = float(string) > assert base == derived, (string, derived) > """ > > On Windows, this first fails at exponent -5202, where float(string) > delivers a result a factor of 10 too large. I was surprised it did > that well! Under Cygwin Python 2.2.3, it consumed > 14 minutes of > CPU time, but never failed. I believe they're using a derivative of > David Gay's excruciatingly complex IEEE-754 perfect-rounding > string<->float routines (which would explain both why it didn't fail > and why it consumed enormous CPU time; the code is excruciatingly > complex because it does perfect rounding quickly for "normal" > inputs, via a large variety of delicate speed tricks; when those > tricks don't apply, it has to simulate unbounded-precision > arithmetic to guarantee perfect rounding). I fail to see the relevance of the example to my proposed hack, except as a proof that the world isn't perfect -- but we already know that. Under my proposal, the number of digits converted would never change, so any sensitivity of the algorithm used to the number of digits converted would be irrelevant. I note that the strtod.c code that's currently in the Python source tree uses a similar (though opposite) trick: it converts the number to the form 0.E before handing it off to atof(). So my proposal still stands. I'm happy to entertain a proof that it's flawed but not one where the flawed input has over 5000 digits *and* depends on a flaw in the platform routines. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Sep 1 18:12:20 2003 From: guido at python.org (Guido van Rossum) Date: Mon Sep 1 20:13:09 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: Your message of "02 Sep 2003 00:18:33 +0200." References: <20030901194433.GK1917@async.com.br> <1062454091.3272.11.camel@emperor.homelinux.net> Message-ID: <200309020012.h820CKD23546@12-236-84-31.client.attbi.com> > It's more than that. We want you to fully understand the patch, and to > spot errors in it even before the glib developers find them. It's unlikely that the glib code would contain errors that Gustavo could spot, before or after cleaning up the patch (no offense to Gustavo meant!). However, a more likely cause of errors would be that the adoption of the code to a new environment breaks an unspoken assumption made by the code. Only truly understanding the code would reveal such assumptions. > We want > you to provide a minimalistic patch, that just implements the required > functionality, and nothing else. We want the patch to be maintainable > due to it being easy to read and follow, instead of being maintainable > due to the fact that it is identical with some code elsewhere in the > world. Right. Just say no to "copy-and-paste code reuse". --Guido van Rossum (home page: http://www.python.org/~guido/) From administrator at vt.edu Mon Sep 1 21:26:35 2003 From: administrator at vt.edu (administrator@vt.edu) Date: Mon Sep 1 20:26:44 2003 Subject: [Python-Dev] Virus Warning Message-ID: <200309020026.BPL01149@vivi.cc.vt.edu> The message you emailed to weiping@vt.edu, dated 09/01/03 20:26:35, contains the W32/Sobig-F virus in the document_9446.pif attachment. The action taken was: deleted the attachment. From europractice-bounces at ls12sn.cs.uni-dortmund.de Tue Sep 2 03:35:50 2003 From: europractice-bounces at ls12sn.cs.uni-dortmund.de (europractice-bounces@ls12sn.cs.uni-dortmund.de) Date: Mon Sep 1 20:36:00 2003 Subject: [Python-Dev] Your message to Europractice awaits moderator approval Message-ID: Your mail to 'Europractice' with the subject Re: Wicked screensaver Is being held until the list moderator can review it for approval. The reason it is being held: Post by non-member to a members-only list Either the message will get posted to the list, or you will receive notification of the moderator's decision. If you would like to cancel this posting, please visit the following URL: http://ls12-www.cs.uni-dortmund.de/mailman/confirm/europractice/51c62fe262577fff9b37d8ce7c4784dab164ac6a From tim.one at comcast.net Mon Sep 1 22:20:55 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Sep 1 21:21:29 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: <200309020003.h8203TD23456@12-236-84-31.client.attbi.com> Message-ID: [Tim] >> In short, I can't be enthusiastic about the patch because it doesn't >> solve the only relevant locale problem I've actually run into. I >> understand that it may well solve many I haven't run into. [Guido] > At this point in your life, Tim, is there any patch you could be truly > enthusiastic about? :-) Yes, but I can't be enthusiastic about a hack, and especially not about a hack that (as I said) doesn't solve the real-life problem spambayes has. > I'm asking because I'd like to see the specific problem that started > this thread solved, At this point, can you state what that specific problem was ? > if necessary using a compromise that means the solution isn't perfect. > I'm even willing to take a step back in the status quo, given that the > status quo isn't perfect anyway, and that compromises mean something has > to give. > > *Maybe* the right solution is that we have to accept a > hard-to-understand overcomplicated piece of code that we don't know > how to maintain (but for which the author asserts that we won't have > to do much maintenance in the foreseeable future). I'm finding it hard to believe that anyone other than me and the author has actually read the patch! It's easy to understand. It's over-complicated for what Python needs, and would be dead easy to understand if the fluff got chopped. The *fear* of this code expressed in this thread is baffling to me, but I suspect it's due to initial shell-shock from the sheer bulk of the unnecessary code in the patch. > But *maybe* there's a simpler solution. >> OTOH, the specific problem I'm acutely worried about would be better >> addressed by changing the way Python marhals float values. > So solve it. Sorry, I don't foresee making time to do that. > The approach used by binary pickles seems entirely reasonable. It's the best binary format we've got. It has problems with 754's special values (as recorded in PEP 42), and loses precision for VAX D format doubles (any double format with greater dynamic range or precision than IEEE-754 double). A decimal string is actually better on all those counts (dynamic range is no problem then; and *some* platforms can preserve IEEE special values via to-string-and-back conversion (Windows cannot)). Decimal strings lose on correctness only because of locale variations; depending on platform, they may also lose on speed, but I don't give much weight to speed here. > All we need to do is change the .pyc magic number. (There's undoubtedly > user code in the world that would break because it requires > interoperability between Python versions. So let the marshal module grow > a way to specify the format.) > ... > Fair enough. So *if* we decide to use the donated conversion code, we > should start by using it unconditionally. I predict that at some > point in the future we'll find a platform whose quirks are not handled > by the donated code, and where it's simpler to use a correct native > equivalent than to try to fix the donated code; but I expect that > point to be pretty far in the future, *or* the platform to be pretty > far from the main stream. Do read the patch. It amounts to if decimal_point != '.': s/./decimal_point/ in one direction and if decimal_point != '.': s/decimal_point/./ in the other. It gets its idea of decimal_point from the platform localeconv(), so if that doesn't lie it's hard to get wrong. In the double->string direction, though, the substitution code appears inadequate to me, since it doesn't try to strip out thousand-separation characters, which some locales produce. For example, on Windows, >>> locale.setlocale(locale.LC_ALL, "german") 'German_Germany.1252' >>> locale.format("%g", 123456.0, 1) '123.456' >>> AFAICT, the patch will leave that output as "123.456". The string->double direction is much easier to be confident about for this reason. >>> ... >>> Here's yet another idea (which probably has flaws as well): instead >>> of substituting the locale's decimal separator, rewrite strings like >>> "3.14" as "314e-2" and rewrite strings like "3.14e5" as "314e3", >>> then pass to strtod(), which assigns the same meaning to such >>> strings in all locales. [long example] > I fail to see the relevance of the example to my proposed hack, except > as a proof that the world isn't perfect -- but we already know that. The point is that only perfect-rounding string->float routines can guarantee to produce identical doubles from mathematically equivalent decimal string representations. Finding counterexamples for non-perfect-rounding libraries is extremely difficult, and/or time-consuming, without studying the source code of a specific library intensely (almost certainly with more intensity than its author gave to writing it!), and I don't have time for that. It's a potential vulnerability. Answering whether it's an actual vulnerability in practice is much more work than I can give to it now. > Under my proposal, the number of digits converted would never change, > so any sensitivity of the algorithm used to the number of digits > converted would be irrelevant. I note that the strtod.c code that's > currently in the Python source tree uses a similar (though opposite) > trick: it converts the number to the form 0.E before > handing it off to atof(). So my proposal still stands. I'm happy to > entertain a proof that it's flawed but not one where the flawed input > has over 5000 digits *and* depends on a flaw in the platform routines. As hacks go, it's probably OK. I don't think it can fail on glibc-based platforms because I think they do perfect-rounding conversions; the Windows conversion routines aren't perfect-rounding, but we don't have their source code so it's impossible for me to give examples offhand where different results could be delivered, or even to swear that there are (or aren't) such cases. I give it a lot of credit for being truly threadsafe. Note that it doesn't address the other half of the locale conversion problem (double->string), which, as I noted above, is the harder half (due to thousands_sep becoming an additional issue). From tim.one at comcast.net Mon Sep 1 22:27:17 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Sep 1 21:27:51 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: <20030901194433.GK1917@async.com.br> Message-ID: [Christian Reis] > ... > Are these representations (NaN, infinity, etc) LC_NUMERIC-dependent? > Or, more generally, locale-dependent? Don't know; C89 didn't say anything at all about them, so existing C practice is all over the map; C99 does say something about them, but whether a C99 compiler supports them is optional (support for them isn't mandatory; if a C implementation does choose to support them, then the spellings for input are standardized, although a locale is allowed to *produce* any spellings whatsoever). From martin at v.loewis.de Tue Sep 2 05:58:53 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Sep 2 00:58:54 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: References: Message-ID: "Tim Peters" writes: > > I'm asking because I'd like to see the specific problem that started > > this thread solved, > > At this point, can you state what that specific problem was ? The user was writing a gtk application (using pygtk), where gtk, internally, would rely on C-library LC_NUMERIC following the local conventions (I believe gtk would call snprintf to display some message to the user). The user then thought that calling locale.setlocale would be sufficient, but it isn't, and there is no way to fix that short of rewriting gtk. > I'm finding it hard to believe that anyone other than me and the author has > actually read the patch! It's easy to understand. On the surface, yes. However, it seems full of hidden assumptions that are difficult to find out and consider. For example, what if the platform snprintf choses to output the thousands-separator? I can't see how that handled in the patch. Regards, Martin From martin at v.loewis.de Tue Sep 2 06:00:30 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Sep 2 01:00:31 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: References: Message-ID: "Tim Peters" writes: > (support for them isn't mandatory; > if a C implementation does choose to support them, then the spellings for > input are standardized, although a locale is allowed to *produce* any > spellings whatsoever). Where exactly does it (C99) say that the spellings are locale-specific? Regards, Martin From europractice-bounces at ls12sn.cs.uni-dortmund.de Tue Sep 2 09:59:03 2003 From: europractice-bounces at ls12sn.cs.uni-dortmund.de (europractice-bounces@ls12sn.cs.uni-dortmund.de) Date: Tue Sep 2 02:59:14 2003 Subject: [Python-Dev] Your message to Europractice awaits moderator approval Message-ID: Your mail to 'Europractice' with the subject Re: Re: My details Is being held until the list moderator can review it for approval. The reason it is being held: Post by non-member to a members-only list Either the message will get posted to the list, or you will receive notification of the moderator's decision. If you would like to cancel this posting, please visit the following URL: http://ls12-www.cs.uni-dortmund.de/mailman/confirm/europractice/85d230b79086b9c6d4cbcc59acfbef0ac5d200ae From cwteoh at tm.net.my Tue Sep 2 20:49:38 2003 From: cwteoh at tm.net.my (PAUL YOUNG) Date: Tue Sep 2 07:59:17 2003 Subject: [Python-Dev] you r sending this on purpose or your pc got virus? please scan your pc References: <0HKJ008JPIIR2M@imta1.tm.net.my> Message-ID: <013c01c37149$7d501f20$0300000a@homerpcpoxxbh6> ----- Original Message ----- From: To: Sent: Monday, September 01, 2003 10:49 PM Subject: Re: Details > See the attached file for details > From sjoerd at acm.org Tue Sep 2 15:31:17 2003 From: sjoerd at acm.org (Sjoerd Mullender) Date: Tue Sep 2 08:31:27 2003 Subject: [Python-Dev] configure and setup.py Message-ID: <20030902123117.9333574130@indus.ins.cwi.nl> I am trying to build Python on a Unix system (Irix in this case, but that's irrelevant) and I am getting a bit frustrated. The problem is this. I tell configure where to find libraries and include files using options such as CPPFLAGS='-I...' and LDFLAGS='-L...' (you can provide those on the configure command line). Making Python then goes relatively smoothly. However, building the dynamically loadable modules with setup.py does *not* work smoothly. The main problem is that setup.py does not use all relevant information from the Makefile. I just submitted a bug report (with patch) for distutils so that it'll use the LDFLAGS variable from the Makefile, but that's not enough. Setup.py checks whether it can find certain libraries (readline comes to mind, that's the one that triggered my investigation). There are a number of extra system-dependent directories hard-coded in setup.py, but it totally ignores the directories that were specified at configure time. I think this is not the right way of doing things. If any knowledge about where compilers look for libraries should be built in anywhere, it's in distutils, not setup.py. Also, and more importantly, distutils should figure out the extra library directories that were specified at configure time and add the information to the list of searchable directories (CCompiler.library_dirs). If this were done, setup.py would find my -lreadline and the readline module would get compiled and I would be able to edit in my interactive Python. The same goes for include files. -- Sjoerd Mullender From tim.one at comcast.net Tue Sep 2 12:29:55 2003 From: tim.one at comcast.net (Tim Peters) Date: Tue Sep 2 11:30:33 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: Message-ID: [Tim] >> I'm finding it hard to believe that anyone other than me and the >> author has actually read the patch! It's easy to understand. [Martin] > On the surface, yes. However, it seems full of hidden assumptions that > are difficult to find out and consider. For example, what if the > platform snprintf choses to output the thousands-separator? I can't > see how that handled in the patch. I mentioned that one too last night -- it doesn't. OTOH, *are* there locales that insert thousands_sep? I don't know. To get thousands_sep to appear via Python's locale.format(), in all locales I've tried so far it requires passing a true value for the optional "grouping" argument. Like >>> import locale >>> locale.setlocale(locale.LC_ALL, "german") 'German_Germany.1252' >>> x = 12345.0 >>> locale.format("%g", x) # no thousands_sep '12345' >>> locale.format("%g", x, 1) '12.345' >>> Now going thru locale.py is far from going thru C, but the same thing happens if I use sprintf() directly from C (no thousands_sep appears, regardless of how I change locale). That's on Win2K. The draft std I have handy here sez: LC_NUMERIC affects the decimal-point character for the formatted input/output functions and the string conversion functions, as well as the nonmonetary formatting information returned by the localeconv function. There's no support there for the notion that "the formatted (etc)" functions *can* be affected by thousands_sep, just that fiddling locale can affect decimal-point and the (passive) values returned by localeconv(). From kiko at async.com.br Tue Sep 2 13:42:30 2003 From: kiko at async.com.br (Christian Reis) Date: Tue Sep 2 11:43:15 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: References: Message-ID: <20030902154230.GB1122@async.com.br> On Tue, Sep 02, 2003 at 11:29:55AM -0400, Tim Peters wrote: > OTOH, *are* there locales that insert thousands_sep? I don't know. To get > thousands_sep to appear via Python's locale.format(), in all locales I've > tried so far it requires passing a true value for the optional "grouping" > argument. Like > > >>> import locale > >>> locale.setlocale(locale.LC_ALL, "german") > 'German_Germany.1252' > >>> x = 12345.0 > >>> locale.format("%g", x) # no thousands_sep > '12345' > >>> locale.format("%g", x, 1) > '12.345' > >>> At least with locale.format, if you want grouping, you pass in the third argument (grouping=1) to format(). An example: >>> locale.setlocale(locale.LC_NUMERIC, 'da_DK') 'da_DK' >>> locale.format("%.2f", 71630, 1) '71.630,00' Now, from the glibc docs: [...] The SUSv2 specifies one further flag character. ' For decimal conversion (i, d, u, f, F, g, G) the output is to be grouped with thousands' grouping characters if the locale information indicates any. Note that many versions of gcc cannot parse this option and will issue a warning. SUSv2 does not include %'F. So unless I'm mistaken, this wouldn't really be an issue in our case if explicit grouping isn't requested inside the python conversion functions. Take care, -- Christian Reis, Senior Engineer, Async Open Source, Brazil. http://async.com.br/~kiko/ | [+55 16] 261 2331 | NMFL From jepler at unpythonic.net Tue Sep 2 11:44:03 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Tue Sep 2 11:44:55 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: References: Message-ID: <20030902154357.GC17554@unpythonic.net> On Tue, Sep 02, 2003 at 11:29:55AM -0400, Tim Peters wrote: > OTOH, *are* there locales that insert thousands_sep? I don't know. [...] > There's no support there for the notion that "the formatted (etc)" functions > *can* be affected by thousands_sep, just that fiddling locale can affect > decimal-point and the (passive) values returned by localeconv(). The Linux/glibc documentation, which cites SUSv2, seems to imply that no locale inserts the thousands separator in formatting operations, except when the ' flag character is included: For some numeric conversions a radix character ('decimal point') or thousands' grouping character is used. The actual character used depends on the LC_NUMERIC part of the locale. The POSIX locale uses '.' as radix character, and does not have a grouping character. Thus, printf("%'.2f", 1234567.89); results in '1234567.89' in the POSIX locale, in '1234567,89' in the nl_NL locale, and in '1.234.567,89' in the da_DK locale. [...] The five flag characters above are defined in the C standard. The SUSv2 specifies one further flag character. ' For decimal conversion (i, d, u, f, F, g, G) the output is to be grouped with thousands' grouping characters if the locale infor- mation indicates any. Note that many versions of gcc cannot parse this option and will issue a warning. SUSv2 does not include %'F. Jeff From tim.one at comcast.net Tue Sep 2 12:49:00 2003 From: tim.one at comcast.net (Tim Peters) Date: Tue Sep 2 11:49:39 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: Message-ID: [Tim] >> if a C implementation does choose to support them, then the >> spellings for input are standardized, although a locale is allowed >> to *produce* any spellings whatsoever). [martin@v.loewis.de] > Where exactly does it (C99) say that the spellings are > locale-specific? I can't find anything in the std supporting the claim. For that matter, I can't find anything in the std supporting the notion that a locale is allowed to insert thousand-separator characters either (can you?). There's lots of stuff allowing a locale to *accept* locale-specific spellings (when parsing strings), in addition to the "C" locale spellings; the other direction (producing strings) appears much less permissive. From martin at v.loewis.de Tue Sep 2 22:16:50 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Sep 2 17:16:52 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: References: Message-ID: "Tim Peters" writes: > I can't find anything in the std supporting the claim. For that matter, I > can't find anything in the std supporting the notion that a locale is > allowed to insert thousand-separator characters either (can you?). No: I'm now convinced that sprintf is *forbidden* to insert the thousands separator. This is why POSIX added the '-flag (%'f); this will produce the thousands-separator. That said: Implementations might choose to ignore the standard in that respect. This issue just supports my thesis that the patch is complicated: If I have to read the C99 standard to find out whether it is correct, it must be complicated. I doubt either the submitters or the original author of the code did that exercise... > There's lots of stuff allowing a locale to *accept* locale-specific > spellings (when parsing strings), in addition to the "C" locale spellings; > the other direction (producing strings) appears much less permissive. Indeed. I'm not sure whether this is intentional, though. Regards, Martin From martin at v.loewis.de Thu Sep 4 19:17:34 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Sep 4 14:17:45 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: <3F5611F8.5060606@daa.com.au> References: <3F5611F8.5060606@daa.com.au> Message-ID: James Henstridge writes: > In case it is of any help, I put together a modified version of the > glib code that uses libpython support routines instead of glib support > routines at: > http://www.gnome.org/~james/python-float-convert.c This is indeed helpful. I like it much more than the code that is on SF; in particular, the elimination of the table is a good thing. > It might help people review the code, as it doesn't contain code > irrelevant to the discussion. The code could probably be shortened by > removing support for hexadecimal float representation in > ascii_strtod(). I would encourage that. There are other format characters that are not supported either, like the POSIX ' flag. Having limitations in that approach is fine - they just need to be documented. Regards, Martin From martin at v.loewis.de Thu Sep 4 20:25:19 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Sep 4 15:25:21 2003 Subject: Solutions for LC_NUMERIC, was Re: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: <20030903134155.GB649@async.com.br> References: <20030903134155.GB649@async.com.br> Message-ID: Christian Reis writes: > "On glibc-based systems, Python is safe from runtime changes in the > LC_NUMERIC locale category. On other platforms, changing LC_NUMERIC > at runtime may cause float conversion routines to fail. This may > impact the Python interpreter internally." I could accept such a limitation. However, I maintain that the aspects must be fully understood, and I agree with Guido that they must be documented in the documentation proper (i.e. as comments in the source code, or the user documentation). So I suggest that you proceed with the PEP and the implementation as-is. The PEP would need to be enhanced to summarize this discussion (perhaps listing the alternatives that you just mentioned); the patch would need to be enhanced to be more simple, readable, and in the style of the rest of the Python source code, and would need to include documentation changes and test cases (although some test cases probably already exist). Regards, Martin From pje at telecommunity.com Thu Sep 4 16:55:32 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Sep 4 15:55:42 2003 Subject: [Python-Dev] Undocumented functions in 'readline' module Message-ID: <5.1.1.6.0.20030904155048.00acfd20@mail.rapidsite.net> While working on the clear_history() patch today, I noticed that there are some readline functions in the current CVS that don't appear in the docs: redisplay() get_history_item() get_current_history_length() Are these intentionally undocumented for some reason, or should I submit a doc bug? From raymond.hettinger at verizon.net Wed Sep 3 16:55:46 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Thu Sep 4 17:34:49 2003 Subject: [Python-Dev] object.__hash__() vs. hash() Message-ID: <000201c37256$5bee3e80$e841fea9@oemcomputer> object.__hash__() returns id(obj) by default. __builtin__.hash() uses PyObject_Hash() which is a bit smarter and avoids id(obj) whenever a comparison function is defined. Does anyone see a problem with my adding a similiar guard to object.__hash__()? This will prevent recurrence of a bug for newstyle objects which define a comparison function but not a hash function. For instance, slice(5).__hash__() is defined, but hash(slice(5)) raises a TypeError. An alternative solution is to change the logic in the slot inheritance to not inherit tp_hash whenever tp_compare or tp_richcompare is defined. This approach is a bit more tangled and may have unforeseen consequences, but has the advantage of preventing __hash__ from showing up method list for the subclass: dir(slice(5)) currently contains '__hash__'. I prefer the first approach of making object.__hash__() smarter. Raymond Hettinger From gminick at hacker.pl Thu Sep 4 23:04:48 2003 From: gminick at hacker.pl (gminick) Date: Thu Sep 4 18:21:22 2003 Subject: [Python-Dev] Discordance in documentation... Message-ID: <20030904200448.GA1164@hannibal> ...or is this just me? Let's take a look, Reference Lib, 4.2.1 Regular Expression Syntax says: "|" A|B, where A and B can be arbitrary REs, creates a regular expression that will match either A or B. [...] REs separated by "|" are tried from left to right, and the first one that allows the complete pattern to match is considered the accepted branch. This means that if A matches, B will never be tested, even if it would produce a longer overall match. [...] And now a little test: import re a = "Fuentes Rushdie Marquez" print re.search("Rushdie|Fuentes", a).group() # returns "Fuentes" According to the documentation I suspected it will return "Rushdie" rather than "Fuentes", but it looks like it returns first part of the string that matches rather than first part of regular expression. -- [ Wojtek Walczak - gminick (at) underground.org.pl ] [ ] [ "...rozmaite zwroty, matowe od patyny dawnosci." ] From postmaster at hermes.oceanet.fr Thu Sep 4 18:00:40 2003 From: postmaster at hermes.oceanet.fr (postmaster@hermes.oceanet.fr) Date: Thu Sep 4 18:24:58 2003 Subject: [Python-Dev] VIRUS DANS VOTRE MAIL Message-ID: <200309041500.h84F0es05747@hermes.oceanet.fr> A L E R T E V I R A L E Notre scanner anti-virus ā trouvé le virus W32/Sobig.f@MM dans le mail que vous avez envoyé aux adresses suivantes : -> Il serait préférable de passer votre systčme ā l'antivirus. Pour référence, voici les entętes du mail : ------------------------- BEGIN HEADERS ----------------------------- Received: from USER-LPXAO0WZPQ ([219.95.15.228]) by ns1.certif.net (8.11.6/8.11.6) with ESMTP id h84ExD806964 for ; Thu, 4 Sep 2003 16:59:15 +0200 Message-Id: <200309041459.h84ExD806964@ns1.certif.net> From: To: Subject: Re: Your application Date: Thu, 4 Sep 2003 22:59:52 +0800 X-MailScanner: Found to be clean Importance: Normal X-Mailer: Microsoft Outlook Express 6.00.2600.0000 X-MSMail-Priority: Normal X-Priority: 3 (Normal) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="_NextPart_000_00CB2F03" -------------------------- END HEADERS ------------------------------ From halley at play-bow.org Fri Sep 5 01:15:35 2003 From: halley at play-bow.org (Bob Halley) Date: Thu Sep 4 20:15:37 2003 Subject: [Python-Dev] Discordance in documentation... In-Reply-To: <20030904200448.GA1164@hannibal> References: <20030904200448.GA1164@hannibal> Message-ID: gminick writes: > import re > a = "Fuentes Rushdie Marquez" > print re.search("Rushdie|Fuentes", a).group() # returns "Fuentes" > > According to the documentation I suspected it will return "Rushdie" > rather than "Fuentes", but it looks like it returns first part of the > string that matches rather than first part of regular expression. You are assuming that re.search("Rushdie|Fuentes", a) is evaluated like m = re.search("Rushdie", a) if m is None: m = re.search("Fuentes", a) But it doesn't work that way. It works more like: m = None for i in xrange(0, len(a)): s = a[i:] m = re.match("Rushdie|Fuentes", s) if not m is None: break I.e., try to match the expression at the start of the string; if that doesn't work, try matching at the next character, etc. From Martin.vonLoewis at hpi.uni-potsdam.de Wed Sep 3 19:40:24 2003 From: Martin.vonLoewis at hpi.uni-potsdam.de (Loewis Martin von) Date: Thu Sep 4 20:27:39 2003 Subject: [Python-Dev] Proper usage of size_t Message-ID: <5AC0F51A926CD442BC08D55DA7DAB1D4692442@3MXMA1R.hpi.uni-potsdam.de> On Win64, the Windows SDK compiler gives tons of warnings that type conversions may involve truncation. For example, in 2.3 stringobject.c, a subset of the warnings is ..\Objects\stringobject.c(285) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data ..\Objects\stringobject.c(316) : warning C4244: 'function' : conversion from '__int64' to 'int', possible loss of data ..\Objects\stringobject.c(554) : warning C4244: 'function' : conversion from '__int64' to 'int', possible loss of data ..\Objects\stringobject.c(812) : warning C4267: 'function' : conversion from 'size_t' to 'int', possible loss of data ..\Objects\stringobject.c(2154) : warning C4244: 'function' : conversion from '__int64' to 'int', possible loss of data ..\Objects\stringobject.c(3410) : warning C4267: 'return' : conversion from 'size_t' to 'int', possible loss of data ..\Objects\stringobject.c(3618) : warning C4267: 'return' : conversion from 'size_t' to 'int', possible loss of data ..\Objects\stringobject.c(3737) : warning C4244: '=' : conversion from '__int64' to 'int', possible loss of data ..\Objects\stringobject.c(4072) : warning C4244: '=' : conversion from '__int64' to 'int', possible loss of data ..\Objects\stringobject.c(4076) : warning C4244: '=' : conversion from '__int64' to 'int', possible loss of data These warnings occur for stuff like i = strlen(p); where i is of type int, but strlen returns size_t, and that might exceed what int can store. Likewise, in _PyString_Resize(&string, s - PyString_AS_STRING(string)); the pointer difference is of type size_t, but PyString_Resize expects int. I'd be willing to go through and replace "int" by "size_t" whereever I find it appropriate, for Python 2.4. I would keep the current object layout for the moment, but change function signatures, which would cause ABI breakage on platforms where sizeof(size_t)!=sizeof(int). What do you think? Regards, Martin From tjreedy at udel.edu Thu Sep 4 21:43:31 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Thu Sep 4 20:43:39 2003 Subject: [Python-Dev] Re: Discordance in documentation... References: <20030904200448.GA1164@hannibal> Message-ID: "gminick" wrote in message news:20030904200448.GA1164@hannibal... > ...or is this just me? > > Let's take a look, Reference Lib, 4.2.1 Regular Expression Syntax says: It is the Library Reference, not Ref Lib. > "|" > A|B, where A and B can be arbitrary REs, creates a regular > expression that will match either A or B. > [...] > REs separated by "|" are tried from left to right, and the > first one that allows the complete pattern to match is considered > the accepted branch. This means that if A matches, B will never > be tested, even if it would produce a longer overall match. [...] I think the following version of the last four lines is correct and clearer. As the target string is scanned, REs separated by "|" are tried from left to right. When one pattern completely matches, that branch is accepted. This means that once A matches, B will not be tested further, even if it would produce a longer overall match. > And now a little test: > > import re > a = "Fuentes Rushdie Marquez" > print re.search("Rushdie|Fuentes", a).group() # returns "Fuentes" > > According to the documentation I suspected it will return "Rushdie" > rather than "Fuentes", but it looks like it returns first part of the > string that matches rather than first part of regular expression. As I hope my rewrite makes clearer, consideration of alternatives is nested within scanning of the source, and not vice verse as you inferred from the current doc. Doc bugs like this can (and should be) reported on SourceForge just like program bugs. http://sourceforge.net/tracker/?group_id=5470 That way, the report stays on the open list until someone decides either to make a fix or that no fix is needed. Terry J. Reedy From kiko at async.com.br Wed Sep 3 11:41:55 2003 From: kiko at async.com.br (Christian Reis) Date: Thu Sep 4 20:56:26 2003 Subject: Solutions for LC_NUMERIC, was Re: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: References: Message-ID: <20030903134155.GB649@async.com.br> On Tue, Sep 02, 2003 at 10:14:10PM -0400, Tim Peters wrote: > > That said: Implementations might choose to ignore the standard in that > > respect. This issue just supports my thesis that the patch is > > complicated: If I have to read the C99 standard to find out whether it > > is correct, it must be complicated. I doubt either the submitters or > > the original author of the code did that exercise... > > Well, any time a C library function gets called, you may have to scour the > standard to answer questions about it -- and if they're libm functions you > can just trust me that the standard doesn't contain useful answers . > That's what comments are for. Well, I disagree that the patch is complicated (it's not -- the code was preserved to simplify porting fixes to and from glib, but that's not a resolution and can be changed); the problem is that the corner cases here aren't trivial to grasp. We have libc functions that rely on global state (locale information), with no locking primitives, and no XP support for locale-independent API. When I wrote to python-dev initially, I wasn't aware of all the issues at hand; what I really wanted was to bring up the matter and try and find ways to solve the problem. The proposal I brought up is a WIP, and I expected it to change after initial discussion had come up (since all Martin had said previous to my PEP was "no" ;-) I have come to agree with Martin (but only after perceiving Tim's problem with Windows and Outlook) that it would be better if we found platform-specific locale-independent conversion functions. However, it seems that we won't be able to find them on *all* platforms (and, from Tim's findings -- I don't have access to Windows -- they're missing on a major platform). Now, agreeing on the solution depends on finding the correct compromise. Here's my reasoning: - The current implementation is broken -- it's not thread-safe, and it makes the locale inconsistent in applications that change LC_NUMERIC via the locale module. - We should try and use locale-independent primitives where available: they offer thread-safety, apart from solving the immediate problem. This is fixed with Gustavo's patch *on glibc platforms*, which uses strtod_l and the glibc 2.3 thread-aware uselocale() and snprintf() API. This is *not* fixed with Gustavo's patch for other platforms. - On platforms where locale-independent primitives are unavailable, we need to provide our own solution. There are three alternatives we have explored here: a) massage the data to/from a locale-accepted format and call the locale-dependent libc functions. This is *not* thread-safe. b) provide our own, complete, implementation of strtod and snprintf. c) link against a library that offers a locale-safe version of the functions. Alternatives b) and c) would be thread-safe. a) is not. The problem with alternative b) is finding somebody with the free cycles to implement the function *AND* convincing python-dev that the code is correct. Gustavo did some looking through the glibc code for the functions in question yesterday and, well, if people scoffed at code from glib, I dare them to review that code. I assume c) could be provided on Windows with cygwin. Solution b) above would solve the problem completely. Solution c) here would solve the problem on platforms where such libraries were available. If nobody can find, contribute, or accept any of the solutions, then I suggest using solution a) and release noting the problem with text such as "On glibc-based systems, Python is safe from runtime changes in the LC_NUMERIC locale category. On other platforms, changing LC_NUMERIC at runtime may cause float conversion routines to fail. This may impact the Python interpreter internally." Note that apart from the glibc part, this is also true today (it's just that a C library is the only thing able to change LC_NUMERIC right now). In the worst case (no thread-safe locale) we are not at lot worse than where we started -- we are not thread-safe, but at least we can get locale consistency for applications that don't toy with LC_NUMERIC at runtime (when it's none of their business ). Take care, -- Christian Reis, Senior Engineer, Async Open Source, Brazil. http://async.com.br/~kiko/ | [+55 16] 261 2331 | NMFL From europractice-bounces at ls12sn.cs.uni-dortmund.de Wed Sep 3 05:13:04 2003 From: europractice-bounces at ls12sn.cs.uni-dortmund.de (europractice-bounces@ls12sn.cs.uni-dortmund.de) Date: Thu Sep 4 20:58:15 2003 Subject: [Python-Dev] Your message to Europractice awaits moderator approval Message-ID: Your mail to 'Europractice' with the subject Re: Thank you! Is being held until the list moderator can review it for approval. The reason it is being held: Post by non-member to a members-only list Either the message will get posted to the list, or you will receive notification of the moderator's decision. If you would like to cancel this posting, please visit the following URL: http://ls12-www.cs.uni-dortmund.de/mailman/confirm/europractice/a5ca58b7a468a440d33b3d60f58917efc673db7b From tim.one at comcast.net Thu Sep 4 22:37:17 2003 From: tim.one at comcast.net (Tim Peters) Date: Thu Sep 4 21:37:25 2003 Subject: [Python-Dev] Proper usage of size_t In-Reply-To: <5AC0F51A926CD442BC08D55DA7DAB1D4692442@3MXMA1R.hpi.uni-potsdam.de> Message-ID: [Martin von Loewis] > On Win64, the Windows SDK compiler gives tons of warnings > that type conversions may involve truncation. For example, > in 2.3 stringobject.c, a subset of the warnings is > > ..\Objects\stringobject.c(285) : warning C4267: '=' : conversion > from 'size_t' to 'int', possible loss of data > ..\Objects\stringobject.c(316) : warning C4244: 'function' : > conversion from '__int64' to 'int', possible loss of data > ..\Objects\stringobject.c(554) : warning C4244: 'function' : > conversion from '__int64' to 'int', possible loss of data > ..\Objects\stringobject.c(812) : warning C4267: 'function' : > conversion from 'size_t' to 'int', possible loss of data > ..\Objects\stringobject.c(2154) : warning C4244: 'function' : > conversion from '__int64' to 'int', possible loss of data > ..\Objects\stringobject.c(3410) : warning C4267: 'return' : > conversion from 'size_t' to 'int', possible loss of data > ..\Objects\stringobject.c(3618) : warning C4267: 'return' : > conversion from 'size_t' to 'int', possible loss of data > ..\Objects\stringobject.c(3737) : warning C4244: '=' : conversion > from '__int64' to 'int', possible loss of data > ..\Objects\stringobject.c(4072) : warning C4244: '=' : conversion > from '__int64' to 'int', possible loss of data > ..\Objects\stringobject.c(4076) : warning C4244: '=' : conversion > from '__int64' to 'int', possible loss of data > > These warnings occur for stuff like > > i = strlen(p); > > where i is of type int, but strlen returns size_t, and that might > exceed what int can store. Likewise, in > > _PyString_Resize(&string, s - PyString_AS_STRING(string)); > > the pointer difference is of type size_t, but PyString_Resize > expects int. > > I'd be willing to go through and replace "int" by "size_t" > whereever I find it appropriate, for Python 2.4. I've done that often in the past, but piecemeal. Note that another raft of changes usually follows from this: it's not only the size of the integral type that may change, but regardless of platform it's also a change from a signed type to an unsigned type. That often triggers a new round of warnings (e.g., MSVC6 warns when mixing signed and unsigned in comparisons). > I would keep the current object layout for the moment, but change > function signatures, which would cause ABI breakage on platforms where > sizeof(size_t)!=sizeof(int). > > What do you think? It's overdue -- go for it! +1. From martin at v.loewis.de Fri Sep 5 04:56:51 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Sep 4 23:56:52 2003 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! In-Reply-To: <20030903065042.18889.qmail@web40107.mail.yahoo.com> References: <20030903065042.18889.qmail@web40107.mail.yahoo.com> Message-ID: Scott Gilbert writes: > I'm guessing you're doing housekeeping (instead of waiting diligently for > the implementation :-), so if you'd like to mark this PEP as abandoned that > makes sense to me... I was doing housekeeping indeed (of my mailbox); I wouldn't be pushing for a resolution on this PEP though. However, if everybody agrees: PEP editors, can you please mark PEP 296 as Withdrawn? Regards, Martin From mwh at python.net Wed Sep 3 13:08:13 2003 From: mwh at python.net (Michael Hudson) Date: Fri Sep 5 00:27:41 2003 Subject: [Python-Dev] readline.clear_history() In-Reply-To: (Steve Holden's message of "Mon, 1 Sep 2003 19:14:54 -0400") References: Message-ID: <2md6eiruxu.fsf@starship.python.net> "Steve Holden" writes: > [Michael Hudson] > >> And, come Panther, there are going to be rather a lot of Python 2.3 >> installs out there. >> > Just as a matter of interest, do you have any idea how many Mac OSX > licensees there are? Nope. Jaguar apparently shipped 100,000 copies in its first weekend (including preorders). It's also on all new macs. Million or so? Cheers, mwh -- I have a cat, so I know that when she digs her very sharp claws into my chest or stomach it's really a sign of affection, but I don't see any reason for programming languages to show affection with pain. -- Erik Naggum, comp.lang.lisp From guido at python.net Thu Sep 4 22:27:01 2003 From: guido at python.net (Guido van Rossum) Date: Fri Sep 5 00:32:07 2003 Subject: [Python-Dev] Undocumented functions in 'readline' module In-Reply-To: Your message of "Thu, 04 Sep 2003 15:55:32 EDT." <5.1.1.6.0.20030904155048.00acfd20@mail.rapidsite.net> References: <5.1.1.6.0.20030904155048.00acfd20@mail.rapidsite.net> Message-ID: <200309050427.h854R2k28334@12-236-84-31.client.attbi.com> > While working on the clear_history() patch today, I noticed that there are > some readline functions in the current CVS that don't appear in the docs: > > redisplay() > get_history_item() > get_current_history_length() > > Are these intentionally undocumented for some reason, or should I submit a > doc bug? By all means submit a fix! --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.net Thu Sep 4 22:31:21 2003 From: guido at python.net (Guido van Rossum) Date: Fri Sep 5 00:32:17 2003 Subject: [Python-Dev] Discordance in documentation... In-Reply-To: Your message of "Thu, 04 Sep 2003 22:04:48 +0200." <20030904200448.GA1164@hannibal> References: <20030904200448.GA1164@hannibal> Message-ID: <200309050431.h854VLm28352@12-236-84-31.client.attbi.com> > ...or is this just me? No, there are many others who are just as misguided. :-) > Let's take a look, Reference Lib, 4.2.1 Regular Expression Syntax says: > > "|" > A|B, where A and B can be arbitrary REs, creates a regular > expression that will match either A or B. > [...] > REs separated by "|" are tried from left to right, and the > first one that allows the complete pattern to match is considered > the accepted branch. This means that if A matches, B will never > be tested, even if it would produce a longer overall match. [...] > > And now a little test: > > import re > a = "Fuentes Rushdie Marquez" > print re.search("Rushdie|Fuentes", a).group() # returns "Fuentes" > > According to the documentation I suspected it will return "Rushdie" > rather than "Fuentes", but it looks like it returns first part of the > string that matches rather than first part of regular expression. There's probably an official answer for this, but maybe this helps you to explain it: re.search("A|B", s) doesn't first search s for A and then for B. Rather, for each successive position of s, it searches for a match *starting at that point* for "A|B". So when using search(), the first match wins. The rule about which branch of the | wins only applies when both match at the same point. (In general, anchored matching is the more fundamental re operation, and searching is done by trying to match at successive positions until a match is found.) --Guido van Rossum (home page: http://www.python.org/~guido/) From gminick at hacker.pl Fri Sep 5 09:51:28 2003 From: gminick at hacker.pl (gminick) Date: Fri Sep 5 03:15:16 2003 Subject: [Python-Dev] Re: Discordance in documentation... In-Reply-To: References: <20030904200448.GA1164@hannibal> Message-ID: <20030905065128.GA240@hannibal> On Thu, Sep 04, 2003 at 08:43:31PM -0400, Terry Reedy wrote: [...] > I think the following version of the last four lines is correct and > clearer. > > As the target string is scanned, REs separated by "|" are tried from > left to right. When one pattern completely matches, that branch is > accepted. This means that once A matches, B will not be tested > further, even if it would produce a longer overall match. Well IMHO this one is better, thanks. [...] > Doc bugs like this can (and should be) reported on SourceForge just > like program bugs. > http://sourceforge.net/tracker/?group_id=5470 I'll do it. I think it can be clearer. ps. Thanks to Bob Halley for excellent explanation with the code ;-) -- [ Wojtek Walczak - gminick (at) underground.org.pl ] [ ] [ "...rozmaite zwroty, matowe od patyny dawnosci." ] From Jack.Jansen at cwi.nl Fri Sep 5 11:02:15 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Fri Sep 5 04:02:20 2003 Subject: [Python-Dev] readline.clear_history() In-Reply-To: Message-ID: <43270311-DF77-11D7-9CE2-0030655234CE@cwi.nl> On Tuesday, September 2, 2003, at 01:14 AM, Steve Holden wrote: >> And, come Panther, there are going to be rather a lot of Python 2.3 >> installs out there. >> > Just as a matter of interest, do you have any idea how many Mac OSX > licensees there are? Apple sells about 3 million machines per year (taking their Q3-02 and Q3-03 figures and interpolating from that:-) and since last year these boot into OSX by default (and, since the last couple of months they only boot into OSX). Adding the conversion factor from older machines I come to about 5 million. And, surprise, that's also what some other people came up with using a different method: . But we're really interested in how many people will be running Panther, and I think that number is going to be lower. Very few people stuck with 10.1, basically only people whose machines wouldn't run 10.2 (pre-G3 processors), because 10.2 was really *way* ahead of 10.1. The 10.2->10.3 difference is probably less (to the non-Pythonic general community:-), so how about 1 million 10.3 users by the end of the year, 3-4 million by mid-2004? Any correspondence of these number with reality is pure chance, though:-) -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From bac at OCF.Berkeley.EDU Fri Sep 5 13:46:19 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Sep 5 15:46:32 2003 Subject: [Python-Dev] python-dev Summary for 2003-08-16 through 2003-08-31 [draft] [re-send] Message-ID: <3F58E80B.2090801@ocf.berkeley.edu> [since the original mailing of this didn't seem to get to the list I am re-sending] This is a rather light summary, so finding my usual grammatical errors shouldn't be too difficult. With so many people on vacation I will probably wait on sending this summary out for a little while. On a personal note, this is my 24th summary. What this means is I have now written enough summaries to cover a year's worth. If you care to see some stats on the list read the "Summary Announcements" section. And if anyone on the list wants the pickled data I have for this or just want to know a specific stat I can look it up for you and tell you. -------------------------------- python-dev Summary for 2003-08-16 through 2003-08-31 ++++++++++++++++++++++++++++++++++++++++++++++++++++ This is a summary of traffic on the `python-dev mailing list`_ from August 16, 2003 through August 31, 2003. It is intended to inform the wider Python community of on-going developments on the list. To comment on anything mentioned here, just post to python-list@python.org or `comp.lang.python`_ with a subject line mentioning what you are discussing. All python-dev members are interested in seeing ideas discussed by the community, so don't hesitate to take a stance on something. And if all of this really interests you then get involved and join `python-dev`_! This is the twenty-fourth summary written by Brett Cannon (a year's worth of summaries by yours truly now under his belt; does this mean I am certifiably insane?). All summaries are archived at http://www.python.org/dev/summary/ . Please note that this summary is written using reStructuredText_ which can be found at http://docutils.sf.net/rst.html . Any unfamiliar punctuation is probably markup for reST_ (otherwise it is probably regular expression syntax or a typo =); you can safely ignore it, although I suggest learning reST; its simple and is accepted for `PEP markup`_ and gives some perks for the HTML output. Also, because of the wonders of programs that like to reformat text, I cannot guarantee you will be able to run the text version of this summary through Docutils_ as-is unless it is from the original text file. .. _PEP Markup: http://www.python.org/peps/pep-0012.html The in-development version of the documentation for Python can be found at http://www.python.org/dev/doc/devel/ and should be used when looking up any documentation on something mentioned here. Python PEPs (Python Enhancement Proposals) are located at http://www.python.org/peps/ . To view files in the Python CVS online, go to http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/ . Reported bugs and suggested patches can be found at the SourceForge_ project page. .. _python-dev: http://www.python.org/dev/ .. _SourceForge: http://sourceforge.net/tracker/?group_id=5470 .. _python-dev mailing list: http://mail.python.org/mailman/listinfo/python-dev .. _comp.lang.python: http://groups.google.com/groups?q=comp.lang.python .. _Docutils: http://docutils.sf.net/ .. _reST: .. _reStructuredText: http://docutils.sf.net/rst.html .. contents:: .. _last summary: http://www.python.org/dev/summary/2003-08-01_2003-08-15.html ===================== Summary Announcements ===================== This is the twenty-fourth summary written by me. Why am I repeating this fact since it is mentioned above? Well, it is significant because this means I have written enough summaries to cover a year's worth of email traffic on python-dev (had I not taken a summary off back in October this milestone would have been hit for the first half of August which represented a physical year since I started doing the summaries). I have managed to see a lot happen on python-dev from a new release, the first PyCon, and a number of flame wars. I personally have managed to learn a *huge* amount about not just Python the language but how to program in Python, C, and how to handle a large programming project. It has truly been worth the experience. But how much of an experience has it been? Well, for a long time now I have been planning on writing some code to calculate how much email I have read, who wrote most of that email, and what threads made up the most. With my move to San Luis Obispo finished I finally had a chance to write said code in an imperfect manner (names, for instance, are a pain because some people have their name set differently at different computers; "Barry Warsaw" compared to "Barry A. Warsaw"; threads are worse thanks to the changing of subject titles in the middle of a thread) so as to give me some approximate numbers. I have read 9469 emails that have passed through the python-dev mailing list. The top six emailers (out of approx. 433 unique emailers) have been: * Brett Cannon (277 emails when you deal with me just using my last initial; 2.9% of all emails) * Barry Warsaw (305 emails when you also count his middle initial; 3.2%) * Skip Montanaro (481 emails; 5.1%) * Martin v. L?wis (627 emails, when calculated looking for all names that had "Martin" and "wis" in them; 6.6%) * Tim Peters (694 emails; 7.3%) * Guido van Rossum (a whopping 1407 emails; 14.8%) The average person posted 21.9 emails over the emails I covered. But only about 24 people had more than a single percentage (more than 94 emails) worth of emails accredited to them. That means that about 5.5% of the unique posters on python-dev accounted for 66.8% of all email (and I have gotten to know a good amount of those 24. As for threads (of which there were *very* approx. 1252 unique threads), the top five are: * "type categories" (115 emails; 1.2% of all emails) * "PEP239 (Rational Numbers) Reference Implementation and new issues" (123 emails; 1.3%) * "PEP-317" (125 emails: 1.3%) * "python/dist/src/Python import.c,2.210,2.211" (146 emails; 1.5%) * "Extended Function syntax" (263 emails; 2.8%) What does this tell you and me? First, that I contribute to my own pain by more than I thought (I can't believe I emailed that many times!). Second, I must *really* like reading. Third, I have an addiction to Python. Fourth, PEPs really do get discussed. And fifth, Python development is alive and well. OK, enough statistics. As for this summary, it turned out rather light thanks to a couple of things. One is the shutdown of mail delivery by mail.python.org during the SoBig virus' peak. This not only cut back on the number of emails, but also led to me deleting *a lot* of bogus email on my other emails accounts. My blanket deleting may have caught legitimate emails so it is possible I accidently deleted some python-dev stuff, although if I did it was minimal. Another contributing factor to the light summary is that a lot of regulars on python-dev were on vacation. This looks like it will happen again for the first half of September so expect the next summary to be light as well. ========= Summaries ========= ------------------------------------------------------------------------------ Python using Parrot; new code interpreter or strange evolutionary parternship? ------------------------------------------------------------------------------ Pirate_ has now reached version 0.01 alpha. It lacks classes and imports but can run a decent amount of Python code. At least there is now a proof-of-concept that Python running on top of the Parrot_ VM is possible. .. _Pirate: http://pirate.tangentcode.com/ .. _Parrot: http://www.parrotcode.org/ Contributing threads: - `pirate 0.01 alpha! `__ ------------------------ Python 2.3.1 on its way? ------------------------ Raymond Hettinger suggesting pushing for a quick release of Python 2.3.1 so that the 2.3 branch could be established as a stable version. Several bugs and performance enhancements have been committed to the 2.3 maintenance branch. Anthony Baxter stepped forward as release czar with Raymond Hettinger saying he would help and Barry Warsaw volunteering his wisdom as a battle-hardened release czar. This discussion also brought up the question of whether a .chm help file for the Windows distribution would be worth using instead of including the HTML distribution of the documentation as it stands now. It was agreed that it was a good thing to have since it allowed for better searching. Tim Peters also discovered the install went faster since it would not have to copy a ton of individual HTML files. Python 2.3.1 has a "verbal" release date of the third week of September; there has not been a PEP to set the release schedule officially. Help would be appreciated in dealing with bug and patch reports on SourceForge. Even if all you do is add a comment saying "this patch looks fine" or "I can reproduce this bug" it can be a great help. Contributing threads: - `Py2.3.1 `__ - `HTMLHelp for Py2.3.1 `__ - `Fixing Patches and Bugs for Py2.3.1 `__ ---------------------------------------- Making looping generators more efficient ---------------------------------------- Would you like to see deeply nested generators be more efficient in returning their values? Clark Evans would and made such a request. He essentially wanted to have nested generator calls propogate their values to the first non-generator call directly and thus bypass all of the generator maintenance code. There was no direct reaction to this. Shane Holloway followed with the idea of having special syntax for when you yield each value of an iterator. The idea, once again, would be to speed this common case in the interpreter by skipping some bookkeeping overhead. A few syntax versions were offered, but the idea was all the same: special-case ``for item in iterable: yield item`` to something like ``yield *iterable``. Contributing threads: - `cooperative generators `__ - `Graph exploration with generators `__ -------------------------------------------------- Use of the logging package in the standard library -------------------------------------------------- Want to help out the development of Python? Know how to use the logging package? Then python-dev wants you! There are several modules in the stdlib that have home-grown logging code that could (and probably should) be using the logging package instead to simplify life. Read the email that started the contributing thread and see if you can't help out by converting the module over to using the logging package today! Contributing threads: - `Unification of logging in Python's Standard Library `__ ----------------------- Some waxings on PEP 310 ----------------------- PEP 310 proposes the 'with' syntax that came up a while back that sparked an immense discussion on python-dev. The idea was to have a more fool-proof way of having an enter and exit method be called before executing some specified code. The common example was acquiring a lock, executing some code, and then releasing the lock all without having to deal with an explicit try/finally statement. Samuele Pedroni tried to clarify how it should work exactly by requiring __exit__ instead of making it optional (read the PEP to understand what this means). Contributing threads: - `PEP 310(with-syntax): close synonym of __exit__ `__ ------------------------------------------------ Proposed PEP for a 'close' method for generators ------------------------------------------------ Samuele Pedroni has written a pre-PEP on defining a way to have generators grow a way to have a 'close' method that is called when their execution is finished so as to handle resources correctly. This is in response to not being able to contain yield statements within try/finally blocks. Contributing threads: - `pre-PEP: Resource-Release Support for Generators `__ ----------------- email-sig created ----------------- Barry Warsaw has created the `email-sig`_ to steer development of version 3 of the email package in hopes of having it done for Python 2.4 . .. _email-sig: http://www.python.org/sigs/email-sig/ Contributing threads: - `New SIG: email-sig `__ From tim.one at comcast.net Sat Sep 6 14:11:14 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Sep 6 13:11:17 2003 Subject: [Python-Dev] Why do we flush before truncating? Message-ID: http://www.python.org/sf/801631 gives a failing program on Windows, paraphrased: f = file('test.dat', 'wb') f.write('1234567890') # 10 bytes f.close() f = file('test.dat','rb+') f.read(5) print f.tell() # prints 5, as expected f.truncate() # leaves the file at 10 bytes print f.tell() # prints 10 The problem is that fileobject.c's file_truncate() calls fflush() before truncating. The C standard says that the effect of calling fflush() is undefined if the most recent operation on a stream opened for update was an input operation. The stream is indeed opened for update here, and the most recent operation performed by the *user* was indeed a read. It so happens that MS's fflush() changes the file position then. But the user didn't call fflush(), Python did, so we can't blame the user for relying on undefined behavior here. The problem can be repaired inside file_truncate() by seeking back to the original file position after the fflush() call -- but the original file position isn't always available now, so I'd also have to add another call to _portable_ftell() before the fflush() to find it. So that gets increasingly complicated. Much simpler would be to remove this block of code (which does fix the test program's problem on Windows, by simply getting rid of the undefined operation): /* Flush the file. */ Py_BEGIN_ALLOW_THREADS errno = 0; ret = fflush(f->f_fp); Py_END_ALLOW_THREADS if (ret != 0) goto onioerror; I don't understand why we're flushing the file. ftruncate() isn't a standard C function, so the standard sheds no light on why we might be doing that. AFAICT, POSIX/SUS doesn't give a reason to flush either: http://www.opengroup.org/onlinepubs/007904975/functions/ftruncate.html From nas-python at python.ca Sat Sep 6 12:23:40 2003 From: nas-python at python.ca (Neil Schemenauer) Date: Sat Sep 6 14:16:45 2003 Subject: [Python-Dev] Why do we flush before truncating? In-Reply-To: References: Message-ID: <20030906182340.GA31698@mems-exchange.org> On Sat, Sep 06, 2003 at 01:11:14PM -0400, Tim Peters wrote: > I don't understand why we're flushing the file. ftruncate() isn't a > standard C function, so the standard sheds no light on why we might be doing > that. The fflush call as been there forever. The truncate method was added in 2.36 by Guido. I think the code was actually from Jim Roskind: http://groups.google.com/groups?selm=199412070213.SAA06932%40infoseek.com He says: Note that since the underlying ftruncate operates on a file descriptor (believe it or not), it was necessary to fflush() the stream before performing the truncate. I thought about doing a seek() as well, but could not find compelling reason to move the stream pointer. That still gives me no clue as to why the fflush() was deemed necessary. I found this posting: http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=35E0DB62.1BDD2D30%40taraz.kz&rnum=1 but, AFACK, the reason the program is not working the way the poster expects is the missing fflush() call before the lseek() call (not the fflush() before the ftruncate()). Neil From guido at python.org Sat Sep 6 12:53:36 2003 From: guido at python.org (Guido van Rossum) Date: Sat Sep 6 14:54:37 2003 Subject: [Python-Dev] Why do we flush before truncating? In-Reply-To: Your message of "Sat, 06 Sep 2003 13:11:14 EDT." References: Message-ID: <200309061853.h86IraQ31679@12-236-84-31.client.attbi.com> > http://www.python.org/sf/801631 > > gives a failing program on Windows, paraphrased: > > f = file('test.dat', 'wb') > f.write('1234567890') # 10 bytes > f.close() > > f = file('test.dat','rb+') > f.read(5) > print f.tell() # prints 5, as expected > > f.truncate() # leaves the file at 10 bytes > print f.tell() # prints 10 > > > The problem is that fileobject.c's file_truncate() calls fflush() before > truncating. The C standard says that the effect of calling fflush() is > undefined if the most recent operation on a stream opened for update was an > input operation. The stream is indeed opened for update here, and the most > recent operation performed by the *user* was indeed a read. It so happens > that MS's fflush() changes the file position then. But the user didn't call > fflush(), Python did, so we can't blame the user for relying on undefined > behavior here. > > The problem can be repaired inside file_truncate() by seeking back to the > original file position after the fflush() call -- but the original file > position isn't always available now, so I'd also have to add another call to > _portable_ftell() before the fflush() to find it. > > So that gets increasingly complicated. Much simpler would be to remove this > block of code (which does fix the test program's problem on Windows, by > simply getting rid of the undefined operation): > > /* Flush the file. */ > Py_BEGIN_ALLOW_THREADS > errno = 0; > ret = fflush(f->f_fp); > Py_END_ALLOW_THREADS > if (ret != 0) > goto onioerror; > > I don't understand why we're flushing the file. ftruncate() isn't a > standard C function, so the standard sheds no light on why we might be doing > that. AFAICT, POSIX/SUS doesn't give a reason to flush either: > > http://www.opengroup.org/onlinepubs/007904975/functions/ftruncate.html ftruncate() is not a standard C function; it's a standard Unix system call. It works on a file descriptor (i.e. a small int), not on a stream (i.e. a FILE *). The fflush() call is necessary if the last call was a write, because in that case the stream's buffer may contain data that the OS file descriptor doesn't have yet. But ftruncate() is irrelevant, because on Windows, it is never called; there's a huge #ifdef MS_WINDOWS block containing Windows specific code, starting with the comment /* MS _chsize doesn't work if newsize doesn't fit in 32 bits, so don't even try using it. */ and the ftruncate() call is made in the #else part. It also looks like the MS_WINDOWS specific code block *does* attempt to record the current file position and seek back to it -- however it does this after fflush() has already messed with it. So perhaps moving the fflush() call into the #else part and doing something Windows-specific instead of calling fflush() to ensure the buffer is flushed inside the MS_WINDOWS part would be the right solution. I just realize that I have always worked under the assumption that fflush() after a read is a no-op; I just checked the 89 std and it says it is undefined. (I must have picked up that misunderstanding from some platform-specific man page.) This can be fixed by doing a ftell() followed by an fseek() call; this is required to flush the buffer if there was unwritten output data in the buffer, and is always allowed. --Guido van Rossum (home page: http://www.python.org/~guido/) From bac at OCF.Berkeley.EDU Sat Sep 6 16:31:52 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sat Sep 6 18:32:42 2003 Subject: [Python-Dev] Changing select.select to accept iterables Message-ID: <3F5A6058.1080304@ocf.berkeley.edu> Bug #798046 basically requests that select.select accept iterables instead of only lists. I took a look at the code and beyond PyList_Check()s and a function called list2set that expects a Python list to be passed that can easily be changed to take in an iterable I see no need to keep the restriction. But this module is old and I suspect it is used a good amount (the thing started at ver. 2.1 by Guido back in 1992). Since it seems to have gone this long without being changed to use the iterator protocol I wanted to double-check that changing it would be okay. The only reason I can think of why it wouldn't be is performance, as suggested by the OP of the bug report. And in general, when something like this that smacks of, "there is no obvious reason to keep it this way", even though it could have been changed a while ago and the CVS log seems to not mention anything against the change, should the change just be made? Or should I just ask like this to be safe? I have a sneaking suspicion the answer is, "just change it, Brett! Use your judgment! Don't be so timid!" But then at least this way I minimize getting chewed out for breaking code. Then again, being yelled at makes you think like a Dutchman faster than anything I know of. =) -Brett From guido at python.org Sat Sep 6 17:04:40 2003 From: guido at python.org (Guido van Rossum) Date: Sat Sep 6 19:04:52 2003 Subject: [Python-Dev] Changing select.select to accept iterables In-Reply-To: Your message of "Sat, 06 Sep 2003 15:31:52 PDT." <3F5A6058.1080304@ocf.berkeley.edu> References: <3F5A6058.1080304@ocf.berkeley.edu> Message-ID: <200309062304.h86N4e601515@12-236-84-31.client.attbi.com> > Bug #798046 basically requests that select.select accept iterables > instead of only lists. I took a look at the code and beyond > PyList_Check()s and a function called list2set that expects a Python > list to be passed that can easily be changed to take in an iterable I > see no need to keep the restriction. > > But this module is old and I suspect it is used a good amount (the thing > started at ver. 2.1 by Guido back in 1992). Since it seems to have gone > this long without being changed to use the iterator protocol I wanted to > double-check that changing it would be okay. The only reason I can > think of why it wouldn't be is performance, as suggested by the OP of > the bug report. > > And in general, when something like this that smacks of, "there is no > obvious reason to keep it this way", even though it could have been > changed a while ago and the CVS log seems to not mention anything > against the change, should the change just be made? Or should I just > ask like this to be safe? I have a sneaking suspicion the answer is, > "just change it, Brett! Use your judgment! Don't be so timid!" But > then at least this way I minimize getting chewed out for breaking code. > > Then again, being yelled at makes you think like a Dutchman faster than > anything I know of. =) I seem to recall that that code has a long history of being hairy and full of platform-specific issues, and I'd rather not see it disturbed by an unspecific desire for more generalization. Why can't the OP produce the input in the form of lists? --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Sat Sep 6 22:54:34 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Sep 6 21:54:39 2003 Subject: [Python-Dev] Changing select.select to accept iterables In-Reply-To: <200309062304.h86N4e601515@12-236-84-31.client.attbi.com> Message-ID: [Brett, about ] [Guido] > I seem to recall that that code has a long history of being hairy > and full of platform-specific issues, and I'd rather not see it > disturbed by an unspecific desire for more generalization. Why can't > the OP produce the input in the form of lists? They could, but they specifically already have Sets of socket objects. That's what C would have used too for select(), if it had sets, so it's a natural desire. The SF report notes that when read and write lists change frequently, it's a lot more efficient to add/remove sockets via O(1) Set operations. Under the covers, select() setup takes O(N) time even if the inputs are native list. Functions that expect input lists of reasonably small size (thousands, sure; millions, probably not) can usually be generalized to iterables easily, with little chance of disruption, by replacing initial "is it a list?" check with a call to PySequence_Fast(), then s/PyList_GetItem/PySequence_Fast_GET_ITEM/. From bac at OCF.Berkeley.EDU Sat Sep 6 20:29:11 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sat Sep 6 22:30:22 2003 Subject: [Python-Dev] Changing select.select to accept iterables In-Reply-To: References: Message-ID: <3F5A97F7.7080700@ocf.berkeley.edu> Tim Peters wrote: > [Brett, about ] > > [Guido] > >>I seem to recall that that code has a long history of being hairy >>and full of platform-specific issues, and I'd rather not see it >>disturbed by an unspecific desire for more generalization. Why can't >>the OP produce the input in the form of lists? > > > They could, but they specifically already have Sets of socket objects. > That's what C would have used too for select(), if it had sets, so it's a > natural desire. The SF report notes that when read and write lists change > frequently, it's a lot more efficient to add/remove sockets via O(1) Set > operations. Under the covers, select() setup takes O(N) time even if the > inputs are native list. > Tim is right about there not being a restriction of not being able to create a list but wanting a better data structure for the job. And again I suspect Time is right about the socket API designers wanting sets if they had it in ANSI C; probably why they call it an fd_set for storing what file descriptors to work on. As for the code being hairy, yes, it does have its share of #ifdefs. But the list checks and code directly handling the lists are outside of any #ifdefs. The list codes is literally just the list checks and a 'for' loop pulling out each item in the list using PyList_GetItem; after that everything operates on what was pulled out of the list. But I understand the reluctance of messing with code that has a reputation and his a pain to test for something that is just a perk. > Functions that expect input lists of reasonably small size (thousands, sure; > millions, probably not) can usually be generalized to iterables easily, with > little chance of disruption, by replacing initial "is it a list?" check with > a call to PySequence_Fast(), then > s/PyList_GetItem/PySequence_Fast_GET_ITEM/. > Is this the preferred way of dealing with places where a small sequence and such will be instead of bothering with iterators, or is this a hold-over from pre-iterators and thus iterators are the proper way to go in new code and in old code where reasonable and relatively painless? Since Guido has previously expressed reluctance in having me touch this code I won't unless he tells me otherwise. -Brett From tim.one at comcast.net Sat Sep 6 23:40:16 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Sep 6 22:40:20 2003 Subject: [Python-Dev] Why do we flush before truncating? In-Reply-To: <20030906182340.GA31698@mems-exchange.org> Message-ID: [Neil Schemenauer] > The fflush call as been there forever. The truncate method was added > in 2.36 by Guido. I think the code was actually from Jim Roskind: > > http://groups.google.com/groups?selm=199412070213.SAA06932%40infoseek.com > > He says: > > Note that since the underlying ftruncate operates on a file > descriptor (believe it or not), it was necessary to fflush() the > stream before performing the truncate. I thought about doing a > seek() as well, but could not find compelling reason to move the > stream pointer. > > That still gives me no clue as to why the fflush() was deemed > necessary. Ack, I glossed over the fileno() call in our file_truncate(). It's usually a Very Bad Idea to mix stream I/O and lower-level I/O operations without flushing your pants off, but I'm having a hard time thinking of a specific reason for doing so in the truncate case. Better safe than trying to out-think all possible implementations, though! > I found this posting: > > http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=35E > 0DB62.1BDD2D30%40taraz.kz&rnum=1 > > but, AFACK, the reason the program is not working the way the poster > expects is the missing fflush() call before the lseek() call (not > the fflush() before the ftruncate()). I think that's right. In the Python case, I verified in a debugger that the file position is 5 immediately before the fflush() call, and 10 immediately after it. It's surprising, but apparently OK by the C std. [Guido > ftruncate() is not a standard C function; I suppose that clarifies my immediately preceding >> ftruncate() isn't a standard C function, ? > it's a standard Unix system call. Yes, and I gave a link to the current POSIX/SUS ftruncate() specification. > It works on a file descriptor (i.e. a small int), not on a > stream (i.e. a FILE *). Right, and I missed that, primarily because Windows doesn't have ftruncate() so I wasn't looking at that part of the code. > The fflush() call is necessary if the last call was a write, because in > that case the stream's buffer may contain data that the OS file > descriptor doesn't have yet. I'm not really clear on why that should matter in the specific case of truncating a file, but will just live with it. > But ftruncate() is irrelevant, because on Windows, it is never called; > there's a huge #ifdef MS_WINDOWS block containing Windows specific > code ... Right, I wrote that code. Windows has no way to say "here's a file, change the size to such-and-such"; the only way is to set the file pointer to the desired size, and then call the no-argument Win32 SetEndOfFile(); Python *used* to use the MS C _chsize() function, but that did insane things when passed a "large" size; the SetEndOfFile() code was introduced as part of fixing Python's Windows largefile support. > ... > It also looks like the MS_WINDOWS specific code block *does* attempt > to record the current file position and seek back to it Yes, because the file position must be changed on Windows in order to change the file size, but *Python's* docs promise that file.truncate() doesn't change the current position (which is natural behavior under POSIX ftruncate() but strained on Windows). > -- however it does this after fflush() has already messed with it. Note that in the Windows test case, it's not simply that the current position wasn't preserved across the file.truncate() call, it's also that the file didn't change size. It's very easy to fix the former while leaving the latter broken. > So perhaps moving the fflush() call into the #else part and doing > something Windows-specific instead of calling fflush() to ensure the > buffer is flushed inside the MS_WINDOWS part would be the right solution. > > I just realize that I have always worked under the assumption that > fflush() after a read is a no-op; I just checked the 89 std and it > says it is undefined. (I must have picked up that misunderstanding > from some platform-specific man page.) This can be fixed by doing a > ftell() followed by an fseek() call; this is required to flush the > buffer if there was unwritten output data in the buffer, and is always > allowed. That's what I was hoping to avoid, but I don't care anymore: after staring it some more, I'm convinced that the current file_truncate() endures a ridiculous amount of complexity trying to gain a tiny bit of speed in what has to be a rare operation. From tim.one at comcast.net Sat Sep 6 23:47:22 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Sep 6 22:47:27 2003 Subject: [Python-Dev] Changing select.select to accept iterables In-Reply-To: <3F5A97F7.7080700@ocf.berkeley.edu> Message-ID: [Tim] >> Functions that expect input lists of reasonably small size >> (thousands, sure; millions, probably not) can usually be generalized >> to iterables easily, with little chance of disruption, by replacing >> initial "is it a list?" check with a call to PySequence_Fast(), then >> s/PyList_GetItem/PySequence_Fast_GET_ITEM/. [Brett C.] > Is this the preferred way of dealing with places where a small > sequence and such will be instead of bothering with iterators, or is > this a hold-over from pre-iterators and thus iterators are the proper > way to go in new code and in old code where reasonable and relatively > painless? It's rare that you *know* an input will be at worst of reasonably small size, and that's why PySequence_Fast() isn't used more often (plus, I suppose, that it's relatively little known). It's perfect for select() inputs, which are not only certain to be reasonably small in working programs (no working program can use millions of sockets simultaneously), but also can't get any benefit from the "early out" possibilities afforded by using the iterator protocol instead. > Since Guido has previously expressed reluctance in having me touch > this code I won't unless he tells me otherwise. The change you'd like to make is very low-risk if you use the PySequence_Fast() approach, mainly simple editor substitutions, and Guido will like the idea of passing Sets to select() once he tries it . From guido at python.org Sat Sep 6 21:49:16 2003 From: guido at python.org (Guido van Rossum) Date: Sat Sep 6 23:49:22 2003 Subject: [Python-Dev] Changing select.select to accept iterables In-Reply-To: Your message of "Sat, 06 Sep 2003 19:29:11 PDT." <3F5A97F7.7080700@ocf.berkeley.edu> References: <3F5A97F7.7080700@ocf.berkeley.edu> Message-ID: <200309070349.h873nGr01747@12-236-84-31.client.attbi.com> > > [Brett, about ] > > > > [Guido] > > > >>I seem to recall that that code has a long history of being hairy > >>and full of platform-specific issues, and I'd rather not see it > >>disturbed by an unspecific desire for more generalization. Why can't > >>the OP produce the input in the form of lists? > > > > > > They could, but they specifically already have Sets of socket objects. > > That's what C would have used too for select(), if it had sets, so it's a > > natural desire. The SF report notes that when read and write lists change > > frequently, it's a lot more efficient to add/remove sockets via O(1) Set > > operations. Under the covers, select() setup takes O(N) time even if the > > inputs are native list. [Brett] > Tim is right about there not being a restriction of not being able to > create a list but wanting a better data structure for the job. > > And again I suspect Time is right about the socket API designers wanting > sets if they had it in ANSI C; probably why they call it an fd_set for > storing what file descriptors to work on. If Python had sets 10 years agon, select would've used sets too. :-) > As for the code being hairy, yes, it does have its share of #ifdefs. > But the list checks and code directly handling the lists are outside of > any #ifdefs. The list codes is literally just the list checks and a > 'for' loop pulling out each item in the list using PyList_GetItem; after > that everything operates on what was pulled out of the list. But I > understand the reluctance of messing with code that has a reputation and > his a pain to test for something that is just a perk. > > > Functions that expect input lists of reasonably small size > > (thousands, sure; millions, probably not) can usually be > > generalized to iterables easily, with little chance of disruption, > > by replacing initial "is it a list?" check with a call to > > PySequence_Fast(), then > > s/PyList_GetItem/PySequence_Fast_GET_ITEM/. > > > > Is this the preferred way of dealing with places where a small sequence > and such will be instead of bothering with iterators, or is this a > hold-over from pre-iterators and thus iterators are the proper way to go > in new code and in old code where reasonable and relatively painless? > > Since Guido has previously expressed reluctance in having me touch this > code I won't unless he tells me otherwise. Since Tim thinks it's okay to change (and since I now know what the OP wanted) you have my blessing to give this a try. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Sep 6 21:55:46 2003 From: guido at python.org (Guido van Rossum) Date: Sat Sep 6 23:55:51 2003 Subject: [Python-Dev] Why do we flush before truncating? In-Reply-To: Your message of "Sat, 06 Sep 2003 22:40:16 EDT." References: Message-ID: <200309070355.h873tkU01779@12-236-84-31.client.attbi.com> > Ack, I glossed over the fileno() call in our file_truncate(). It's usually > a Very Bad Idea to mix stream I/O and lower-level I/O operations without > flushing your pants off, but I'm having a hard time thinking of a specific > reason for doing so in the truncate case. Better safe than trying to > out-think all possible implementations, though! I think the fflush() is there for the following case: a file is opened for update, some data is written that overwrites some bytes in the middle but not yet flushed, and then the file is truncated to that position. Since ftruncate works on the file descriptor, you'd want the data flushed before truncating, otherwise things just get too complicated. > I suppose that clarifies my immediately preceding > > >> ftruncate() isn't a standard C function, > > ? I somehow misread what you wrote as "is a standard C function". > > I just realize that I have always worked under the assumption that > > fflush() after a read is a no-op; I just checked the 89 std and it > > says it is undefined. (I must have picked up that misunderstanding > > from some platform-specific man page.) This can be fixed by doing a > > ftell() followed by an fseek() call; this is required to flush the > > buffer if there was unwritten output data in the buffer, and is always > > allowed. > > That's what I was hoping to avoid, but I don't care anymore: after staring > it some more, I'm convinced that the current file_truncate() endures a > ridiculous amount of complexity trying to gain a tiny bit of speed in what > has to be a rare operation. Right. --Guido van Rossum (home page: http://www.python.org/~guido/) From bac at OCF.Berkeley.EDU Sat Sep 6 22:16:29 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sun Sep 7 00:17:25 2003 Subject: [Python-Dev] Changing select.select to accept iterables In-Reply-To: <200309070349.h873nGr01747@12-236-84-31.client.attbi.com> References: <3F5A97F7.7080700@ocf.berkeley.edu> <200309070349.h873nGr01747@12-236-84-31.client.attbi.com> Message-ID: <3F5AB11D.4070409@ocf.berkeley.edu> Guido van Rossum wrote: > > Since Tim thinks it's okay to change (and since I now know what the OP > wanted) you have my blessing to give this a try. > OK. I will go with Tim's suggestion and just do the brain-dead PySequence_Fast/PySequence_Fast_GET_ITEM substitution solution and not bother with the PyObject_GetIter/PyIter_Next solution. -Brett From bac at OCF.Berkeley.EDU Sat Sep 6 23:32:25 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sun Sep 7 01:33:22 2003 Subject: [Python-Dev] Changing select.select to accept iterables In-Reply-To: <200309070349.h873nGr01747@12-236-84-31.client.attbi.com> References: <3F5A97F7.7080700@ocf.berkeley.edu> <200309070349.h873nGr01747@12-236-84-31.client.attbi.com> Message-ID: <3F5AC2E9.2010207@ocf.berkeley.edu> Guido van Rossum wrote: >>>[Brett, about ] > > Since Tim thinks it's okay to change (and since I now know what the OP > wanted) you have my blessing to give this a try. > OK, done. I tore out the PyList_Check in select_select() and just put the PySequence_Fast() call in list2set(), which I renamed seq2set(), so as to centralize the code and make managing refcounts simpler. I pasted in an interpreter session at the end of this email showing some checking I did to make sure I didn't botch anything. If someone could double-check them and see if they can think of something I may have missed I would appreciate it. I made the docstring (as seen below) say that the arguments must be sequences since that is what PySequence_Fast says it takes in, although it obviously works with containers. Any objections? Assuming no one can think of anything that I missed I will then commit (with appropriate Misc/NEWS and doc changes), change the bug report to an feature request since that is what it really is, and close it. -Brett [passes test_select...] >>> import test.test_select timeout = 0 'testing...\n' timeout = 1 'testing...\n' timeout = 2 'testing...\n' timeout = 4 'testing...\n' timeout = 8 'testing...\n' timeout = 16 'testing...\n' timeout = None 'testing...\n' timeout = None 'testing...\n' timeout = None 'testing...\n' timeout = None 'testing...\n' timeout = None '' EOF [24363 refs] [works with sets as requested...] >>> from sets import Set [26557 refs] >>> from select import select [26559 refs] >>> read_FILE = file("README", 'r') [26567 refs] >>> write_FILE = file("@test", 'w') [26574 refs] >>> print select(Set([write_FILE]), Set([read_FILE]), [], 0) ([], [], []) [26607 refs] [modified docstring to say it works with sequences...] >>> help(select) Help on built-in function select: select(...) select(rlist, wlist, xlist[, timeout]) -> (rlist, wlist, xlist) Wait until one or more file descriptors are ready for some kind of I/O. The first three arguments are sequences of file descriptors to be waited for: ... [new error message on improper arguments...] >>> print select(1, 2, 3, 0) Traceback (most recent call last): File "", line 1, in ? TypeError: arguments 1-3 must be sequences [39435 refs] [managed to not screw up refcounts =) ...] >>> print select(Set([write_FILE]), Set([read_FILE]), [], 0) ([], [], []) [39435 refs] >>> print select(Set([write_FILE]), Set([read_FILE]), [], 0) ([], [], []) [39435 refs] [... and of course still works the way it originally did.] >>> print select([write_FILE], [read_FILE], [], 0) ([], [], []) [39435 refs] From skip at mojam.com Sun Sep 7 08:00:34 2003 From: skip at mojam.com (Skip Montanaro) Date: Sun Sep 7 08:00:43 2003 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200309071200.h87C0Yi17146@manatee.mojam.com> Bug/Patch Summary ----------------- 467 open / 4093 total bugs (-1) 185 open / 2358 total patches (-2) New Bugs -------- absolute import patch breaks external users of test.regrtest (2003-08-31) http://python.org/sf/798274 Clarify trailing comma in func arg list (2003-09-01) http://python.org/sf/798652 windows sys.path contains nonexistant directory (2003-09-01) http://python.org/sf/798876 distutils ignored LDFLAGS in Makefile (2003-09-02) http://python.org/sf/799088 CPPFLAGS should not be aded to ldshard command (2003-09-02) http://python.org/sf/799104 documentation for sys.platform is unclear (2003-09-02) http://python.org/sf/799369 tk_focusNext() fails (2003-09-02) http://python.org/sf/799428 color highlighting not working on EDIT windows (2003-09-04) http://python.org/sf/800432 build fails (2003-09-04) http://python.org/sf/800710 test_dumbdbm failing (2003-09-04) http://python.org/sf/800824 Doc bug in PEP-0278 (2003-09-04) http://python.org/sf/800828 Making "|" directive from REs a bit clearer (2003-09-04) http://python.org/sf/800899 Bad RE in scanf example (2003-09-05) http://python.org/sf/801306 New slice behaviour. Bug or feature? (2003-09-05) http://python.org/sf/801349 New Patches ----------- More urllib2 examples (2003-08-31) http://python.org/sf/798244 imaplib : Add support for the THREAD command (2003-08-31) http://python.org/sf/798297 Improve "veryhigh.tex" API docs (2003-09-01) http://python.org/sf/798638 add HTTPResponse.getheaders() (2003-09-03) http://python.org/sf/800236 Add 'clear_history()' to 'readline' module (2003-09-04) http://python.org/sf/800697 zipimport 64-bit fix (2003-09-06) http://python.org/sf/801821 Adding rsplit() to string and unicode objects. (2003-09-06) http://python.org/sf/801847 Closed Bugs ----------- Fatal Python Error during program shutdown (2001-02-05) http://python.org/sf/231207 filecmp.dircmp case sensitivity bug (2001-08-20) http://python.org/sf/453515 Double underscore needs clarification (2002-02-19) http://python.org/sf/520325 setting file buffer size is unreliable (2002-09-02) http://python.org/sf/603724 __file__ attribute missing from dynamicly loaded module (2003-03-05) http://python.org/sf/698282 WINDOW in py_curses.h needs ncurses-devel (2003-03-11) http://python.org/sf/701751 Documentation formatting bugs (2003-04-25) http://python.org/sf/727692 recent-files defaulted to strange place and bad permissions (2003-07-31) http://python.org/sf/780887 pynche does not start (2003-07-31) http://python.org/sf/780996 urllib output: Worker thread.. (2003-08-08) http://python.org/sf/785584 build fails on Tru64 Unix (osf1V5) (2003-08-13) http://python.org/sf/788183 Unsupported 'locale' setting causes Idle to crash on startup (2003-08-13) http://python.org/sf/788378 sgmllib parser problem (2003-08-23) http://python.org/sf/793753 pyconfig.h defines _POSIX_C_SOURCE, conflicting with feature (2003-08-23) http://python.org/sf/793764 cygwin builds do not embed (2003-08-24) http://python.org/sf/794140 email module param parsing bug (2003-08-24) http://python.org/sf/794466 locale.setlocale(locale.LC_ALL, "de") raises exception (2003-08-29) http://python.org/sf/797447 Closed Patches -------------- Editing of __str__ and __repr__ docs (2003-04-25) http://python.org/sf/727789 curses has_key emulation fix (2003-06-23) http://python.org/sf/759208 Add a stream type and a merge type to itertoolsmodule.c (2003-07-19) http://python.org/sf/774414 os.path.exists should use os.access when possible (2003-08-10) http://python.org/sf/786237 explicitly provide a buffer in PyFile_SetBufSize() (2003-08-13) http://python.org/sf/788249 add SafeConfigParser to __all__ (2003-08-18) http://python.org/sf/790443 inconsistency with implementation(logging) (2003-08-19) http://python.org/sf/791153 timetuple() returns a struct_time (2003-08-20) http://python.org/sf/792338 __file__ attribute missing from dynamicly loaded module (2003-08-25) http://python.org/sf/794826 From hernan at orgmf.com.ar Sun Sep 7 16:52:36 2003 From: hernan at orgmf.com.ar (Boriz Izaguirre) Date: Sun Sep 7 09:54:50 2003 Subject: HTMLHelp for Py2.3.1 (Was: Re: [Python-Dev] Py2.3.1) Message-ID: [Thomas] >[Fred L. Drake] >> IDLE currently looks for the index.html file in a few places (which >> depend on platform); if it can't find it, it uses the documentation on >> python.org. >> >> It should be too hard to change it to load the HTML Help viewer if it >> finds the .chm file on Windows, and to still fall back to the HTML or >> the online documentation if the .chm can't be found. > >Changing it is trivial, EditorWindow.help_url must point to Python23.chm >(if it exists). I can do this. Beware that for .html you want webbrowser.open(url) and for .chm you want os.startfile(url) There used to be a patch in Idle-dev for this. http://tinyurl.com/mj1s >Even nicer would be context-sensitive keyword help, but it seems IDLE >doesn't support it, right? Standard .chm format already includes useful index data to do context-sensitive search. A couple of years ago I made an Idle extension to manage this. It wasn't difficult then, and I think it's even easier now. The problem I found is that you pass the selected text (the one you are looking for) to HTMLHelp system by calling a Win32 API. You'll need win32all installed. That's the way PythonWin works, by the way. I guess this makes it a no-no for standard Python, right? Regards, -Hernan From goodger at python.org Sun Sep 7 10:56:03 2003 From: goodger at python.org (David Goodger) Date: Sun Sep 7 09:57:11 2003 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! In-Reply-To: References: <20030903065042.18889.qmail@web40107.mail.yahoo.com> Message-ID: <3F5B38F3.8050302@python.org> Martin v. L?wis wrote: > PEP editors, can you please mark PEP 296 as Withdrawn? Done. -- David Goodger From gward at python.net Sun Sep 7 13:19:00 2003 From: gward at python.net (Greg Ward) Date: Sun Sep 7 12:19:04 2003 Subject: [Python-Dev] Documenting branch policy Message-ID: <20030907161900.GA3195@cthulhu.gerg.ca> [cc'ing pydotorg since this involves a change to web content, but followup to python-dev please] Has anyone taken the time to sit down and document the development process surrounding release and maintenance branches? Speaking as an infrequent contributor to Python, it's never been entirely clear to me how to deal with branches. Seems to me like adding a paragraph or two to this page: http://www.python.org/dev/process.html would be a good start. Here's a first crack, based largely on guesswork, extrapolation from my day job (which also uses maintenance branches), and common sense: """

CVS Branch Policy

The Python development process uses CVS branches for a couple of purposes:

  • maintenance branches, for minor Python releases (2.2.3, 2.3.1, etc.)
  • release branches, for the exclusive use of the release manager(s)
  • feature branches, for large, disruptive additions/changes that take a long time to stabilize
Maintenance branches

Every major Python version (2.1, 2.2, 2.3, etc.) is accompanied by a maintenance branch. Minor Python releases are always made from a maintenance branch -- e.g. Python 2.2.3 was made on the release22-maint branch, and Python 2.3.1 will be made on release23-maint. Maintenance branches are long-lived; they remain open as long as minor releases based on the corresponding major release are a possibility. Maintenance branches are for bug fixes only; maintaining backwards compatibility at every level is imperative. (In particular, all extension modules and bytecode for Python 2.x.y must continue to work with Python 2.x.y+1 -- binary compatibility in the C API and bytecode must be maintained.)

Any Python developer may checkin bug fixes on a maintenance branch; it is the release manager's responsibility to create the maintenance branch after releasing a new major Python version.

Release branches

Every release up to and including the final release for a new major Python version is accompanied by a release branches. For example, there were release branches for 2.3a1, 2.3a2, 2.3b1, 2.3b2, 2.3c1, and 2.3 itself; there will not be release branches for 2.3.1, 2.3.2, and so forth. Release branches allow the release manager to make small, last-minute changes without being affected by mainline development. It is the release manager's responsibility to merge changes from each release branch back onto the trunk after the release is finished.

Unless you are one of the release managers for a new major Python version, you should have nothing to do with release branches.

Feature branches

Occasionally, a new feature will be deemed large and disruptive enough that it will be added on a branch. For example, the "new-style classes" feature of Python 2.2 was added on descr-branch (the non-obvious name stems from descriptors, the new C-level data structure underlying new-style classes), which took several months to stabilize before it was merged onto the trunk. Managing a feature branch is tricky business, and the longer it takes to stabilize the new feature, the trickier it gets.

Feature branches are generally created and used by a single developer. Do not create a feature branch without discussing it on python-dev.

""" Please let me know how close this is to describing reality! Greg -- Greg Ward http://www.gerg.ca/ Cheops' Law: Nothing *ever* gets built on schedule or within budget. From martin at v.loewis.de Sun Sep 7 20:16:07 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sun Sep 7 15:16:10 2003 Subject: [Python-Dev] Re: [Pydotorg] Documenting branch policy In-Reply-To: <20030907161900.GA3195@cthulhu.gerg.ca> References: <20030907161900.GA3195@cthulhu.gerg.ca> Message-ID: Greg Ward writes: > Has anyone taken the time to sit down and document the development > process surrounding release and maintenance branches? You mean PEP 6 and PEP 102? > Maintenance branches are for bug fixes > only; This question is heavily debated. Alex Martelli, for example, favours a policy where new (in a strict sense) features are acceptable if they don't break anything, and are "minor", in some sense. > maintaining backwards compatibility at every level is imperative. > (In particular, all extension modules and bytecode for Python > 2.x.y must continue to work with Python > 2.x.y+1 -- binary compatibility in the C API and > bytecode must be maintained.)

Even that requirement got dropped at one time, on grounds of the specific API being irrelevant for all practical applications. >

Any Python developer may checkin bug fixes on a maintenance branch; > it is the release manager's responsibility to create the maintenance > branch after releasing a new major Python version.

The typical guidelines apply: If in doubt, post to SF. Regards, Martin From gward at python.net Sun Sep 7 16:30:05 2003 From: gward at python.net (Greg Ward) Date: Sun Sep 7 15:30:09 2003 Subject: [Python-Dev] Re: [Pydotorg] Documenting branch policy In-Reply-To: References: <20030907161900.GA3195@cthulhu.gerg.ca> Message-ID: <20030907193005.GA1106@cthulhu.gerg.ca> On 07 September 2003, Martin v. L?wis said: > Greg Ward writes: > > > Has anyone taken the time to sit down and document the development > > process surrounding release and maintenance branches? > > You mean PEP 6 and PEP 102? Ahh, PEP 6 is what I was looking for. Never mind my proposed change then. Thanks -- Greg -- Greg Ward http://www.gerg.ca/ "Question authority!" "Oh yeah? Says who?" From Jack.Jansen at cwi.nl Sun Sep 7 23:25:08 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Sun Sep 7 16:25:14 2003 Subject: [Python-Dev] New functionality in micro releases (was: Documenting branch policy) In-Reply-To: References: <20030907161900.GA3195@cthulhu.gerg.ca> Message-ID: <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> On 7-sep-03, at 21:15, Martin v. L?wis wrote: >> Maintenance branches are for bug fixes >> only; > > This question is heavily debated. Alex Martelli, for example, favours > a policy where new (in a strict sense) features are acceptable if they > don't break anything, and are "minor", in some sense. With Python 2.3 included in MacOSX 10.3 I would be heavily opposed to this. I know, I've done it myself all the time in the past (with MacPython for OS9), but with Python 2.3 we have the situation that the majority of Python installations (I think it's safe to take the guess that MacOSX 10.3 installations will soon outnumber all other Pythons together) will stay at 2.3 until the next release of MacOSX. This means that by adding even the slightest bit of functionality you will make life difficult for developers: they themselves will probably track Python releases, but their customers most likely won't. Note that the MacPython infrastructure (bundlebuilder and such) really invites people to distribute their applications to third parties. And judging by the discussions on pythonmac-sig a lot of people are already doing this. Even bug fixes already have an impact (because the developer won't see a bug that will show up on the deployment machine), but adding functionality would exacerbate the issue. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From just at letterror.com Sun Sep 7 23:36:43 2003 From: just at letterror.com (Just van Rossum) Date: Sun Sep 7 16:36:30 2003 Subject: [Python-Dev] New functionality in micro releases (was: Documenting branch policy) In-Reply-To: <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> Message-ID: Jack Jansen wrote: > On 7-sep-03, at 21:15, Martin v. L?wis wrote: > > This question is heavily debated. Alex Martelli, for example, > > favours a policy where new (in a strict sense) features are > > acceptable if they don't break anything, and are "minor", in some > > sense. > > With Python 2.3 included in MacOSX 10.3 I would be heavily opposed to > this. I know, I've done it myself all the time in the past (with > MacPython for OS9), but with Python 2.3 we have the situation that > the majority of Python installations (I think it's safe to take the > guess that MacOSX 10.3 installations will soon outnumber all other > Pythons together) will stay at 2.3 until the next release of MacOSX. It's been a huge pain with 2.2 vs. 2.2.1 even: all stuff written for 2.2.1 that uses booleans does _not_ work on plain 2.2, which is what's installed on OSX 10.2. Which sortof kills the advantage of having a Python installed with the OS to begin with. In retrospect, I would have been strongly against adding booleans to 2.2.1 for this reason. Apart from real bugs, I think every program that works on 2.x.y should work on 2.x.z, regardless of whether y > z or y < z. Just From guido at python.org Sun Sep 7 16:23:38 2003 From: guido at python.org (Guido van Rossum) Date: Sun Sep 7 18:24:09 2003 Subject: [Python-Dev] Re: [Pydotorg] Documenting branch policy In-Reply-To: Your message of "07 Sep 2003 21:15:51 +0200." References: <20030907161900.GA3195@cthulhu.gerg.ca> Message-ID: <200309072223.h87MNd311094@12-236-84-31.client.attbi.com> > > Maintenance branches are for bug fixes > > only; > > This question is heavily debated. Alex Martelli, for example, favours > a policy where new (in a strict sense) features are acceptable if they > don't break anything, and are "minor", in some sense. Which pretty much matches the policy I used informally before all this was written down in PEPs. (Admittedly, 1.5.2 went way beyond this, and I don't wish to go back to those days.) > > maintaining backwards compatibility at every level is imperative. > > (In particular, all extension modules and bytecode for Python > > 2.x.y must continue to work with Python > > 2.x.y+1 -- binary compatibility in the C API and > > bytecode must be maintained.)

> > Even that requirement got dropped at one time, on grounds of the > specific API being irrelevant for all practical applications. So the rule becomes all extension modules and bytecode "in actual use" must continue to work. BTW I don't think we've ever changed the bytecode in a micro release after 1.5.2, and I want to continue to be strict about that. > >

Any Python developer may checkin bug fixes on a maintenance branch; > > it is the release manager's responsibility to create the maintenance > > branch after releasing a new major Python version.

> > The typical guidelines apply: If in doubt, post to SF. Or discuss it here. (I personally don't see new bugs on SF, but I do read all of python-dev -- though this should come as no surprise, as according to Brett's stats, I also write most of it. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From hpk at trillke.net Mon Sep 8 01:50:00 2003 From: hpk at trillke.net (holger krekel) Date: Sun Sep 7 18:50:11 2003 Subject: [Python-Dev] PyPy "Berlin" sprint (29th Sep - 4th Oct 2003) Message-ID: <20030908005000.D22698@prim.han.de> hello python developers, the fourth PyPy coding sprint will take place in Berlin (Germany) from (morning of) 29th of September to (evening of) 4th of October 2003. All Python developers are welcome to join the sprint. See below for participation details. What is PyPy? ------------- PyPy is a remimplementation of Python in the Python language itself. Simplicity and flexibilty are the foremost goals. With PyPy we want it to be easy to extend the language or generate a more minimal language implementation. One key technique to making this feasible is to specialize our abstract/general PyPy-Python implementation into native code (e.g. C-code). Eventually the concepts of PSYCO and Stackless are to be integrated into PyPy. There is, of course, much more to it so feel free to consult some documents especially our Oscon-2003 paper on this page: http://codespeak.net/pypy/index.cgi?doc We know that many areas and issues are still somewhat vague although PyPy can already run many python programs [*]. Fortunately, Python is pretty tolerant about expressing ideas in vague ways. After all, hacking at PyPy aims (and already proved!) to preserve the fun of hacking in Python. History, status and Berlin goals -------------------------------- The project started in Jan/Feb 2003 with an initiative from Christian Tismer, Holger Krekel and Armin Rigo. During the course many more developers joined our sprints and there are now between 5 and 15 people involved in developing PyPy. The three sprints resulted in a fully working interpreter, some development infrastructure and the "standard object space" which is an abstraction for operations on standard (CPython) objects. There is no public release yet, you more or less have to install a subversion-client and checkout the trunk. Here is our 'howtosvn' which contains up-to-date clients for multiple platforms along with some instructions how to get started: http://codespeak.net/pypy/index.cgi?doc/devel/howtosvn.html Main goals for the Berlin sprint are - integration/enhancement of a Python parser and the compiler package - generating native code from our Python interpreter/types implementations - enhancing/correcting/completing what we have (various tasks) How to participate in Berlin (29th of Sept. to 4th of October) -------------------------------------------------------------- If you are interested to participate please subscribe at our sprint organization list http://codespeak.net/mailman/pypy-sprint and please list yourself at http://codespeak.net/moin/pypy/moin.cgi/SprintAttendants so that we can organize accomodation and arrange details. Costs should be pretty low (except from travelling costs, of course). Currently nine people are scheduled to come. Btw, the sprint starts early on the 29th of september so it's best to arrive on sunday, 28th. Also we probably want to have one day off during the sprint week to do some sight (or cafe+pub) seeing. Actually it's much better to participate the whole week to better get into it. cheers, holger [*] and currently only 20-40 thousand times slower than CPython! which clearly shows that we are successfully follow the "premature optimization is the root of all evil" maxime. From bac at OCF.Berkeley.EDU Wed Sep 3 02:11:03 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sun Sep 7 21:44:07 2003 Subject: [Python-Dev] python-dev Summary for 2003-08-16 through 2003-08-31 [draft] Message-ID: <3F55A217.50403@ocf.berkeley.edu> This is a rather light summary, so finding my usual grammatical errors shouldn't be too difficult. With so many people on vacation I will probably wait on sending this summary out for a little while. On a personal note, this is my 24th summary. What this means is I have now written enough summaries to cover a year's worth. If you care to see some stats on the list read the "Summary Announcements" section. And if anyone on the list wants the pickled data I have for this or just want to know a specific stat I can look it up for you and tell you. -------------------------------- python-dev Summary for 2003-08-16 through 2003-08-31 ++++++++++++++++++++++++++++++++++++++++++++++++++++ This is a summary of traffic on the `python-dev mailing list`_ from August 16, 2003 through August 31, 2003. It is intended to inform the wider Python community of on-going developments on the list. To comment on anything mentioned here, just post to python-list@python.org or `comp.lang.python`_ with a subject line mentioning what you are discussing. All python-dev members are interested in seeing ideas discussed by the community, so don't hesitate to take a stance on something. And if all of this really interests you then get involved and join `python-dev`_! This is the twenty-fourth summary written by Brett Cannon (a year's worth of summaries by yours truly now under his belt; does this mean I am certifiably insane?). All summaries are archived at http://www.python.org/dev/summary/ . Please note that this summary is written using reStructuredText_ which can be found at http://docutils.sf.net/rst.html . Any unfamiliar punctuation is probably markup for reST_ (otherwise it is probably regular expression syntax or a typo =); you can safely ignore it, although I suggest learning reST; its simple and is accepted for `PEP markup`_ and gives some perks for the HTML output. Also, because of the wonders of programs that like to reformat text, I cannot guarantee you will be able to run the text version of this summary through Docutils_ as-is unless it is from the original text file. .. _PEP Markup: http://www.python.org/peps/pep-0012.html The in-development version of the documentation for Python can be found at http://www.python.org/dev/doc/devel/ and should be used when looking up any documentation on something mentioned here. Python PEPs (Python Enhancement Proposals) are located at http://www.python.org/peps/ . To view files in the Python CVS online, go to http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/ . Reported bugs and suggested patches can be found at the SourceForge_ project page. .. _python-dev: http://www.python.org/dev/ .. _SourceForge: http://sourceforge.net/tracker/?group_id=5470 .. _python-dev mailing list: http://mail.python.org/mailman/listinfo/python-dev .. _comp.lang.python: http://groups.google.com/groups?q=comp.lang.python .. _Docutils: http://docutils.sf.net/ .. _reST: .. _reStructuredText: http://docutils.sf.net/rst.html .. contents:: .. _last summary: http://www.python.org/dev/summary/2003-08-01_2003-08-15.html ===================== Summary Announcements ===================== This is the twenty-fourth summary written by me. Why am I repeating this fact since it is mentioned above? Well, it is significant because this means I have written enough summaries to cover a year's worth of email traffic on python-dev (had I not taken a summary off back in October this milestone would have been hit for the first half of August which represented a physical year since I started doing the summaries). I have managed to see a lot happen on python-dev from a new release, the first PyCon, and a number of flame wars. I personally have managed to learn a *huge* amount about not just Python the language but how to program in Python, C, and how to handle a large programming project. It has truly been worth the experience. But how much of an experience has it been? Well, for a long time now I have been planning on writing some code to calculate how much email I have read, who wrote most of that email, and what threads made up the most. With my move to San Luis Obispo finished I finally had a chance to write said code in an imperfect manner (names, for instance, are a pain because some people have their name set differently at different computers; "Barry Warsaw" compared to "Barry A. Warsaw"; threads are worse thanks to the changing of subject titles in the middle of a thread) so as to give me some approximate numbers. I have read 9469 emails that have passed through the python-dev mailing list. The top six emailers (out of approx. 433 unique emailers) have been: * Brett Cannon (277 emails when you deal with me just using my last initial; 2.9% of all emails) * Barry Warsaw (305 emails when you also count his middle initial; 3.2%) * Skip Montanaro (481 emails; 5.1%) * Martin v. L?wis (627 emails, when calculated looking for all names that had "Martin" and "wis" in them; 6.6%) * Tim Peters (694 emails; 7.3%) * Guido van Rossum (a whopping 1407 emails; 14.8%) The average person posted 21.9 emails over the emails I covered. But only about 24 people had more than a single percentage (more than 94 emails) worth of emails accredited to them. That means that about 5.5% of the unique posters on python-dev accounted for 66.8% of all email (and I have gotten to know a good amount of those 24. As for threads (of which there were *very* approx. 1252 unique threads), the top five are: * "type categories" (115 emails; 1.2% of all emails) * "PEP239 (Rational Numbers) Reference Implementation and new issues" (123 emails; 1.3%) * "PEP-317" (125 emails: 1.3%) * "python/dist/src/Python import.c,2.210,2.211" (146 emails; 1.5%) * "Extended Function syntax" (263 emails; 2.8%) What does this tell you and me? First, that I contribute to my own pain by more than I thought (I can't believe I emailed that many times!). Second, I must *really* like reading. Third, I have an addiction to Python. Fourth, PEPs really do get discussed. And fifth, Python development is alive and well. OK, enough statistics. As for this summary, it turned out rather light thanks to a couple of things. One is the shutdown of mail delivery by mail.python.org during the SoBig virus' peak. This not only cut back on the number of emails, but also led to me deleting *a lot* of bogus email on my other emails accounts. My blanket deleting may have caught legitimate emails so it is possible I accidently deleted some python-dev stuff, although if I did it was minimal. Another contributing factor to the light summary is that a lot of regulars on python-dev were on vacation. This looks like it will happen again for the first half of September so expect the next summary to be light as well. ========= Summaries ========= ------------------------------------------------------------------------------ Python using Parrot; new code interpreter or strange evolutionary parternship? ------------------------------------------------------------------------------ Pirate_ has now reached version 0.01 alpha. It lacks classes and imports but can run a decent amount of Python code. At least there is now a proof-of-concept that Python running on top of the Parrot_ VM is possible. .. _Pirate: http://pirate.tangentcode.com/ .. _Parrot: http://www.parrotcode.org/ Contributing threads: - `pirate 0.01 alpha! `__ ------------------------ Python 2.3.1 on its way? ------------------------ Raymond Hettinger suggesting pushing for a quick release of Python 2.3.1 so that the 2.3 branch could be established as a stable version. Several bugs and performance enhancements have been committed to the 2.3 maintenance branch. Anthony Baxter stepped forward as release czar with Raymond Hettinger saying he would help and Barry Warsaw volunteering his wisdom as a battle-hardened release czar. This discussion also brought up the question of whether a .chm help file for the Windows distribution would be worth using instead of including the HTML distribution of the documentation as it stands now. It was agreed that it was a good thing to have since it allowed for better searching. Tim Peters also discovered the install went faster since it would not have to copy a ton of individual HTML files. Python 2.3.1 has a "verbal" release date of the third week of September; there has not been a PEP to set the release schedule officially. Help would be appreciated in dealing with bug and patch reports on SourceForge. Even if all you do is add a comment saying "this patch looks fine" or "I can reproduce this bug" it can be a great help. Contributing threads: - `Py2.3.1 `__ - `HTMLHelp for Py2.3.1 `__ - `Fixing Patches and Bugs for Py2.3.1 `__ ---------------------------------------- Making looping generators more efficient ---------------------------------------- Would you like to see deeply nested generators be more efficient in returning their values? Clark Evans would and made such a request. He essentially wanted to have nested generator calls propogate their values to the first non-generator call directly and thus bypass all of the generator maintenance code. There was no direct reaction to this. Shane Holloway followed with the idea of having special syntax for when you yield each value of an iterator. The idea, once again, would be to speed this common case in the interpreter by skipping some bookkeeping overhead. A few syntax versions were offered, but the idea was all the same: special-case ``for item in iterable: yield item`` to something like ``yield *iterable``. Contributing threads: - `cooperative generators `__ - `Graph exploration with generators `__ -------------------------------------------------- Use of the logging package in the standard library -------------------------------------------------- Want to help out the development of Python? Know how to use the logging package? Then python-dev wants you! There are several modules in the stdlib that have home-grown logging code that could (and probably should) be using the logging package instead to simplify life. Read the email that started the contributing thread and see if you can't help out by converting the module over to using the logging package today! Contributing threads: - `Unification of logging in Python's Standard Library `__ ----------------------- Some waxings on PEP 310 ----------------------- PEP 310 proposes the 'with' syntax that came up a while back that sparked an immense discussion on python-dev. The idea was to have a more fool-proof way of having an enter and exit method be called before executing some specified code. The common example was acquiring a lock, executing some code, and then releasing the lock all without having to deal with an explicit try/finally statement. Samuele Pedroni tried to clarify how it should work exactly by requiring __exit__ instead of making it optional (read the PEP to understand what this means). Contributing threads: - `PEP 310(with-syntax): close synonym of __exit__ `__ ------------------------------------------------ Proposed PEP for a 'close' method for generators ------------------------------------------------ Samuele Pedroni has written a pre-PEP on defining a way to have generators grow a way to have a 'close' method that is called when their execution is finished so as to handle resources correctly. This is in response to not being able to contain yield statements within try/finally blocks. Contributing threads: - `pre-PEP: Resource-Release Support for Generators `__ ----------------- email-sig created ----------------- Barry Warsaw has created the `email-sig`_ to steer development of version 3 of the email package in hopes of having it done for Python 2.4 . .. _email-sig: http://www.python.org/sigs/email-sig/ Contributing threads: - `New SIG: email-sig `__ From mfb at lotusland.dyndns.org Sun Sep 7 23:13:02 2003 From: mfb at lotusland.dyndns.org (Matthew F. Barnes) Date: Sun Sep 7 23:25:47 2003 Subject: [Python-Dev] Consistent logging in the standard library Message-ID: <4101.192.168.1.1.1062990782.squirrel@server.lotusland.dyndns.org> Having just read the "Use of the logging package in the standard library" section of Brett's latest python-dev summary [1], I thought I might try to generate some discussion on this topic since I brought it up in the first place. There was actually a bit of a longer discussion of this topic on comp.lang.python [2], where A.M. Kuchling pointed out a number of good issues that will need to be considered and suggested that I write a PEP on the topic. In fact I've been working on one since, but it's not yet ready to be submitted (it's my first, so it's taking awhile). The basic idea in the PEP is this: maintain a level of indirection between the logger objects and the modules or class instances invoking them, so that custom application-specific loggers can be substituted for the "standard" ones. The method for doing so is modelled after the socket module's new timeout functionality [3], where each module that performs logging defines a pair of module-level functions named getdefaultlogger() and setdefaultlogger() that return and accept (respectively) a logging.Logger instance. The initial default logger for each of these modules is named after the module itself (i.e. logging.getLogger(__name__)) and its behavior mimics that of the Python 2.3 version of the module in terms of format and destination of the logging messages. So an application that wants to override the logging behavior of a standard library module can do so simply by passing a customized logger object to the module's setdefaultlogger() function. This works out even better for modules where all the logging is encapsulated in a class. The PEP calls for each class that performs logging to define an attribute called self.logger, and upon instantiation initialize self.logger to the module's *current* default logger (again, modelled after how the timeout value for a socket object is initialized). This allows each instance of a class to have its own unique logging behavior. The first item for discussion is whether this seems like a reasonable approach. The second is identifying which standard library modules are candidates for modification using this approach. I've identified the following by simply searching for the word "logging" in the source code, but I'm not sure whether this list is complete: asyncore BaseHTTPServer cgi doctest imaplib unittest the distutils package I have patches ready for asyncore and BaseHTTPServer and doctest, which fit the model I described above quite nicely. I determined that no changes are necessary for the cgi module, since it doesn't even use the logging functions it defines (initlog, dolog, etc.). I'm currently working on a way to replicate imaplib's command history using LogRecords, which may call for a new type of logging.BufferingHandler subclass that maintains a circular queue of LogRecords. And the distutils package seems prepped for using the logging package already. The oddball is unittest. Unittest is what motivated me to bring up the logging issue in the first place, but I can't (or more precisely - am not in a position to) decide what direction that module should take w.r.t. if and how it should use the logging package. Should the TextTestRunner class be modified to use a logging.Logger instead of a _WritelnDecorator, or should a separate TestRunner subclass (a LoggingTestRunner perhaps) be defined instead? I'm hoping that Steve Purcell or others may have some thoughts on this. I would also appreciate hearing any alternatives to the approach I've suggested, as well as comments on the strengths and weaknesses of each. Matthew Barnes [1] [2] [3] From skip at pobox.com Sun Sep 7 19:04:33 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Sep 8 09:07:05 2003 Subject: [Python-Dev] New functionality in micro releases (was: Documenting branch policy) In-Reply-To: <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> References: <20030907161900.GA3195@cthulhu.gerg.ca> <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> Message-ID: <16219.47489.553300.299720@montanaro.dyndns.org> >>>>> "Jack" == Jack Jansen writes: Jack> On 7-sep-03, at 21:15, Martin v. L?wis wrote: >>> Maintenance branches are for bug fixes >>> only; >> >> This question is heavily debated. Alex Martelli, for example, favours >> a policy where new (in a strict sense) features are acceptable if they >> don't break anything, and are "minor", in some sense. Jack> ... but with Python 2.3 we have the situation that the majority of Jack> Python installations (I think it's safe to take the guess that Jack> MacOSX 10.3 installations will soon outnumber all other Pythons Jack> together) will stay at 2.3 until the next release of MacOSX. I wouldn't be so sure of that. Aside from the many Linux distributions which include some version of Python, it's clear that HP/Compaq is including some version of Python with their new Windows computers (based on all the newbie questions about this fielded at webmaster@python.org). Still, the number of vendors delivering Python with their computers and the long string of problems caused by RedHat shipping Python 1.5.2 w/ 7.x long after 2.0 was history suggests that micro releases be reserved entirely for bug fixes. Skip From barry at python.org Mon Sep 8 14:18:30 2003 From: barry at python.org (Barry Warsaw) Date: Mon Sep 8 09:18:32 2003 Subject: [Python-Dev] Documenting branch policy In-Reply-To: <20030907161900.GA3195@cthulhu.gerg.ca> References: <20030907161900.GA3195@cthulhu.gerg.ca> Message-ID: <1063027104.16344.9.camel@anthem> On Sun, 2003-09-07 at 12:19, Greg Ward wrote: >
Release branches
> >

Every release up to and including the final release for a new major > Python version is accompanied by a release branches. There's even some debate about these (I've caught up with the thread, so I know you've withdrawn this change). I've been backing off the use of release branches because they create more complexity in a world where none of us have much time to deal with it. For 2.3, python-dev was really good about heeding calls for checkin freezes, and the time between wanting to cut the release and actually doing it was pretty short, so I don't see much need for release branches. I updated PEP 101 to reflect this. -Barry From harri.pasanen at trema.com Mon Sep 8 16:39:56 2003 From: harri.pasanen at trema.com (Harri Pasanen) Date: Mon Sep 8 09:40:07 2003 Subject: [Python-Dev] New functionality in micro releases (was: =?iso-8859-1?q?Documenting branch?= policy) In-Reply-To: <16219.47489.553300.299720@montanaro.dyndns.org> References: <20030907161900.GA3195@cthulhu.gerg.ca> <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> <16219.47489.553300.299720@montanaro.dyndns.org> Message-ID: <200309081539.56700.harri.pasanen@trema.com> On Monday 08 September 2003 01:04, Skip Montanaro wrote: > >>>>> "Jack" == Jack Jansen writes: > > Jack> ... but with Python 2.3 we have the situation that the > majority of Jack> Python installations (I think it's safe to take > the guess that Jack> MacOSX 10.3 installations will soon outnumber > all other Pythons Jack> together) will stay at 2.3 until the next > release of MacOSX. > > I wouldn't be so sure of that. Aside from the many Linux > distributions which include some version of Python, it's clear that > HP/Compaq is including some version of Python with their new > Windows computers (based on all the newbie questions about this > fielded at webmaster@python.org). Still, the number of vendors > delivering Python with their computers and the long string of > problems caused by RedHat shipping Python 1.5.2 w/ 7.x long after > 2.0 was history suggests that micro releases be reserved entirely > for bug fixes. > Seconded, from where I'm sitting MaxOSX is not even on the radar, but Linux is all over the place, on various platforms, and all those we use have Python out of the box. RedHat on Itanium still seems to have 1.5.2 for instance. Mandrake 9.1 is on 2.2.2. Mandrake 9.2 will have Python 2.3. Personally I would have guessed there to be more Mandrake Linux installations than MacOSX installations - but in reality I have no idea. Just a datapoint... Harri From guido at python.org Mon Sep 8 07:59:02 2003 From: guido at python.org (Guido van Rossum) Date: Mon Sep 8 09:59:21 2003 Subject: [Python-Dev] New functionality in micro releases (was: Documenting branch policy) In-Reply-To: Your message of "Sun, 07 Sep 2003 18:04:33 CDT." <16219.47489.553300.299720@montanaro.dyndns.org> References: <20030907161900.GA3195@cthulhu.gerg.ca> <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> <16219.47489.553300.299720@montanaro.dyndns.org> Message-ID: <200309081359.h88Dx2G11829@12-236-84-31.client.attbi.com> > Jack> ... but with Python 2.3 we have the situation that the majority of > Jack> Python installations (I think it's safe to take the guess that > Jack> MacOSX 10.3 installations will soon outnumber all other Pythons > Jack> together) will stay at 2.3 until the next release of MacOSX. [Skip] > I wouldn't be so sure of that. Aside from the many Linux distributions > which include some version of Python, it's clear that HP/Compaq is including > some version of Python with their new Windows computers (based on all the > newbie questions about this fielded at webmaster@python.org). Still, the > number of vendors delivering Python with their computers and the long string > of problems caused by RedHat shipping Python 1.5.2 w/ 7.x long after 2.0 was > history suggests that micro releases be reserved entirely for bug fixes. I agree with Jack & Just that MacOSX is a strong argument for keeping new features, no matter how harmless, out of micro releases. But I'm not sure how Red Hat shipping 1.5.2 proves this point; there were no micro releases of 1.5 after that. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Mon Sep 8 10:09:55 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Sep 8 10:10:20 2003 Subject: [Python-Dev] New functionality in micro releases (was: Documenting branch policy) In-Reply-To: <200309081359.h88Dx2G11829@12-236-84-31.client.attbi.com> References: <20030907161900.GA3195@cthulhu.gerg.ca> <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> <16219.47489.553300.299720@montanaro.dyndns.org> <200309081359.h88Dx2G11829@12-236-84-31.client.attbi.com> Message-ID: <16220.36275.971661.113208@montanaro.dyndns.org> Guido> But I'm not sure how Red Hat shipping 1.5.2 proves this point; Guido> there were no micro releases of 1.5 after that. True, but it demonstrates how much agony changes to the language - regardless of the versioning - can cause. Skip From guido at python.org Mon Sep 8 08:19:48 2003 From: guido at python.org (Guido van Rossum) Date: Mon Sep 8 10:20:16 2003 Subject: [Python-Dev] New functionality in micro releases (was: Documenting branch policy) In-Reply-To: Your message of "Mon, 08 Sep 2003 09:09:55 CDT." <16220.36275.971661.113208@montanaro.dyndns.org> References: <20030907161900.GA3195@cthulhu.gerg.ca> <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> <16219.47489.553300.299720@montanaro.dyndns.org> <200309081359.h88Dx2G11829@12-236-84-31.client.attbi.com> <16220.36275.971661.113208@montanaro.dyndns.org> Message-ID: <200309081419.h88EJmA11895@12-236-84-31.client.attbi.com> > Guido> But I'm not sure how Red Hat shipping 1.5.2 proves this point; > Guido> there were no micro releases of 1.5 after that. > > True, but it demonstrates how much agony changes to the language - > regardless of the versioning - can cause. > > Skip Then that's an argument against changes to the language in general. We've had that argument, and the resulting policy is specified in PEPs 4 and 5 (amongst others). --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Mon Sep 8 11:08:30 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Sep 8 11:08:35 2003 Subject: [Python-Dev] New functionality in micro releases (was: Documenting branch policy) In-Reply-To: <200309081419.h88EJmA11895@12-236-84-31.client.attbi.com> References: <20030907161900.GA3195@cthulhu.gerg.ca> <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> <16219.47489.553300.299720@montanaro.dyndns.org> <200309081359.h88Dx2G11829@12-236-84-31.client.attbi.com> <16220.36275.971661.113208@montanaro.dyndns.org> <200309081419.h88EJmA11895@12-236-84-31.client.attbi.com> Message-ID: <16220.39790.575212.507229@montanaro.dyndns.org> >> True, but it demonstrates how much agony changes to the language - >> regardless of the versioning - can cause. Guido> Then that's an argument against changes to the language in Guido> general. We've had that argument, and the resulting policy is Guido> specified in PEPs 4 and 5 (amongst others). I wasn't trying to suggest the language shouldn't evolve, just that ... Oh, never mind. I'm sure you actually know what I meant. S From fdrake at acm.org Mon Sep 8 14:27:52 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon Sep 8 13:28:02 2003 Subject: [Python-Dev] New functionality in micro releases (was: Documenting branch policy) In-Reply-To: References: <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> Message-ID: <16220.48152.685701.574897@grendel.zope.com> Just van Rossum writes: > It's been a huge pain with 2.2 vs. 2.2.1 even: all stuff written for > 2.2.1 that uses booleans does _not_ work on plain 2.2, which is what's No code written for 2.2.1 should be using bool(), True, or False; those were not documented in 2.2.1 *at all* as far as I can tell. The description of bool() was added to the docs in 2.2.3, and I'm not sure why, though it is noted that it was added in 2.2.1. I'd be happy to rip it out of the documentation, or add a stronger compatibility warning, for 2.2.4. > installed on OSX 10.2. Which sortof kills the advantage of having a > Python installed with the OS to begin with. In retrospect, I would have > been strongly against adding booleans to 2.2.1 for this reason. Apart > from real bugs, I think every program that works on 2.x.y should work on > 2.x.z, regardless of whether y > z or y < z. Depending on just what you mean "every program that works", this could completely prevent these bugfix releases. We might not be able to remove a core dump since it would allow code to run that was not run before, thereby changing the behavior of the code. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From python at rcn.com Mon Sep 8 19:29:10 2003 From: python at rcn.com (Raymond Hettinger) Date: Mon Sep 8 18:33:50 2003 Subject: [Python-Dev] New functionality in micro releases (was:Documenting branch policy) References: <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> <16220.48152.685701.574897@grendel.zope.com> Message-ID: <007201c37658$a077a0e0$a511c797@oemcomputer> [Just van Rossum] > > Apart > > from real bugs, I think every program that works on 2.x.y should work on > > 2.x.z, regardless of whether y > z or y < z. [Fred Drake] > Depending on just what you mean "every program that works", this could > completely prevent these bugfix releases. We might not be able to > remove a core dump since it would allow code to run that was not run > before, thereby changing the behavior of the code. JvR did qualify it with: "apart from real bugs". The only question is how "real" does it have to be. I think all of the following should be fair game: * not matching documented behavior (for example: '%F" % x) * usability fixes * performance issues (strptime caching for example) * something that can only be crashed by an obscure case (Armin Rigo's itertools.izip hack) * leaks (the massive leak in the array module) * security fixes * enhancing the docs (chm files, examples, new docs, etc.) * adding capabilities to standalone Tools/scripts or IDLE * updates to separately maintained packages (like "email") so that between major releases, Python won't get terribly far behind the stable releases of the package. I think of micro releases as being the same as service patches which are supposed to be painless upgrades that any admin or user can load without fear of breaking existing code. That should likely be the standard rather than requiring that all programs work irrespective of the micro version. If an application is affected by a bug in an earlier micro version, it isn't terribly unreasonable to ask the users to download the service patch which should be relatively effortless as compared with installing a new major release. Raymond Hettinger From guido at python.org Mon Sep 8 17:13:33 2003 From: guido at python.org (Guido van Rossum) Date: Mon Sep 8 19:13:52 2003 Subject: [Python-Dev] New functionality in micro releases (was:Documenting branch policy) In-Reply-To: Your message of "Mon, 08 Sep 2003 18:29:10 EDT." <007201c37658$a077a0e0$a511c797@oemcomputer> References: <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> <16220.48152.685701.574897@grendel.zope.com> <007201c37658$a077a0e0$a511c797@oemcomputer> Message-ID: <200309082313.h88NDX712738@12-236-84-31.client.attbi.com> > * not matching documented behavior (for example: '%F" % x) Depends on how you fix these. Fixing the code may break more code than fixing the docs (assuming the mismatch existed for a long time)! > * usability fixes What are these? Seems like a loophole the size of a large truck. > * performance issues (strptime caching for example) > * something that can only be crashed by an obscure case > (Armin Rigo's itertools.izip hack) > * leaks (the massive leak in the array module) > * security fixes But every attempt should be made to fix these without changing APIs. > * enhancing the docs (chm files, examples, new docs, etc.) > * adding capabilities to standalone Tools/scripts or IDLE Questionable, if these are installed. Esp. now that IDLE is part of the standard library. > * updates to separately maintained packages (like "email") > so that between major releases, Python won't get terribly > far behind the stable releases of the package. Again, this should be done with the utmost backwards compatibility in mind, and even then, new features are new features, and thus may cause the problem Jack and Just warn about. > I think of micro releases as being the same as service patches which are > supposed to be painless upgrades that any admin or user can load without > fear of breaking existing code. That should likely be the standard rather > than requiring that all programs work irrespective of the micro version. > > If an application is affected by a bug in an earlier micro version, it > isn't terribly unreasonable to ask the users to download the service patch > which should be relatively effortless as compared with installing a new > major release. Right, but keep in mind that few people except hardcore Python users are going to upgrade their Python, no matter how painless, just to run some small piece of software they downloaded. They'll just toss the download as one of so many broken things, proof that free software sucks. --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Mon Sep 8 21:02:13 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Sep 8 20:02:24 2003 Subject: [Python-Dev] New functionality in micro releases (was:Documenting branch policy) In-Reply-To: <200309082313.h88NDX712738@12-236-84-31.client.attbi.com> References: <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> <16220.48152.685701.574897@grendel.zope.com> <007201c37658$a077a0e0$a511c797@oemcomputer> Message-ID: <5.1.1.6.0.20030908195222.01f1a860@telecommunity.com> At 04:13 PM 9/8/03 -0700, Guido van Rossum wrote: >Right, but keep in mind that few people except hardcore Python users >are going to upgrade their Python, no matter how painless, just to run >some small piece of software they downloaded. They'll just toss the >download as one of so many broken things, proof that free software >sucks. I've even seen *developers* do this... Somebody who works in my department here griped about PEAK not working on their machine and when I went to troubleshoot it, I found they had a lesser version (2.2.1) of Python than the docs explicitly required (2.2.2, which fixed a couple of relevant new-style class bugs). (Makes me think I should put some code in to check the Python version and exit with a helpful error message.) It does suggest that being able to specify the version of Python required by a script or module, would be a helpful idiom. sys.requireversion(), perhaps? Guess I should make a quick check to be sure you haven't already used the time machine and put this in... :) Of course, the sad bit is that even if there were a sys.requireversion(), it wouldn't be in the versions where it was actually needed: the older ones! From sholden at holdenweb.com Mon Sep 8 22:06:39 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Sep 9 00:52:25 2003 Subject: [Python-Dev] New functionality in micro releases (was:Documentingbranch policy) In-Reply-To: <200309082313.h88NDX712738@12-236-84-31.client.attbi.com> Message-ID: [Raymond Hettinger] > > * not matching documented behavior (for example: '%F" % x) > [Guido] > Depends on how you fix these. Fixing the code may break more code > than fixing the docs (assuming the mismatch existed for a long time)! > Such as the socket mismatch that was fixed in (IIRC) 1.6, to loud complaints from half the network programming world? This still bites occasionally when old cord has to be brought forward to more recent versions (though, of course, it isn't hard to make the necessary changes). [...] > > * updates to separately maintained packages (like "email") > > so that between major releases, Python won't get terribly > > far behind the stable releases of the package. > > Again, this should be done with the utmost backwards compatibility in > mind, and even then, new features are new features, and thus may cause > the problem Jack and Just warn about. > Yup. Any potential for code breakage is going to cause somebody somewhere a certain amount of pain. It's these edge cases that cause most of the problems, since overall the version-to-version compatibility picture is actually remarkably good. But people will cheerfully ignore the 99.99% of their code that *doesn't* break, and bitch about the little bits that do :-) > > I think of micro releases as being the same as service > patches which are > > supposed to be painless upgrades that any admin or user can > load without > > fear of breaking existing code. That should likely be the > standard rather > > than requiring that all programs work irrespective of the > micro version. > > > > If an application is affected by a bug in an earlier micro > version, it > > isn't terribly unreasonable to ask the users to download > the service patch > > which should be relatively effortless as compared with > installing a new > > major release. > > Right, but keep in mind that few people except hardcore Python users > are going to upgrade their Python, no matter how painless, just to run > some small piece of software they downloaded. They'll just toss the > download as one of so many broken things, proof that free software > sucks. > If you want something to be true sufficiently hard, anything convenient will be seen as supporting evidence. Raymond did, however, pinpoint the reason why I haven't yet started using True and False. what-*me*-conservative-ly y'rs - steve -- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ From eppstein at ics.uci.edu Mon Sep 8 18:48:50 2003 From: eppstein at ics.uci.edu (David Eppstein) Date: Tue Sep 9 01:07:48 2003 Subject: [Python-Dev] Re: New functionality in micro releases (was:Documenting branch policy) References: <007201c37658$a077a0e0$a511c797@oemcomputer> <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> <16220.48152.685701.574897@grendel.zope.com> <007201c37658$a077a0e0$a511c797@oemcomputer> <200309082313.h88NDX712738@12-236-84-31.client.attbi.com> <5.1.1.6.0.20030908195222.01f1a860@telecommunity.com> Message-ID: In article <5.1.1.6.0.20030908195222.01f1a860@telecommunity.com>, "Phillip J. Eby" wrote: > It does suggest that being able to specify the version of Python required > by a script or module, would be a helpful idiom. sys.requireversion(), > perhaps? Guess I should make a quick check to be sure you haven't already > used the time machine and put this in... :) > > Of course, the sad bit is that even if there were a sys.requireversion(), > it wouldn't be in the versions where it was actually needed: the older ones! Ok, so what's needed isn't a new sys.feature, it's some short boilerplate code you could include in all your python apps, to make sure anyone who tries to run it on an old system is informed of the problem. Something like: import sys class PythonTooOld(Exception): pass if sys.version < '2.2': raise PythonTooOld, 'This code needs Python 2.2 or greater.' -- David Eppstein http://www.ics.uci.edu/~eppstein/ Univ. of California, Irvine, School of Information & Computer Science From aleaxit at yahoo.com Tue Sep 9 08:20:37 2003 From: aleaxit at yahoo.com (Alex Martelli) Date: Tue Sep 9 01:20:43 2003 Subject: [Python-Dev] New functionality in micro releases (was:Documenting branch policy) In-Reply-To: <5.1.1.6.0.20030908195222.01f1a860@telecommunity.com> References: <007201c37658$a077a0e0$a511c797@oemcomputer> <5.1.1.6.0.20030908195222.01f1a860@telecommunity.com> Message-ID: <200309090720.37252.aleaxit@yahoo.com> On Tuesday 09 September 2003 02:02 am, Phillip J. Eby wrote: ... > Of course, the sad bit is that even if there were a sys.requireversion(), > it wouldn't be in the versions where it was actually needed: the older > ones! Actually this would "fail-RIGHT": on older versions the program would die right there with an AttributeException and a traceback about a line such as sys.requireversion('2.3.2'), which is already a clear enough indication of exactly what is wrong -- on newer ones with an even clearer exception, of course. In any case, one could dress up the sys.requireversion call in a try/except to make its absence no problem. Alex From fdrake at acm.org Mon Sep 8 23:37:59 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue Sep 9 01:49:37 2003 Subject: [Python-Dev] New functionality in micro releases (was:Documenting branch policy) In-Reply-To: <5.1.1.6.0.20030908195222.01f1a860@telecommunity.com> References: <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> <16220.48152.685701.574897@grendel.zope.com> <007201c37658$a077a0e0$a511c797@oemcomputer> <5.1.1.6.0.20030908195222.01f1a860@telecommunity.com> Message-ID: <16221.15623.111594.33933@grendel.zope.com> Phillip J. Eby writes: > It does suggest that being able to specify the version of Python required > by a script or module, would be a helpful idiom. sys.requireversion(), > perhaps? Guess I should make a quick check to be sure you haven't already > used the time machine and put this in... :) I'm not aware of any such thing. > Of course, the sad bit is that even if there were a sys.requireversion(), > it wouldn't be in the versions where it was actually needed: the older ones! Comparison with sys.version_info would be trivial, and seems sufficient: if sys.version_info < (2,2,2): print "Get a newer version of Python!" else: main() -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From mfb at lotusland.dyndns.org Tue Sep 9 02:25:11 2003 From: mfb at lotusland.dyndns.org (Matthew F. Barnes) Date: Tue Sep 9 02:38:11 2003 Subject: [Python-Dev] New functionality in micro releases(was:Documenting branch policy) In-Reply-To: <16221.15623.111594.33933@grendel.zope.com> References: <5FE933B2-E171-11D7-A B7D-000A27B19B96@cwi.nl><16220.48152.685701.574897@grendel.zope.com><007201c376 58$a077a0e0$a511c797@oemcomputer><5.1.1.6.0.20030908195222.01f1a860@te lecommunity.com> <16221.15623.111594.33933@grendel.zope.com> Message-ID: <4269.192.168.1.1.1063088711.squirrel@server.lotusland.dyndns.org> Fred L. Drake, Jr. said: > Comparison with sys.version_info would be trivial, and seems > sufficient: > > if sys.version_info < (2,2,2): > print "Get a newer version of Python!" > else: > main() FWIW - I often use something close to this near the top of my own modules (realizing that it gets skipped when the -O option is provided): assert sys.version_info >= (2,2,2), 'requires Python 2.2.2 or better' Matthew Barnes From Jack.Jansen at cwi.nl Tue Sep 9 12:26:54 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Tue Sep 9 05:27:01 2003 Subject: [Python-Dev] Documenting branch policy In-Reply-To: <1063027104.16344.9.camel@anthem> References: <20030907161900.GA3195@cthulhu.gerg.ca> <1063027104.16344.9.camel@anthem> Message-ID: On 8-sep-03, at 15:18, Barry Warsaw wrote: >>

Every release up to and including the final release for a new major >> Python version is accompanied by a release branches. > > There's even some debate about these (I've caught up with the thread, > so > I know you've withdrawn this change). I've been backing off the use of > release branches because they create more complexity in a world where > none of us have much time to deal with it. They served me well for the MacPython-OS9 releases. But as 2.3 is going to be the last of those anyway I could live without them, I guess. Although: they'd still be useful in case of unforeseen problems with a distribution that are not code-related. Think of things like the Windows installer turning out to be broken on certain machine, so you want to build the installer again (but not Python itself). -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From barry at python.org Tue Sep 9 13:09:36 2003 From: barry at python.org (Barry Warsaw) Date: Tue Sep 9 08:09:37 2003 Subject: [Python-Dev] New functionality in micro releases (was:Documenting branch policy) In-Reply-To: <007201c37658$a077a0e0$a511c797@oemcomputer> References: <5FE933B2-E171-11D7-AB7D-000A27B19B96@cwi.nl> <16220.48152.685701.574897@grendel.zope.com> <007201c37658$a077a0e0$a511c797@oemcomputer> Message-ID: <1063109371.16781.21.camel@anthem> On Mon, 2003-09-08 at 18:29, Raymond Hettinger wrote: > * updates to separately maintained packages (like "email") > so that between major releases, Python won't get terribly > far behind the stable releases of the package. You have to take things on a case-by-case basis. I actually agonized quite a bit about the email package in 2.2, but after discussing things with Guido and the mimelib community, there was general consensus that not fixing/upgrading the package was worse than doing so. But I doubt we'll do the same thing for Python 2.3, because the package is more stable now, has been integrated longer, and the planned changes are probably more radical. This does point out a problem with the batteries-included philosophy though: it's hard to upgrade packages that come with Python. Say we release email 3.0 and you want to use it in Python 2.3, what do you do? You can't just do a distutils install, because site-packages is searched after the standard library. I think we need more flexibility in installing distutils packages, so we can install in various locations that override standard packages (e.g. user-centric, system-centric, etc.). In general though, it's a good thing to default to a policy where micro releases are bug fixes only. Given the reality of the resources available, time between major releases, and the fact that Python development is a wholly volunteer effort, some amount of ugliness is bound to invade the process. That's where the BDFL uses his judgement to make a decision and we live with it. Take the True/False thing. OOH, it was a new feature added to a micro release. OTOH it was backward compatible, helped future compatibility, and was easily coded around to avoid micro-version skew. Taken as a whole, and given when the decision was made, it was probably a good one although it caused a little bit of pain. -Barry From barry at python.org Tue Sep 9 13:14:11 2003 From: barry at python.org (Barry Warsaw) Date: Tue Sep 9 08:14:12 2003 Subject: [Python-Dev] Documenting branch policy In-Reply-To: References: <20030907161900.GA3195@cthulhu.gerg.ca> <1063027104.16344.9.camel@anthem> Message-ID: <1063109647.16781.24.camel@anthem> On Tue, 2003-09-09 at 05:26, Jack Jansen wrote: > On 8-sep-03, at 15:18, Barry Warsaw wrote: > > >>

Every release up to and including the final release for a new major > >> Python version is accompanied by a release branches. > > > > There's even some debate about these (I've caught up with the thread, > > so > > I know you've withdrawn this change). I've been backing off the use of > > release branches because they create more complexity in a world where > > none of us have much time to deal with it. > > They served me well for the MacPython-OS9 releases. But as 2.3 is going > to > be the last of those anyway I could live without them, I guess. > > Although: they'd still be useful in case of unforeseen problems with a > distribution that are not code-related. Think of things like the Windows > installer turning out to be broken on certain machine, so you want to > build > the installer again (but not Python itself). That's a good point. Release branches are probably overkill for alpha/beta/rc releases, but probably make sense for final releases. We actually did create one for 2.3, but I screwed up when I named it and that caused a tiny bit of pain in moving to the maintenance branch. OTOH, for a final release, maybe release branch == maintenance branch. -Barry From fdrake at acm.org Tue Sep 9 09:16:47 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue Sep 9 08:16:53 2003 Subject: [Pydotorg] Re: [Python-Dev] Documenting branch policy In-Reply-To: <1063109647.16781.24.camel@anthem> References: <20030907161900.GA3195@cthulhu.gerg.ca> <1063027104.16344.9.camel@anthem> <1063109647.16781.24.camel@anthem> Message-ID: <16221.50351.725375.52522@grendel.zope.com> Barry Warsaw writes: > That's a good point. Release branches are probably overkill for > alpha/beta/rc releases, but probably make sense for final releases. We > actually did create one for 2.3, but I screwed up when I named it and > that caused a tiny bit of pain in moving to the maintenance branch. > OTOH, for a final release, maybe release branch == maintenance branch. I think that's sufficient. There's no need for them to be separate. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From gward at python.net Tue Sep 9 09:24:47 2003 From: gward at python.net (Greg Ward) Date: Tue Sep 9 08:24:53 2003 Subject: [Python-Dev] Documenting branch policy In-Reply-To: <1063109647.16781.24.camel@anthem> References: <20030907161900.GA3195@cthulhu.gerg.ca> <1063027104.16344.9.camel@anthem> <1063109647.16781.24.camel@anthem> Message-ID: <20030909122447.GB15042@cthulhu.gerg.ca> On 09 September 2003, Barry Warsaw said: > That's a good point. Release branches are probably overkill for > alpha/beta/rc releases, but probably make sense for final releases. We > actually did create one for 2.3, but I screwed up when I named it and > that caused a tiny bit of pain in moving to the maintenance branch. > OTOH, for a final release, maybe release branch == maintenance branch. Well, if nothing else this thread has demonstrated that I'm *not* the only one who's a little confused by the release/maintenance branching policy. *phew*! Greg -- Greg Ward http://www.gerg.ca/ Just because you're paranoid doesn't mean they *aren't* out to get you. From anthony at interlink.com.au Wed Sep 10 02:18:30 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Sep 9 11:18:52 2003 Subject: [Python-Dev] New functionality in micro releases (was:Documenting branch policy) In-Reply-To: <1063109371.16781.21.camel@anthem> Message-ID: <200309091518.h89FIURh024097@localhost.localdomain> >>> Barry Warsaw wrote > This does point out a problem with the batteries-included philosophy > though: it's hard to upgrade packages that come with Python. Say we > release email 3.0 and you want to use it in Python 2.3, what do you do? > You can't just do a distutils install, because site-packages is searched > after the standard library. I think we need more flexibility in > installing distutils packages, so we can install in various locations > that override standard packages (e.g. user-centric, system-centric, > etc.). The approach taken by the XML add-on package doesn't seem to be the right approach, either. At the moment your choices are to manually install the new code over the old, or else have a separate install area that's manually added to the start of the PYTHONPATH (this is what mailman does, right?) Neither of these approaches seem ideal. The latter is what I do for various systems that need newer versions of packages that are bundled with python, it'd be nice if there was a better way - but I can't see one. Anthony From tim.one at comcast.net Tue Sep 9 12:25:50 2003 From: tim.one at comcast.net (Tim Peters) Date: Tue Sep 9 11:25:48 2003 Subject: [Python-Dev] New functionality in micro releases (was:Documentingbranch policy) In-Reply-To: <200309091518.h89FIURh024097@localhost.localdomain> Message-ID: [Barry Warsaw] > ... > This does point out a problem with the batteries-included philosophy I was still basically asleep when I saw this msg, and my bleary eyes first read that as "barriers-included". > though: it's hard to upgrade packages that come with Python. Indeed. Batteries, barriers, same thing . From barry at python.org Tue Sep 9 16:50:15 2003 From: barry at python.org (Barry Warsaw) Date: Tue Sep 9 11:50:16 2003 Subject: [Python-Dev] New functionality in micro releases (was:Documenting branch policy) In-Reply-To: <200309091518.h89FIURh024097@localhost.localdomain> References: <200309091518.h89FIURh024097@localhost.localdomain> Message-ID: <1063122609.16781.48.camel@anthem> On Tue, 2003-09-09 at 11:18, Anthony Baxter wrote: > The approach taken by the XML add-on package doesn't seem to be the > right approach, either. At the moment your choices are to manually install > the new code over the old, or else have a separate install area that's > manually added to the start of the PYTHONPATH (this is what mailman > does, right?) Right, and it sucks because there are multiple top-level entry points into the system (e.g. command line and cron scripts) and each has to mangle the path very early on. Then there's the pain of not being able to run for the source directory because the configure/install procedure sets up the path hacking module with the proper paths. Blech. -Barry From guido at python.org Tue Sep 9 10:17:15 2003 From: guido at python.org (Guido van Rossum) Date: Tue Sep 9 12:17:37 2003 Subject: [Python-Dev] New functionality in micro releases (was:Documentingbranch policy) In-Reply-To: Your message of "Mon, 08 Sep 2003 21:06:39 EDT." References: Message-ID: <200309091617.h89GHFb13948@12-236-84-31.client.attbi.com> > [Guido] > > Depends on how you fix these. Fixing the code may break more code > > than fixing the docs (assuming the mismatch existed for a long time)! [Steve] > Such as the socket mismatch that was fixed in (IIRC) 1.6, to loud > complaints from half the network programming world? This still bites > occasionally when old cord has to be brought forward to more recent > versions (though, of course, it isn't hard to make the necessary > changes). That was not a micro release. I'm glad we fixed that one then; now it would be a much bigger disaster. (And it was not a matter of adjusting to the docs, but adjusting to the fact that not all socets deal with host/port pairs.) > [...] > Yup. Any potential for code breakage is going to cause somebody > somewhere a certain amount of pain. It's these edge cases that cause > most of the problems, since overall the version-to-version compatibility > picture is actually remarkably good. But people will cheerfully ignore > the 99.99% of their code that *doesn't* break, and bitch about the > little bits that do :-) For people with a large code base and real customers, the only working approach is long, careful testing with each new Python release. Hence the call for Python-in-a-tie. (Though I haven't heard much about that recently -- I've even heard a call to upgrade the tie to 2.3.) --Guido van Rossum (home page: http://www.python.org/~guido/) From pinki_mallick at mentorg.com Tue Sep 9 12:25:18 2003 From: pinki_mallick at mentorg.com (Mallick, Pinki) Date: Tue Sep 9 14:25:24 2003 Subject: [Python-Dev] memory related issues in Python - bug?? Message-ID: Hi, I have a very big XML file that I am trying to parse using xml.dom and xml.dom.minidom. This file has around 100 nodes. Each of these nodes(say 'child_X') have a number of child nodes and each of these child nodes in turn has a text node as its child. So when I try to retrieve all the data from each 'child_X' node, I use this code, grandChildren = child_X.childNodes() for grandChildNode in grandChildren: equation = equation + str(grandChildNode.firstChild.data) This works properly for all files. But when I am trying to use it on a very large XML file, one of the node's data is incomplete. eg. if grandChildNode.toxml() returns "y18" for this node, then "str(grandChildNode.firstChild.data)" returns only "y" instead of "y18". This looks like some memory problem. Is there any workaround for this problem. Thanks for any suggestion or help, Pinki Mallick From fdrake at acm.org Tue Sep 9 15:31:41 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue Sep 9 14:31:50 2003 Subject: [Python-Dev] memory related issues in Python - bug?? In-Reply-To: References: Message-ID: <16222.7309.203774.926825@grendel.zope.com> Mallick, Pinki writes: > I have a very big XML file that I am trying to parse using xml.dom > and xml.dom.minidom. This file has around 100 nodes. Each of these ... This message was also posted to the XML-SIG (where it belongs) and the webmaster address for python.org (where it doesn't). Please don't spam several lists with a question like this. I've responded to the question in the appropriate forum. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From aahz at pythoncraft.com Tue Sep 9 16:16:31 2003 From: aahz at pythoncraft.com (Aahz) Date: Tue Sep 9 15:16:35 2003 Subject: [Python-Dev] memory related issues in Python - bug?? In-Reply-To: References: Message-ID: <20030909191631.GA1022@panix.com> On Tue, Sep 09, 2003, Mallick, Pinki wrote: > > I have a very big XML file that I am trying to parse using xml.dom > and xml.dom.minidom. You've sent this question to two inappropriate places: webmaster@python.org and python-dev@python.org Please use the Usenet group comp.lang.python (or subscribe to python-list) and post your question there. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "It is easier to optimize correct code than to correct optimized code." --Bill Harlan From martin at v.loewis.de Tue Sep 9 21:30:51 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Sep 9 16:30:52 2003 Subject: [Python-Dev] New functionality in micro releases (was:Documenting branch policy) In-Reply-To: <200309091518.h89FIURh024097@localhost.localdomain> References: <200309091518.h89FIURh024097@localhost.localdomain> Message-ID: Anthony Baxter writes: > The approach taken by the XML add-on package doesn't seem to be the > right approach, either. At the moment your choices are to manually install > the new code over the old, or else have a separate install area that's > manually added to the start of the PYTHONPATH (this is what mailman > does, right?) Are you referring to PyXML here? It allows to extend/replace the existing library with neither copying the new code over the old nor with adding anything to PYTHONPATH. Instead, the core library expects to be replaced, and is actively looking for a replacement. Regards, Martin From guido at python.org Tue Sep 9 22:32:51 2003 From: guido at python.org (Guido van Rossum) Date: Wed Sep 10 00:33:13 2003 Subject: [Python-Dev] Quick: want to give a Python talk at Linuxworld NY? Message-ID: <200309100432.h8A4Wps15627@12-236-84-31.client.attbi.com> I was just approached by Chris DiBona (chris at damagestudios dot com) about talking at the next Linuxworld in New York, in January. There are plenty of PHP and Perl talks but no Python talk yet. Time is of the essence, they are finalizing the program *now*. Reward is a free pass to everything. Spread the word! --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Sep 10 00:15:37 2003 From: guido at python.org (Guido van Rossum) Date: Wed Sep 10 02:16:00 2003 Subject: [Python-Dev] Consistent logging in the standard library In-Reply-To: Your message of "Sun, 07 Sep 2003 22:13:02 CDT." <4101.192.168.1.1.1062990782.squirrel@server.lotusland.dyndns.org> References: <4101.192.168.1.1.1062990782.squirrel@server.lotusland.dyndns.org> Message-ID: <200309100615.h8A6FbM15765@12-236-84-31.client.attbi.com> > The method for doing so is modelled after the socket module's new timeout > functionality [3], where each module that performs logging defines a pair > of module-level functions named getdefaultlogger() and setdefaultlogger() > that return and accept (respectively) a logging.Logger instance. The > initial default logger for each of these modules is named after the module > itself (i.e. logging.getLogger(__name__)) and its behavior mimics that of > the Python 2.3 version of the module in terms of format and destination of > the logging messages. Hm... Isn't the logging module's ability to set separate *handlers* sufficient to get the desired flexibility? Rather than the application specifying a different *logger* object, the application can specify a different *handler* object. I think this was the originally intended design. It places less of a burden on the modules that use logging (no need to define getdefaultlogger() and setdefaultlogger()) and makes it possible for the different handlers to be specified by the logging module's configuration file, without the application having to support this. Then what remains is a convention for the name of the logger used by standard library modules; perhaps the module name should suffice? > So an application that wants to override the logging behavior of a > standard library module can do so simply by passing a customized logger > object to the module's setdefaultlogger() function. This works out even > better for modules where all the logging is encapsulated in a class. > > The PEP calls for each class that performs logging to define an attribute > called self.logger, and upon instantiation initialize self.logger to the > module's *current* default logger (again, modelled after how the timeout > value for a socket object is initialized). This allows each instance of a > class to have its own unique logging behavior. What behavior exactly are you thinking of? A logger object has almost no semantics apart from passing log events on to handlers; by design, all the work is done in the handlers. > The first item for discussion is whether this seems like a reasonable > approach. The second is identifying which standard library modules are > candidates for modification using this approach. I've identified the > following by simply searching for the word "logging" in the source code, > but I'm not sure whether this list is complete: > > asyncore > BaseHTTPServer > cgi > doctest > imaplib > unittest > the distutils package > > I have patches ready for asyncore and BaseHTTPServer and doctest, which > fit the model I described above quite nicely. I think it would be useful to let these use the logging module even without the proposed setdefaultlogger() etc. architecture. > I determined that no changes are necessary for the cgi module, since > it doesn't even use the logging functions it defines (initlog, > dolog, etc.). They're intended for use by CGI applications. They should probably be rewritten to direct the calls to the logging module. > I'm currently working on a way to replicate imaplib's command > history using LogRecords, which may call for a new type of > logging.BufferingHandler subclass that maintains a circular queue of > LogRecords. That might be a useful feature on its own for the logging package. Note though, that at a grander scale, the RotatingFileLogger class already supports a way to keep only the last N messages (for much larger N, though :-). > And the distutils package seems prepped for using the logging > package already. > > The oddball is unittest. Unittest is what motivated me to bring up the > logging issue in the first place, but I can't (or more precisely - am not > in a position to) decide what direction that module should take w.r.t. if > and how it should use the logging package. Should the TextTestRunner > class be modified to use a logging.Logger instead of a _WritelnDecorator, > or should a separate TestRunner subclass (a LoggingTestRunner perhaps) be > defined instead? I'm hoping that Steve Purcell or others may have some > thoughts on this. To reach Steve, you should probably post to the PyUnit list (sorry, you'll have to Google for details, I only vaguely recall that there is such a list). Personally, I think that unittest is such a special case that it might be best to leave it alone... > I would also appreciate hearing any alternatives to the approach I've > suggested, as well as comments on the strengths and weaknesses of each. --Guido van Rossum (home page: http://www.python.org/~guido/) From vilmanis at internode.on.net Wed Sep 10 23:09:29 2003 From: vilmanis at internode.on.net (Yuri Vilmanis) Date: Wed Sep 10 08:40:37 2003 Subject: [Python-Dev] Base-n integer formatting Message-ID: <200309102209.29603.vilmanis@internode.on.net> I have three feature requests for you: no. 1 is quite fundamental, no. 2 would be a 'nice' useful feature and no. 3 is a bit more esoteric, but very useful in certain applications 1) print formatting for integers is currently only available for octal, hex, and decimal. This seems rather odd, as string to integer conversion is available for bases 2 through 36. End result: when I want to do things in base 6 (rare, but it has happened twice) or much more commonly in binary, I can read the numbers in, but cannot print the results - ie quite useless. 2) Input and output formatting for unary (which is commonly referred to as base 1, yes, I know it's not 'really' base 1 by the general deinition of base, but "base 1" is the convention) would also be good, as would roman numeral support, especially in word-processing applications or IO from any formatted document with roman page/section numbers 3) negative bases: I have needed this more times than I think some people would care to imagine. These representations have several very useful properties such as implementation-independent signdness (hence arbirtrary word length), and some more interesting properties I won't expand on here. Context: I find python very handy not only as a programming/scripting language but also as a scratchpad for various number theory investigations. Unfortunately the output formatting leaves all but the most basic investigations out in the cold, and lack of negative base input/output makes it harder to use python to look for certain patterns in result sets, not to mention getting the data in... Thanks, Yuri From mchermside at ingdirect.com Wed Sep 10 10:30:08 2003 From: mchermside at ingdirect.com (Chermside, Michael) Date: Wed Sep 10 09:30:19 2003 Subject: [Python-Dev] Base-n integer formatting Message-ID: <7F171EB5E155544CAC4035F0182093F042126F@INGDEXCHSANC1.ingdirect.com> Yuri: I think you have some interesting ideas, but this is really not the best place to present them. The newsgroup comp.lang.python, also a mailing list at python-list@python.org, is really the best place to discuss these ideas. Python-dev is intended for discussion about the development of Python. New ideas or language features are best introduced first on c.l.py, then based on the feedback there, written up as a PEP, *THEN* folks will start discussing it on python-dev as they figure out HOW (and perhaps whether) to implement it. There's an exception -- regular Python developers sometimes introduce ideas on this list, but that's a very short list of people, and it's a bit different, because if they bring it up it's because they're already volunteering to implement it (and probably have a prototype up and running already). If you'll contact me _separately_, I'll be glad to give you my thoughts on your suggestions. -- Michael Chermside -----Original Message----- From: Yuri Vilmanis [mailto:vilmanis@internode.on.net] Sent: Wednesday, September 10, 2003 8:39 AM To: python-dev@python.org Subject: [Python-Dev] Base-n integer formatting I have three feature requests for you: no. 1 is quite fundamental, no. 2 would be a 'nice' useful feature and no. 3 is a bit more esoteric, but very useful in certain applications 1) print formatting for integers is currently only available for octal, hex, and decimal. This seems rather odd, as string to integer conversion is available for bases 2 through 36. End result: when I want to do things in base 6 (rare, but it has happened twice) or much more commonly in binary, I can read the numbers in, but cannot print the results - ie quite useless. 2) Input and output formatting for unary (which is commonly referred to as base 1, yes, I know it's not 'really' base 1 by the general deinition of base, but "base 1" is the convention) would also be good, as would roman numeral support, especially in word-processing applications or IO from any formatted document with roman page/section numbers 3) negative bases: I have needed this more times than I think some people would care to imagine. These representations have several very useful properties such as implementation-independent signdness (hence arbirtrary word length), and some more interesting properties I won't expand on here. Context: I find python very handy not only as a programming/scripting language but also as a scratchpad for various number theory investigations. Unfortunately the output formatting leaves all but the most basic investigations out in the cold, and lack of negative base input/output makes it harder to use python to look for certain patterns in result sets, not to mention getting the data in... Thanks, Yuri _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev This email may contain confidential or privileged information. If you believe you have received the message in error, please notify the sender and delete the message without copying or disclosing it. From mfb at lotusland.dyndns.org Wed Sep 10 09:29:46 2003 From: mfb at lotusland.dyndns.org (Matthew F. Barnes) Date: Wed Sep 10 09:43:11 2003 Subject: [Python-Dev] Consistent logging in the standard library In-Reply-To: <200309100615.h8A6FbM15765@12-236-84-31.client.attbi.com> References: <4101.192.168.1.1.1062990782.squirrel@server.lotusland.dyndns.org> <200309100615.h8A6FbM15765@12-236-84-31.client.attbi.com> Message-ID: <3412.192.168.1.1.1063200586.squirrel@server.lotusland.dyndns.org> Guido van Rossum said: > Hm... Isn't the logging module's ability to set separate *handlers* > sufficient to get the desired flexibility? Rather than the > application specifying a different *logger* object, the application > can specify a different *handler* object. I think this was the > originally intended design. It places less of a burden on the modules > that use logging (no need to define getdefaultlogger() and > setdefaultlogger()) and makes it possible for the different handlers > to be specified by the logging module's configuration file, without > the application having to support this. I was thinking of the logging hierarchy when I came up with this idea. If a module references a specific logger object explicitly when a logging message needs to be issued (e.g. logging.getLogger(__name__).info(msg)), then the module's logging output is forever handled at a fixed place in the hierarchy. An application may want to organize things differently. For example, it may want to redirect asyncore's log messages to a logger named "myapp.network.asyncore" so that its messages are propagated to the "myapp.network" and "myapp" loggers defined by the application. Those loggers could apply additional handlers or filters to asyncore's log messages. In such a case the application could set this up as follows: asyncore.setdefaultlogger(logging.getLogger('myapp.network.asyncore')) > What behavior exactly are you thinking of? A logger object has almost > no semantics apart from passing log events on to handlers; by design, > all the work is done in the handlers. I used the word "behavior" to refer to the series of handlers and filters that are applied to a log message. Different instances of the same class could log to different logger objects, and thereby have different "logging behaviors". Unfortunately none of the modules I've identified in the stdlib seem to provide a motivating example of why you'd ever want to do this. I can try to elaborate further if the benefit is not clear. >> I determined that no changes are necessary for the cgi module, since >> it doesn't even use the logging functions it defines (initlog, >> dolog, etc.). > > They're intended for use by CGI applications. They should probably be > rewritten to direct the calls to the logging module. I don't have much experience with this module so correct me if I'm wrong, but to my knowledge the cgi module does not call its own logging functions. This implies to me that it's essentially up to the CGI application to perform its own logging. If that's true then I wonder if CGI applications would be better off just be using the logging package directly, rather than routing log messages through the cgi module. The existing log functions in cgi should probably remain for backwards compatibility, but I just have to wonder if it's really worth updating them. >> I'm currently working on a way to replicate imaplib's command >> history using LogRecords, which may call for a new type of >> logging.BufferingHandler subclass that maintains a circular queue of >> LogRecords. > > That might be a useful feature on its own for the logging package. > Note though, that at a grander scale, the RotatingFileLogger class > already supports a way to keep only the last N messages (for much > larger N, though :-). Thanks, I'll look into that. > To reach Steve, you should probably post to the PyUnit list (sorry, > you'll have to Google for details, I only vaguely recall that there is > such a list). If Pyunit-interest is what you mean, the last time I checked it didn't seem to be very active anymore and was filled with mostly spam. I'll try to reach Steve directly. > Personally, I think that unittest is such a special case that it might > be best to leave it alone... I think I agree with you. Unless Steve has any ideas, I'm going to drop it from the forthcoming PEP. From guido at python.org Wed Sep 10 08:18:00 2003 From: guido at python.org (Guido van Rossum) Date: Wed Sep 10 10:18:26 2003 Subject: [Python-Dev] Base-n integer formatting In-Reply-To: Your message of "Wed, 10 Sep 2003 22:09:29 +0930." <200309102209.29603.vilmanis@internode.on.net> References: <200309102209.29603.vilmanis@internode.on.net> Message-ID: <200309101418.h8AEI0f16363@12-236-84-31.client.attbi.com> Yuri, Those conversions are so easily coded in Python itself that I see no reason to build them into the language or standard library. --Guido van Rossum (home page: http://www.python.org/~guido/) From Paul.Moore at atosorigin.com Wed Sep 10 17:49:21 2003 From: Paul.Moore at atosorigin.com (Moore, Paul) Date: Wed Sep 10 11:50:09 2003 Subject: [Python-Dev] RE: [python-win32] pb with string conversion to float and GetObject call Message-ID: <16E1010E4581B049ABC51D4975CEDB8803060C76@UKDCX001.uk.int.atosorigin.com> From: popov [mailto:outlook@evpopov.com] > Here's a strange one: > > print float('8.4') > > Works as expected. > But: > > import win32com.client > obj = win32com.client.GetObject ("winmgmts:") > print float('8.4') > > Throws a 'ValueError: invalid literal for float(): 8.4' exception on the > print line ! > If you use '8,4' instead of '8.4', then it works (!). So it seems that some > locals have been modified here by the call to GetObject (without this call - > leaving only the import statement - the problem does not show up). > Note that using 'LDAP:' instead of 'winmgmts:', everything is ok. So maybe > the culprit is the winmgmts provider... > If someone could test this and let me know the results. Thx. There's a discussion going on on python-dev at the moment (the messages mention LC_NUMERIC in the subject) on basically this issue. Python's float conversion routines rely on the underlying C locale remaining unchanged. It's known that 3rd party extensions can violate this, and also that Microsoft MAPI mucks up the locale behind the scenes. From what you are reporting, it looks like the WMI stuff also does this. I don't know of any fix for sure - you might be able to use the functions from the locale module to set the locale value back to "C". Assuming that an acceptable fix is found, something should be in Python 2.4 (I don't know if the fix will be deemed appropriate for a maintenance release of 2.3). Hope this helps, Paul. PS I'm copying this to python-dev, as evidence of another area where this issue comes up - I hope that's acceptable. From mcherm at mcherm.com Wed Sep 10 09:58:25 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Wed Sep 10 11:58:26 2003 Subject: [Python-Dev] Consistent logging in the standard library Message-ID: <1063209505.3f5f4a21b564e@mcherm.com> Quoth Matthew F. Barnes: > An application may want to organize things differently. For example, it > may want to redirect asyncore's log messages to a logger named > "myapp.network.asyncore" Seems like an uncommon request, but one which shouldn't be made impossible for those who want it. But couldn't it be achieved by writing a handler which simply re-logged the message at the new spot in the hierarchy? From fdrake at acm.org Wed Sep 10 13:04:07 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed Sep 10 12:04:18 2003 Subject: [Python-Dev] Consistent logging in the standard library In-Reply-To: <1063209505.3f5f4a21b564e@mcherm.com> References: <1063209505.3f5f4a21b564e@mcherm.com> Message-ID: <16223.19319.350549.596704@grendel.zope.com> Michael Chermside writes: > Seems like an uncommon request, but one which shouldn't be made > impossible for those who want it. But couldn't it be achieved by > writing a handler which simply re-logged the message at the new > spot in the hierarchy? That would solve that use case handily; if that's the right approach, such a handler should be provided as part of the standard set of handlers. The use case for a .setlogger() method on the various objects that perform logging is different, so that is still needed. Generally, I'm not convinced that a getdefaultlogger() is needed or helpful (it seems an implementation default for the module), though I'm not at all opposed to the setdefaultlogger() for the affected modules. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From kiko at async.com.br Wed Sep 10 14:08:59 2003 From: kiko at async.com.br (Christian Reis) Date: Wed Sep 10 12:10:25 2003 Subject: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: References: Message-ID: <20030910160858.GE2405@async.com.br> On Mon, Sep 01, 2003 at 02:30:23PM -0400, Tim Peters wrote: > There's no way of using C's locale gimmicks that's threadsafe, short of all > callers agreeing to follow a beyond-standard-C exclusion protocol -- which > is the same as saying "no way" in reality. So that's part of one problem no > patch of this ilk *can* solve. It's not that the patch doesn't try hard > enough, it's that this approach is inherently inadequate to solve all of > this particular problem. > > > It is just that the patch does not "feel" right, given that there must > > be "native" locale-inaware parsing of floating point constants > > somewhere on each platform (atleast on those that support C++98). > > I haven't found one on Windows (doesn't mean it doesn't exist, does mean > it's apparently well hidden if it does exist). Just to follow up, today I found a thread on opengroup.org that discusses locale-safe APIs in the C library. They don't suggest anything very positive in the way of standardization :-/ http://www.opengroup.org/austin/mailarchives/austin-group-l/msg00763.html Take care, -- Christian Reis, Senior Engineer, Async Open Source, Brazil. http://async.com.br/~kiko/ | [+55 16] 261 2331 | NMFL From Mariselaschwebach at pcvo.com Wed Sep 10 22:32:24 2003 From: Mariselaschwebach at pcvo.com (Women Helpline) Date: Wed Sep 10 15:37:26 2003 Subject: [Python-Dev] Luscious, Seductive L|ps can be yours Message-ID: <551x9i9qj5mae9$7u-6@rh1.c.xp.d6> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20030910/b1cc80f2/attachment-0001.htm From mfb at lotusland.dyndns.org Wed Sep 10 17:21:47 2003 From: mfb at lotusland.dyndns.org (Matthew F. Barnes) Date: Wed Sep 10 17:35:06 2003 Subject: [Python-Dev] Consistent logging in the standard library In-Reply-To: <16223.19319.350549.596704@grendel.zope.com> References: <1063209505.3f5f4a21b564e@mcherm.com> <16223.19319.350549.596704@grendel.zope.com> Message-ID: <30174.130.76.32.145.1063228907.squirrel@server.lotusland.dyndns.org> Fred L. Drake, Jr. said: > > Michael Chermside writes: > > Seems like an uncommon request, but one which shouldn't be made > > impossible for those who want it. But couldn't it be achieved by > > writing a handler which simply re-logged the message at the new > > spot in the hierarchy? > > That would solve that use case handily; if that's the right approach, > such a handler should be provided as part of the standard set of > handlers. I agree, this seems much cleaner than what I've proposed. In fact, the "ForwardingHandler" idea could be submitted to SourceForge as a separate patch for logging, and then the issue drops out of this "consistent logging" PEP altogether. So that changes my proposal as follows: - The getdefaultlogger() and setdefaultlogger() functions go away. - The self.logger attribute of class instances can be initialized to logging.getLogger(__name__). - Some kind of "ForwardingHandler" can be added to the module's logger to forward messages elsewhere in the logging hierarchy. But this is not likely to be a common use case. Matthew Barnes From fuerte at sci.fi Thu Sep 11 20:23:05 2003 From: fuerte at sci.fi (Harri Pesonen) Date: Thu Sep 11 12:23:02 2003 Subject: [Python-Dev] Making python C-api thread safe Message-ID: <3F60A169.3070409@sci.fi> Background: I'm new to Python. I just embedded Python into a multi-threaded application, and extended Python by callbacks as well. During this process I realised that Python is not internally as beautiful as externally. It is not thread safe. This surprised me, because Python is a quite young programming language. I started thinking how to fix this, how to make Python thread safe. Currently Python does not support multiprocessor machines or hyperthreading processors. It is only a matter of time when this has to be fixed, better soon than later. How to make C-api thread safe: It's quite simple, in fact. Thread should first call From aahz at pythoncraft.com Thu Sep 11 13:44:22 2003 From: aahz at pythoncraft.com (Aahz) Date: Thu Sep 11 12:44:27 2003 Subject: [Python-Dev] Making python C-api thread safe In-Reply-To: <3F60A169.3070409@sci.fi> References: <3F60A169.3070409@sci.fi> Message-ID: <20030911164421.GA897@panix.com> On Thu, Sep 11, 2003, Harri Pesonen wrote: > > Background: I'm new to Python. This issue gets discussed regularly on comp.lang.python; python-dev is for discussion of specific changes to the Python language and libraries. Until you are familiar with the way threading works in Python and have some concrete proposal to make, please keep the discussion on c.l.py. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "It is easier to optimize correct code than to correct optimized code." --Bill Harlan From fuerte at sci.fi Thu Sep 11 20:53:10 2003 From: fuerte at sci.fi (Harri Pesonen) Date: Thu Sep 11 12:53:11 2003 Subject: [Python-Dev] Making python C-API thread safe (try 2) Message-ID: <3F60A876.7020204@sci.fi> Sorry about the previous post! :-) Background: I'm new to Python. I just embedded Python into a multi-threaded application, and extended Python by callbacks as well. During this process I realised that Python is not internally as beautiful as externally. It is not thread safe. This surprised me, because Python is a quite young programming language. I started thinking how to fix this, how to make Python thread safe. Currently Python does not support multiprocessor machines or hyperthreading processors. It is only a matter of time when this has to be fixed, better soon than later. How to make C-API thread safe: It's quite simple, in fact. Thread should first call PyThreadState* Py_NewInterpreter() and use this thread state pointer in every following API call. This thread state pointer should point to a memory structure, that holds everything that Python interpreter needs. There should be no global variables, not even None, except const char* Py_GetVersion() and such. So every Python function should have this thread state as the first argument: int PyRun_SimpleString(PyThreadState*tstate, char *command) and so on. Also callbacks should have it, about every API function. Advantages: The following functions would not be needed any more, it would make the API much simpler: void PyEval_InitThreads( ) void PyEval_AcquireLock( ) void PyEval_ReleaseLock( ) void PyEval_AcquireThread( PyThreadState *tstate) void PyEval_ReleaseThread( PyThreadState *tstate) PyThreadState* PyEval_SaveThread( ) void PyEval_RestoreThread( PyThreadState *tstate) Py_BEGIN_ALLOW_THREADS Py_END_ALLOW_THREADS Py_BLOCK_THREADS Py_UNBLOCK_THREADS PyInterpreterState* PyInterpreterState_New( ) void PyInterpreterState_Clear( PyInterpreterState *interp) void PyInterpreterState_Delete( PyInterpreterState *interp) PyThreadState* PyThreadState_New( PyInterpreterState *interp) void PyThreadState_Clear( PyThreadState *tstate) void PyThreadState_Delete( PyThreadState *tstate) PyThreadState* PyThreadState_Get( ) PyThreadState* PyThreadState_Swap( PyThreadState *tstate) The current documentation says: "In order to support multi-threaded Python programs, the interpreter regularly releases and reacquires the lock -- by default, every 100 bytecode instructions". This would not be needed anymore. It would make the interpreter faster. Making this change would take a couple of weeks, if the job would be divided between modules to different persons. The change is trivial, but it would have to be made everywhere. Disadvantages: Everything old would break. Thread safe Python should have a version number significantly higher, like Python 3. Old applications would work with old Python versions, with old Python DLLs, but they would need to be rewritten for the new API. It would be a trivial task, but still. Disclaimer: I already got mail saying: " This issue gets discussed regularly on comp.lang.python; python-dev is for discussion of specific changes to the Python language and libraries. Until you are familiar with the way threading works in Python and have some concrete proposal to make, please keep the discussion on c.l.py." The current situation is very unfortunate, having a single threaded implementation in the modern computer age. I have a concrete proposal, which would break everything :-) but I think that is the only solution. Thanks for reading, Harri From pje at telecommunity.com Thu Sep 11 14:10:06 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Sep 11 13:10:24 2003 Subject: [Python-Dev] Making python C-API thread safe (try 2) In-Reply-To: <3F60A876.7020204@sci.fi> Message-ID: <5.1.1.6.0.20030911130607.02426ec0@telecommunity.com> At 07:53 PM 9/11/03 +0300, Harri Pesonen wrote: >Sorry about the previous post! :-) > >Background: I'm new to Python. I just embedded Python into a >multi-threaded application, and extended Python by callbacks as well. >During this process I realised that Python is not internally as beautiful >as externally. It is not thread safe. This surprised me, because Python is >a quite young programming language. I started thinking how to fix this, >how to make Python thread safe. Currently Python does not support >multiprocessor machines or hyperthreading processors. It is only a matter >of time when this has to be fixed, better soon than later. > >How to make C-API thread safe: It's quite simple, in fact. Thread should >first call > >PyThreadState* Py_NewInterpreter() > >and use this thread state pointer in every following API call. This thread >state pointer should point to a memory structure, that holds everything >that Python interpreter needs. There should be no global variables, not >even None, except This only makes individual Python interpreters thread-safe. That might certainly be useful for some embedded applications, but it does nothing for thread-safety *within* an interpreter. In other words, Python would *still* need a global interpreter lock, in order for threaded programs written in Python to behave properly. So, your solution would help only creators of multithreaded embedded applications, wherein each thread desired an isolated Python interpreter. Meanwhile, it would impose both a one-time and ongoing cost penalty on *all other* Python users, without giving them any benefit in return. From fuerte at sci.fi Thu Sep 11 21:47:49 2003 From: fuerte at sci.fi (Harri Pesonen) Date: Thu Sep 11 13:47:46 2003 Subject: [Python-Dev] Making python C-API thread safe (try 2) In-Reply-To: <5.1.1.6.0.20030911130607.02426ec0@telecommunity.com> References: <5.1.1.6.0.20030911130607.02426ec0@telecommunity.com> Message-ID: <3F60B545.6030100@sci.fi> Phillip J. Eby wrote: > At 07:53 PM 9/11/03 +0300, Harri Pesonen wrote: > >> Sorry about the previous post! :-) >> >> Background: I'm new to Python. I just embedded Python into a >> multi-threaded application, and extended Python by callbacks as well. >> During this process I realised that Python is not internally as >> beautiful as externally. It is not thread safe. This surprised me, >> because Python is a quite young programming language. I started >> thinking how to fix this, how to make Python thread safe. Currently >> Python does not support multiprocessor machines or hyperthreading >> processors. It is only a matter of time when this has to be fixed, >> better soon than later. >> >> How to make C-API thread safe: It's quite simple, in fact. Thread >> should first call >> >> PyThreadState* Py_NewInterpreter() >> >> and use this thread state pointer in every following API call. This >> thread state pointer should point to a memory structure, that holds >> everything that Python interpreter needs. There should be no global >> variables, not even None, except > > > This only makes individual Python interpreters thread-safe. That > might certainly be useful for some embedded applications, but it does > nothing for thread-safety *within* an interpreter. > > In other words, Python would *still* need a global interpreter lock, > in order for threaded programs written in Python to behave properly. > > So, your solution would help only creators of multithreaded embedded > applications, wherein each thread desired an isolated Python > interpreter. Meanwhile, it would impose both a one-time and ongoing > cost penalty on *all other* Python users, without giving them any > benefit in return. You are right, I thought only about making interpreters thread safe, thanks for pointing this out. But this is still needed, it is unavoidable in the long run. What do you mean about the penalty? If you mean having the state pointer in stack in every call, I don't think so. If we think about making Python threads thread-safe, within the interpreter, then I think that it should be left for the application deverloper, in principle. He should create "critical sections" and other locks when needed (I don't know how it is done currently). I can imagine that None and other values will cause problems, when two threads increment or decrement the reference count at the same time. Of course there is a simple solution for that, InterlockedIncrement and InterlockedDecrement in Win32 at least. I can imagine that there are other problems with garbage collector, and so on. I don't want to go into details about this, I got my message through, I was almost losing sleep when I realised the current situation. But my basic message is this: Python needs to be made thread safe. Making the individual interpreters thread safe is trivial, and benefits many people, and is a necessary first step; making threads within interpreter thread safe is possible as well, at least if you leave something for the developer, as you should, as you do in every other programming language as well. Harri From brian at sweetapp.com Thu Sep 11 12:06:43 2003 From: brian at sweetapp.com (Brian Quinlan) Date: Thu Sep 11 14:05:57 2003 Subject: [Python-Dev] Making python C-API thread safe (try 2) In-Reply-To: <3F60B545.6030100@sci.fi> Message-ID: <001001c3788f$75798320$21795418@dell1700> > Making the individual interpreters thread safe is trivial, and benefits > many people, I'm not sure that I believe this statement. Isn't this of benefit only to people who want to have multiple interpreter instances running concurrently in an embedding situation? I doubt that there is an army of people who want to do that. > and is a necessary first step; making threads within > interpreter thread safe is possible as well, at least if you leave > something for the developer, as you should, as you do in every other > programming language as well. Of course it is possible (and would likely be of benefit to more than a handful of Python users). The problem is that it requires effort, would likely require massive code breakage and the locking required would likely significantly decrease Python's single-thread performance. Cheers, Brian From aahz at pythoncraft.com Thu Sep 11 15:21:36 2003 From: aahz at pythoncraft.com (Aahz) Date: Thu Sep 11 14:21:41 2003 Subject: [Python-Dev] Making python C-API thread safe (try 2) In-Reply-To: <001001c3788f$75798320$21795418@dell1700> References: <3F60B545.6030100@sci.fi> <001001c3788f$75798320$21795418@dell1700> Message-ID: <20030911182136.GA12857@panix.com> Please, people, let's take this off python-dev and move it to comp.lang.python. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "It is easier to optimize correct code than to correct optimized code." --Bill Harlan From dave at boost-consulting.com Thu Sep 11 16:11:54 2003 From: dave at boost-consulting.com (David Abrahams) Date: Thu Sep 11 15:13:05 2003 Subject: [Python-Dev] PyUnicode_FromEncodedObject Message-ID: I'd post a SF bug but it keeps insisting I log in, even after I have already. There's a bug, either in the code or docs for PyUnicode_FromEncodedObject. The docs read: "Unicode objects are passed back as-is with incremented refcount. Note: These cannot be decoded; passing a non-NULL value for encoding will result in a TypeError." 'tain't so; the following shows that the error is unconditional. if (obj == NULL) { PyErr_BadInternalCall(); return NULL; } #if 0 /* For b/w compatibility we also accept Unicode objects provided that no encodings is given and then redirect to PyObject_Unicode() which then applies the additional logic for Unicode subclasses. NOTE: This API should really only be used for object which represent *encoded* Unicode ! */ if (PyUnicode_Check(obj)) { if (encoding) { PyErr_SetString(PyExc_TypeError, "decoding Unicode is not supported"); return NULL; } return PyObject_Unicode(obj); } #else if (PyUnicode_Check(obj)) { PyErr_SetString(PyExc_TypeError, "decoding Unicode is not supported"); return NULL; } #endif -- Dave Abrahams Boost Consulting www.boost-consulting.com From fuerte at sci.fi Thu Sep 11 23:16:03 2003 From: fuerte at sci.fi (Harri Pesonen) Date: Thu Sep 11 15:16:06 2003 Subject: [Python-Dev] Making python C-API thread safe (try 2) In-Reply-To: <5.1.1.6.0.20030911142317.02b88640@telecommunity.com> References: <5.1.1.6.0.20030911130607.02426ec0@telecommunity.com> <5.1.1.6.0.20030911130607.02426ec0@telecommunity.com> <5.1.1.6.0.20030911142317.02b88640@telecommunity.com> Message-ID: <3F60C9F3.4060107@sci.fi> Phillip J. Eby wrote: > At 08:47 PM 9/11/03 +0300, Harri Pesonen wrote: > >> But my basic message is this: Python needs to be made thread safe. >> Making the individual interpreters thread safe is trivial, and >> benefits many people, and is a necessary first step; > > > It's far from trivial - you're talking about invalidating every piece > of C code written for Python over a multi-year people by dozens upon > dozens of extension authors. The change is trivial in Python C API. I already said that it would break everything outside the Python distribution, but the change in other applications is also trivial. > It doesn't benefit many people: only those using isolated interpreters > embedded in a multithreaded C program. I don't know how many people are writing threads in Python, either. I guess that not so many. In my case I only need a thread safe interpreter, I don't create threads in Python code. So just having what I described would be enough for me: no need for global interpreter lock, and Python would be really multithreading. It would benefit many people, I'm sure. > >> making threads within interpreter thread safe is possible as well, at >> least if you leave something for the developer, as you should, as you >> do in every other programming language as well. > > > You misunderstand. Those "critical sections" are for the most part in > Python's C code, not in the Python script. Yes, I'm aware of the None problem at least (only one instance of it). Please enlighten me about the other critical sections? Object allocation/freeing? > I'm guessing you haven't done much writing of C extensions for Python > (or Python core C), or else you'd realize why trying to make > INCREF/DECREF threadsafe would absolutely decimate performance. > Reference count updates happen *way* too often in normal code flow. > I also knew that already. But how else can you do it? Of course, changing Python to not have a single None would help a lot. Or, perhaps it could have a single None, but in case of None, the reference count would have no meaning, it would never be deallocated, because it would be checked in code. Maybe it does it already, I don't know. I'm also wondering why this problem has not been addressed before? If I had the power to change Python, this would be the first thing I did. Harri From jeremy at alum.mit.edu Thu Sep 11 16:32:45 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Thu Sep 11 15:32:55 2003 Subject: [Python-Dev] Making python C-API thread safe (try 2) In-Reply-To: <3F60C9F3.4060107@sci.fi> References: <5.1.1.6.0.20030911130607.02426ec0@telecommunity.com> <5.1.1.6.0.20030911130607.02426ec0@telecommunity.com> <5.1.1.6.0.20030911142317.02b88640@telecommunity.com> <3F60C9F3.4060107@sci.fi> Message-ID: <1063308764.2069.235.camel@localhost.localdomain> On Thu, 2003-09-11 at 15:16, Harri Pesonen wrote: > I'm also wondering why this problem has not been addressed before? If I > had the power to change Python, this would be the first thing I did. Try coming up with a patch. I expect it would be considered provided that it was maximally backwards compatible with the existing C API and did not reduce performance on benchmarks like pystone. That is to say, sure it would be nice, but at what cost? Jeremy From fuerte at sci.fi Fri Sep 12 00:31:25 2003 From: fuerte at sci.fi (Harri Pesonen) Date: Thu Sep 11 16:31:22 2003 Subject: [Python-Dev] Making python C-API thread safe (try 2) In-Reply-To: <1063308764.2069.235.camel@localhost.localdomain> References: <5.1.1.6.0.20030911130607.02426ec0@telecommunity.com> <5.1.1.6.0.20030911130607.02426ec0@telecommunity.com> <5.1.1.6.0.20030911142317.02b88640@telecommunity.com> <3F60C9F3.4060107@sci.fi> <1063308764.2069.235.camel@localhost.localdomain> Message-ID: <3F60DB9D.8060205@sci.fi> Jeremy Hylton wrote: >On Thu, 2003-09-11 at 15:16, Harri Pesonen wrote: > > >>I'm also wondering why this problem has not been addressed before? If I >>had the power to change Python, this would be the first thing I did. >> >> > >Try coming up with a patch. I expect it would be considered provided >that it was maximally backwards compatible with the existing C API and >did not reduce performance on benchmarks like pystone. > It would be totally incompatible with the existing C API. The performance would be better. >That is to say, sure it would be nice, but at what cost? > > At cost of breaking up with the past, but eventually it has to be done. If I had time, I would create this multithreading Python myself, call it a different language perhaps. I don't know if the Python licence allows this. Harri -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20030911/a31103f3/attachment.htm From martin at v.loewis.de Thu Sep 11 21:46:35 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Sep 11 16:46:35 2003 Subject: [Python-Dev] PyUnicode_FromEncodedObject In-Reply-To: References: Message-ID: David Abrahams writes: > I'd post a SF bug but it keeps insisting I log in, even after I have > already. It does not insist, it just explains that you have logged on. Unfortunately, there is no way to make this message disappear once you have logged on, so all of us see it all the time. > > There's a bug, either in the code or docs for > PyUnicode_FromEncodedObject. The docs read: > > "Unicode objects are passed back as-is with incremented > refcount. Note: These cannot be decoded; passing a non-NULL value > for encoding will result in a TypeError." Please do submit a bug report for that. Regards, Martin From martin at v.loewis.de Thu Sep 11 21:54:11 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Sep 11 16:54:13 2003 Subject: [Python-Dev] Making python C-API thread safe (try 2) In-Reply-To: <3F60C9F3.4060107@sci.fi> References: <5.1.1.6.0.20030911130607.02426ec0@telecommunity.com> <5.1.1.6.0.20030911130607.02426ec0@telecommunity.com> <5.1.1.6.0.20030911142317.02b88640@telecommunity.com> <3F60C9F3.4060107@sci.fi> Message-ID: Harri Pesonen writes: > The change is trivial in Python C API. I already said that it would > break everything outside the Python distribution, but the change in > other applications is also trivial. If it is trivial, would you mind posting a patch somewhere? > > You misunderstand. Those "critical sections" are for the most part > > in Python's C code, not in the Python script. > > Yes, I'm aware of the None problem at least (only one instance of > it). Please enlighten me about the other critical sections? Object > allocation/freeing? Yes, that, plus: - allocation of/access to small numbers - access to global variables in extension modules (e.g. cursesmodule.c:PyCursesError) - type objects etc. > Of course, changing Python to not have a single None would help a > lot. Or, perhaps it could have a single None, but in case of None, the > reference count would have no meaning, it would never be deallocated, > because it would be checked in code. Maybe it does it already, I don't > know. So how can you know a patch would be trivial? > I'm also wondering why this problem has not been addressed before? If > I had the power to change Python, this would be the first thing I did. Please go ahead and post a patch. You might find that the patch is difficult to write, and, once written, will have many errors. Once those are fixed, you might find that Python becomes painfully slow, and lose a lot of portability. Regards, Martin From martin at v.loewis.de Fri Sep 12 00:07:56 2003 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu Sep 11 17:08:40 2003 Subject: [Python-Dev] Making python C-API thread safe (try 2) In-Reply-To: <3F60DB9D.8060205@sci.fi> References: <5.1.1.6.0.20030911130607.02426ec0@telecommunity.com> <5.1.1.6.0.20030911130607.02426ec0@telecommunity.com> <5.1.1.6.0.20030911142317.02b88640@telecommunity.com> <3F60C9F3.4060107@sci.fi> <1063308764.2069.235.camel@localhost.localdomain> <3F60DB9D.8060205@sci.fi> Message-ID: <3F60E42C.1040501@v.loewis.de> Harri Pesonen wrote: > If I had time, I would create this multithreading Python myself, call it > a different language perhaps. I don't know if the Python licence allows > this. The license is only on the implementation (source code), not on the language itself. You are free to provide alternative implementations of the language, without having to ask anybody for permission. But please don't call such a thing trivial when you know it is inherently complex. Regards, Martin From greg at cosc.canterbury.ac.nz Fri Sep 12 16:11:28 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Sep 11 23:11:46 2003 Subject: [Python-Dev] Making python C-API thread safe (try 2) In-Reply-To: <3F60C9F3.4060107@sci.fi> Message-ID: <200309120311.h8C3BSe01781@oma.cosc.canterbury.ac.nz> > Yes, I'm aware of the None problem at least (only one instance of it). > Please enlighten me about the other critical sections? Object > allocation/freeing? None is only the molecule at the tip of the iceberg. If you're talking about threads within a Python interpreter, doing just about anything at all to any Python object is a critical section. The amount of locking required to deal with that at a fine-grained level would totally kill performance, not to mention being hugely tedious and error-prone to implement. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From dave at boost-consulting.com Fri Sep 12 00:35:40 2003 From: dave at boost-consulting.com (David Abrahams) Date: Thu Sep 11 23:36:48 2003 Subject: [Python-Dev] PyUnicode_FromEncodedObject In-Reply-To: (Martin v. =?iso-8859-1?q?L=F6wis's?= message of "11 Sep 2003 22:46:13 +0200") References: Message-ID: martin@v.loewis.de (Martin v. L?wis) writes: > David Abrahams writes: > >> I'd post a SF bug but it keeps insisting I log in, even after I have >> already. > > It does not insist, it just explains that you have logged > on. Unfortunately, there is no way to make this message disappear once > you have logged on, so all of us see it all the time. No, seriously. There is no "submit new" button/link for me. -- Dave Abrahams Boost Consulting www.boost-consulting.com From fuerte at sci.fi Fri Sep 12 08:56:55 2003 From: fuerte at sci.fi (Harri Pesonen) Date: Fri Sep 12 00:56:56 2003 Subject: [Python-Dev] Making python C-API thread safe (try 2) In-Reply-To: <5.1.1.6.0.20030911162016.02027750@telecommunity.com> References: <5.1.1.6.0.20030911142317.02b88640@telecommunity.com> <5.1.1.6.0.20030911130607.02426ec0@telecommunity.com> <5.1.1.6.0.20030911130607.02426ec0@telecommunity.com> <5.1.1.6.0.20030911142317.02b88640@telecommunity.com> <5.1.1.6.0.20030911162016.02027750@telecommunity.com> Message-ID: <3F615217.8060106@sci.fi> Phillip J. Eby wrote: > Please do not CC: my mail to Python-Dev again; I intentionally did not > include python-dev on my CC: because it was asked that we move this > thread elsewhere. > > At 10:16 PM 9/11/03 +0300, Harri Pesonen wrote: > >> Phillip J. Eby wrote: >> >>> At 08:47 PM 9/11/03 +0300, Harri Pesonen wrote: >>> >>>> But my basic message is this: Python needs to be made thread safe. >>>> Making the individual interpreters thread safe is trivial, and >>>> benefits many people, and is a necessary first step; >>> >>> >>> >>> It's far from trivial - you're talking about invalidating every >>> piece of C code written for Python over a multi-year people by >>> dozens upon dozens of extension authors. >> >> >> The change is trivial in Python C API. I already said that it would >> break everything outside the Python distribution, but the change in >> other applications is also trivial. > > > How do you propose that C code called *from* Python *receive* the > threadstate pointer? Exactly like that. Is there a problem? I'm suggesting that every function call gets that pointer, unless the function can get it from some other argument, that contains a pointer to it. > >>> It doesn't benefit many people: only those using isolated >>> interpreters embedded in a multithreaded C program. >> >> >> I don't know how many people are writing threads in Python, either. I >> guess that not so many. In my case I only need a thread safe >> interpreter, I don't create threads in Python code. So just having >> what I described would be enough for me: no need for global >> interpreter lock, and Python would be really multithreading. It would >> benefit many people, I'm sure. > > > Obviously, it's enough for you, or you wouldn't be proposing it. What > does it do for me? Nothing whatsoever, except add needless overhead > and make me rewrite every C extension I've ever written for Python. > So, by and large, you're not going to get much support for your change > from Python developers, especially those who write C extensions, or > depend on extensions written by others. Probably. That's why I'm thinking now that the language should be called something else, like MPython for "multi-threading Python". It would be 99% compatible with the existing Python syntax, but have different internals. > >> Yes, I'm aware of the None problem at least (only one instance of >> it). Please enlighten me about the other critical sections? Object >> allocation/freeing? > > > Data structure manipulations, e.g. all use of dictionaries. Python > spends most of its time doing dictionary lookups or modifications, all > of which need to be protected. After sleeping over night, I think that I got it. :-) The simple solution is, that each thread created in Python gets its own independent interpreter state as well. And there could be a separate thread-global interpreter state for shared memory access. Access to this global state would always be synchronized. There could even be multiple named global states, so that the thread interlocking could be minimized. The python syntax for creating objects in this global state should be invented: synchronize a = "abcd" Also when creating the new thread, it's arguments would be copied from the creating state to the new state. What does it sound? Of course it would be incompatible with the current threading system in Python, but it would be totally multithreading, no global interpreter lock needed. It would be faster than current Python, there would be no need to free or acquire the lock when calling OS functions, and no need to check how many byte codes have been processed, etc. > >>> I'm guessing you haven't done much writing of C extensions for >>> Python (or Python core C), or else you'd realize why trying to make >>> INCREF/DECREF threadsafe would absolutely decimate performance. >>> Reference count updates happen *way* too often in normal code flow. >> >> I also knew that already. But how else can you do it? > > > The way it's done now! :) I understand why the current Python works like it does. But I think that it's time for the next generation. If you don't do it, and I have no time now to do it, I'm still sure that this is done at some point, rather sooner than later. > >> Of course, changing Python to not have a single None would help a >> lot. Or, perhaps it could have a single None, but in case of None, >> the reference count would have no meaning, it would never be >> deallocated, because it would be checked in code. Maybe it does it >> already, I don't know. > > > I really don't mean to be rude (another reason I'm writing this to you > privately), but this paragraph shows you are *really* new to Python > both at the level of coding in Python and coding with Python's C API. > I wish I could explain in detail why, but there's really far too much > that you don't understand and it would take me too long. I will > attempt to summarize a very few points, however: first, identity > (pointer comparison) is a part of the Python language, so you can't > have multiple None instances any more than you can have more than one > value be NULL in C. Second, at the C level, all Python objects > (including None) have an absolutely uniform API, so having refcount > behavior be different for different kinds of objects is not at all > practical. Third, if you had more than one Py_None at the C level, > you'd either have to make Py_None a macro, or rewrite all the C. If > you don't think that's a problem, you have absolutely no idea how much > C code out there is written to the Python API. Yes, Py_None would be a macro. All access to interpreter state would go through the interpreter state pointer that is always in stack, the first argument each C API function gets. That pointer should be named so that the macros will always work ("tState", for example, so that Py_None macro would expand to tState->mPy_None, for example). > >> I'm also wondering why this problem has not been addressed before? > > > It has; the cure is worse than the disease. A few years ago, somebody > wrote a "free-threading" version of Python, which locked individual > data objects rather than use the global interpreter lock. The > performance for single-threaded programs was abominable, and the > performance gain even on multiprocessor machines was not thought worth > the cost. So the project was scrapped. There would be no locking in my proposal, except when accessing the shared memory global thread state. I don't know, I got mail about writing a PEP. It is clear that it would not be accepted, because it would break the existing API. The change is so big that I think that it has to be called a different language. This is the last message I will make about this matter (before actually starting to code it), so I'm posting this to python-list as well, because this is too important to be ignored. Python *needs* to be free-threading... Harri From martin at v.loewis.de Fri Sep 12 06:56:01 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Fri Sep 12 01:56:03 2003 Subject: [Python-Dev] PyUnicode_FromEncodedObject In-Reply-To: References: Message-ID: David Abrahams writes: > > It does not insist, it just explains that you have logged > > on. Unfortunately, there is no way to make this message disappear once > > you have logged on, so all of us see it all the time. > > No, seriously. There is no "submit new" button/link for me. What happens if you navigate to http://sourceforge.net/tracker/?group_id=5470&atid=105470 ? Do you have cookies enabled in your browser? Regards, Martin From dave at boost-consulting.com Fri Sep 12 08:38:49 2003 From: dave at boost-consulting.com (David Abrahams) Date: Fri Sep 12 07:39:57 2003 Subject: [Python-Dev] PyUnicode_FromEncodedObject In-Reply-To: (Martin v. =?iso-8859-1?q?L=F6wis's?= message of "12 Sep 2003 07:55:47 +0200") References: Message-ID: martin@v.loewis.de (Martin v. L?wis) writes: > David Abrahams writes: > >> > It does not insist, it just explains that you have logged >> > on. Unfortunately, there is no way to make this message disappear once >> > you have logged on, so all of us see it all the time. >> >> No, seriously. There is no "submit new" button/link for me. > > What happens if you navigate to > > http://sourceforge.net/tracker/?group_id=5470&atid=105470 > > ? Do you have cookies enabled in your browser? Yep. Submitted now. -- Dave Abrahams Boost Consulting www.boost-consulting.com From python at rcn.com Fri Sep 12 14:17:18 2003 From: python at rcn.com (Raymond Hettinger) Date: Fri Sep 12 14:48:13 2003 Subject: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6 Message-ID: <001d01c3795d$c2ba1400$30bc958d@oemcomputer> [Raymond] > > I would like to backport this patch to Py2.3.1. > > The effort to provide a full mapping interface to all > > mapping like objects was attempted in Py2.3 and several > > modules for the bsddb package were updated, but this > > one was missed and the package was left half converted. > > > > IIRC, dbhash and bsddb don't affect the Apple MacIntosh > > users. Also, since this effort was started for bsddb and > > only half completed, I view it to be a bit of a bugfix as > > well as being featurelike. It certainly affects the usability > > of the module (the looping example and related text in > > the docs were both wrong -- that would not have happened > > if the normal looping expectations were supported). [GvR] > Can you discuss this on python-dev? Guys, are you okay with backporting this? Raymond Hettinger Index: __init__.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/bsddb/__init__.py,v retrieving revision 1.5 retrieving revision 1.6 diff -C2 -d -r1.5 -r1.6 *** __init__.py 24 Apr 2003 16:02:44 -0000 1.5 --- __init__.py 12 Sep 2003 06:33:37 -0000 1.6 *************** *** 53,58 **** #---------------------------------------------------------------------- ! class _DBWithCursor: """ A simple wrapper around DB that makes it look like the bsddbobject in --- 53,59 ---- #---------------------------------------------------------------------- + import UserDict ! class _DBWithCursor(UserDict.DictMixin): """ A simple wrapper around DB that makes it look like the bsddbobject in *************** *** 145,148 **** --- 146,157 ---- return self.db.sync() + def __iter__(self): + try: + yield self.first()[0] + next = self.next + while 1: + yield next()[0] + except _bsddb.DBNotFoundError: + return #---------------------------------------------------------------------- From skip at pobox.com Fri Sep 12 15:02:13 2003 From: skip at pobox.com (Skip Montanaro) Date: Fri Sep 12 15:02:27 2003 Subject: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: <001d01c3795d$c2ba1400$30bc958d@oemcomputer> References: <001d01c3795d$c2ba1400$30bc958d@oemcomputer> Message-ID: <16226.6197.573407.585746@montanaro.dyndns.org> >> > IIRC, dbhash and bsddb don't affect the Apple MacIntosh users. I don't know what you mean here. Do you mean Mac OS 9 or Mac OS X? I use bsddb all the time on Mac OS X. Raymond> [GvR] >> Can you discuss this on python-dev? Raymond> Guys, are you okay with backporting this? What are we discussing? Just adding an iterator to objects opened with the old bsddb API? Skip From python at rcn.com Fri Sep 12 15:19:01 2003 From: python at rcn.com (Raymond Hettinger) Date: Fri Sep 12 15:23:55 2003 Subject: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6 References: <001d01c3795d$c2ba1400$30bc958d@oemcomputer> <16226.6197.573407.585746@montanaro.dyndns.org> Message-ID: <005a01c37962$b94a0e20$30bc958d@oemcomputer> [Raymond] > >> > IIRC, dbhash and bsddb don't affect the Apple MacIntosh users. [Skip] > I don't know what you mean here. Do you mean Mac OS 9 or Mac OS X? I > use bsddb all the time on Mac OS X. Hmm, I'll have to fix the docs too. They currently list bsddb as being only for Unix and Windows. [Skip] > What are we discussing? Just adding an iterator to objects opened with the > old bsddb API? Yes. Iterators and the rest of the mapping API (__iter__, pop, popitem, iteritems, setdefault, etc). Like dumbdbm and shelve, this is done by including UserDict.DictMixin as a base class. Raymond Hettinger ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From skip at pobox.com Fri Sep 12 15:55:44 2003 From: skip at pobox.com (Skip Montanaro) Date: Fri Sep 12 16:10:09 2003 Subject: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: <005a01c37962$b94a0e20$30bc958d@oemcomputer> References: <001d01c3795d$c2ba1400$30bc958d@oemcomputer> <16226.6197.573407.585746@montanaro.dyndns.org> <005a01c37962$b94a0e20$30bc958d@oemcomputer> Message-ID: <16226.9408.548163.145368@montanaro.dyndns.org> >> I don't know what you mean here. Do you mean Mac OS 9 or Mac OS X? >> I use bsddb all the time on Mac OS X. Raymond> Hmm, I'll have to fix the docs too. They currently list bsddb Raymond> as being only for Unix and Windows. For non-GUI stuff Mac OS X is more Unix than Macintosh. ;-) Raymond> Yes. Iterators and the rest of the mapping API (__iter__, pop, Raymond> popitem, iteritems, setdefault, etc). Raymond> Like dumbdbm and shelve, this is done by including Raymond> UserDict.DictMixin as a base class. I say go for it unless there's a serious backward compatibility problem. I suspect the vast majority of the people who use bsddb do so through the anydbm module, so if usage that way doesn't break you should be okay. Skip From jeremy at alum.mit.edu Fri Sep 12 16:44:15 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Fri Sep 12 16:44:29 2003 Subject: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: <001d01c3795d$c2ba1400$30bc958d@oemcomputer> References: <001d01c3795d$c2ba1400$30bc958d@oemcomputer> Message-ID: <1063399454.2069.299.camel@localhost.localdomain> On Fri, 2003-09-12 at 14:17, Raymond Hettinger wrote: > [Raymond] > > > I would like to backport this patch to Py2.3.1. > > > The effort to provide a full mapping interface to all > > > mapping like objects was attempted in Py2.3 and several > > > modules for the bsddb package were updated, but this > > > one was missed and the package was left half converted. > > > > > > IIRC, dbhash and bsddb don't affect the Apple MacIntosh > > > users. Also, since this effort was started for bsddb and > > > only half completed, I view it to be a bit of a bugfix as > > > well as being featurelike. It certainly affects the usability > > > of the module (the looping example and related text in > > > the docs were both wrong -- that would not have happened > > > if the normal looping expectations were supported). > > [GvR] > > Can you discuss this on python-dev? > > Guys, are you okay with backporting this? Isn't this just the sort of little feature that was the subject of recent discussion on the dangers of backporting. It would mean that someone writing an app against Python 2.3 could never be sure whether the feature existed. In practice, that would mean developers using 2.3.x would only find out about problems after deployment when they bump into a user with the original 2.3. I'm not sure how convincing I find this argument, but it's got some merit. Jeremy From python at rcn.com Fri Sep 12 19:50:40 2003 From: python at rcn.com (Raymond Hettinger) Date: Fri Sep 12 20:11:51 2003 Subject: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6 References: <001d01c3795d$c2ba1400$30bc958d@oemcomputer> <1063399454.2069.299.camel@localhost.localdomain> Message-ID: <008a01c3798a$f7198b40$b439c797@oemcomputer> [Raymond] > > > > I would like to backport this patch to Py2.3.1. > > > > The effort to provide a full mapping interface to all > > > > mapping like objects was attempted in Py2.3 and several > > > > modules for the bsddb package were updated, but this > > > > one was missed and the package was left half converted. > > > > > > > > IIRC, dbhash and bsddb don't affect the Apple MacIntosh > > > > users. Also, since this effort was started for bsddb and > > > > only half completed, I view it to be a bit of a bugfix as > > > > well as being featurelike. It certainly affects the usability > > > > of the module (the looping example and related text in > > > > the docs were both wrong -- that would not have happened > > > > if the normal looping expectations were supported). [GvR] > > > Can you discuss this on python-dev? [Raymond] > > Guys, are you okay with backporting this? [Jeremy] > Isn't this just the sort of little feature that was the subject of > recent discussion on the dangers of backporting. It would mean that > someone writing an app against Python 2.3 could never be sure whether > the feature existed. In practice, that would mean developers using > 2.3.x would only find out about problems after deployment when they bump > into a user with the original 2.3. > > I'm not sure how convincing I find this argument, but it's got some > merit. It has some. That's why GvR had me bring it to python-dev so you guys could help decide on the proper balance. But for that one thought, the decision to apply is clear cut. Thoughts in favor of applying: * the existing interface is a pain and is misdocumented in Py2.3.0 * the patch was already half complete for 2.3.0 (applied to dbobj and dbshelve) __init__.py was just missed). * the patch was already applied to dumbdbm and shelve for Py2.3.0 * if needed, it's not hard to write code that automatically adjusts for Py2.3.0: if not f.__class__.__bases__: f.__class__.__bases = (UserDict.DictMixin,) * waiting another 18 months to put this in just results in that many more installed Pythons out there without the patch Raymond From martin at v.loewis.de Sat Sep 13 00:52:37 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sat Sep 13 00:52:50 2003 Subject: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: <008a01c3798a$f7198b40$b439c797@oemcomputer> References: <001d01c3795d$c2ba1400$30bc958d@oemcomputer> <1063399454.2069.299.camel@localhost.localdomain> <008a01c3798a$f7198b40$b439c797@oemcomputer> Message-ID: "Raymond Hettinger" writes: > It has some. That's why GvR had me bring it to python-dev > so you guys could help decide on the proper balance. > But for that one thought, the decision to apply is clear cut. There is no proper balance. If you believe in the policy "no new features", this policy is absolute - it would be pointless to accept some new features, but reject others on grounds of the policy. This policy is mutually exclusive with "new features are fine as long as they don't break anything". You can get opinions, but you cannot get consensus. Regards, Martin From python at rcn.com Sat Sep 13 02:14:12 2003 From: python at rcn.com (Raymond Hettinger) Date: Sat Sep 13 02:19:06 2003 Subject: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6 References: <001d01c3795d$c2ba1400$30bc958d@oemcomputer><1063399454.2069.299.camel@localhost.localdomain><008a01c3798a$f7198b40$b439c797@oemcomputer> Message-ID: <007c01c379be$40efa460$b439c797@oemcomputer> [Raymond] > > It has some. That's why GvR had me bring it to python-dev > > so you guys could help decide on the proper balance. > > But for that one thought, the decision to apply is clear cut. [Martin] > There is no proper balance. If you believe in the policy "no new > features", this policy is absolute - it would be pointless to accept > some new features, but reject others on grounds of the policy. > > This policy is mutually exclusive with "new features are fine as long > as they don't break anything". You can get opinions, but you cannot > get consensus. Well put! Even bugfixes are features in that something will work under Py2.3.1 that won't work under Py2.3.0. So, even an absolutist policy on our end shouldn't create a false sense of security or relieve a developer from testing his or her application on the lowest numbered Python for which it is claimed to run (including micro-releases). That has always been true. If you developed something under Py2.2.3 and expected it to run under Py2.2.0, the only way to be sure would be to test it with Py2.2.0. In this particular case, we are (IMHO) much better off adding an iterable interface than leaving current situation with "db.first()" and "while 1: db.next()" wrapped in a try/except for an undocumented exception class -- yuck. Worse still, the docs for Py2.3.0 cannot be changed and they are wrong on several counts (the methods return items instead of keys; EOF is signaled by an exception instead of None; and the example doesn't run). Raymond ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From arigo at tunes.org Sat Sep 13 09:46:53 2003 From: arigo at tunes.org (Armin Rigo) Date: Sat Sep 13 09:48:38 2003 Subject: [Python-Dev] pre-PEP: Resource-Release Support for Generators In-Reply-To: <200308262050.h7QKoqD04780@12-236-84-31.client.attbi.com> References: <5.2.1.1.0.20030826151628.023f8d68@pop.bluewin.ch> <00a401c36c0f$76952160$c304a044@oemcomputer> <200308262050.h7QKoqD04780@12-236-84-31.client.attbi.com> Message-ID: <20030913134653.GA12752@vicky.ecs.soton.ac.uk> Hello, (My reply didn't seem to have reached python-dev... second try.) On Tue, Aug 26, 2003 at 01:50:52PM -0700, Guido van Rossum wrote: > Another comment on Samuele's PEP: It is sort of sad that the *user* of > a generator has to know that the generator's close() must be called. > Normally, the beauty of using a try/finally for cleanup is that your > callers don't need to know about it. But I see no way around this. What about letting the 'for' handle this? It's the most common way generators are used. When a 'for' loop on a generator-iterator finishes it would call the close() method of the iterator, which on generators would (say) simulate a return from the latest 'yield'. Well, close() might not be such a good name because it would probably break exsiting code (e.g. closing files unexpectedly), but __exit__() might do. In other words we could import some of the proposed functionality of the 'with' keyword (PEP 310) into 'for'. I think it makes sense because 'for' is already defined in term of implicit calls to next(), the only method of iterators; so if a second method is added, 'for' can be taught about it too. Armin From pedronis at bluewin.ch Sat Sep 13 10:18:51 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Sat Sep 13 10:17:11 2003 Subject: [Python-Dev] pre-PEP: Resource-Release Support for Generators In-Reply-To: <20030913134653.GA12752@vicky.ecs.soton.ac.uk> References: <200308262050.h7QKoqD04780@12-236-84-31.client.attbi.com> <5.2.1.1.0.20030826151628.023f8d68@pop.bluewin.ch> <00a401c36c0f$76952160$c304a044@oemcomputer> <200308262050.h7QKoqD04780@12-236-84-31.client.attbi.com> Message-ID: <5.2.1.1.0.20030913160918.02773bc0@pop.bluewin.ch> At 14:46 13.09.2003 +0100, Armin Rigo wrote: >Hello, > >(My reply didn't seem to have reached python-dev... second try.) > >On Tue, Aug 26, 2003 at 01:50:52PM -0700, Guido van Rossum wrote: > > Another comment on Samuele's PEP: It is sort of sad that the *user* of > > a generator has to know that the generator's close() must be called. > > Normally, the beauty of using a try/finally for cleanup is that your > > callers don't need to know about it. But I see no way around this. > >What about letting the 'for' handle this? It's the most common way generators >are used. When a 'for' loop on a generator-iterator finishes it would call the >close() method of the iterator, which on generators would (say) simulate a >return from the latest 'yield'. > >Well, close() might not be such a good name because it would probably break >exsiting code (e.g. closing files unexpectedly), but __exit__() might do. In >other words we could import some of the proposed functionality of the 'with' >keyword (PEP 310) into 'for'. I think it makes sense because 'for' is already >defined in term of implicit calls to next(), the only method of iterators; so >if a second method is added, 'for' can be taught about it too. I expect generators to grow also a __exit__ if they grow a close and PEP 310 is accepted, this idiom will then be common with g = gen() for v in g: ... but it will be common for files and file-like too. It can be reasonable to discuss whether we want a special syntax then that conflates 'with' and 'for' behavior. But we can experiment a while before seeing whether writing the idiom is unbearably tedious ;). OTOH conflating 'with' and 'for' just for generators seems a rather ad-hoc breaking of orthoganility of the two, you could not write anymore code like this: g = gen() for v in g: ... do something up to a point ... ... for v in g: ... now this is rare but still breaking orthoganility of primitives is something I would think twice about. regards. From pedronis at bluewin.ch Sat Sep 13 10:23:20 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Sat Sep 13 10:21:40 2003 Subject: [Python-Dev] pre-PEP: Resource-Release Support for Generators In-Reply-To: <5.2.1.1.0.20030913160918.02773bc0@pop.bluewin.ch> References: <20030913134653.GA12752@vicky.ecs.soton.ac.uk> <200308262050.h7QKoqD04780@12-236-84-31.client.attbi.com> <5.2.1.1.0.20030826151628.023f8d68@pop.bluewin.ch> <00a401c36c0f$76952160$c304a044@oemcomputer> <200308262050.h7QKoqD04780@12-236-84-31.client.attbi.com> Message-ID: <5.2.1.1.0.20030913162229.026d3248@pop.bluewin.ch> > > >OTOH conflating 'with' and 'for' just for generators seems a rather ad-hoc >breaking of >orthoganility of the two, you could not write anymore code like this: sorry, hmm, orthogonality that I know what is is. From worktogether at runbox.com Sat Sep 13 18:42:54 2003 From: worktogether at runbox.com (Dr. Newell) Date: Sat Sep 13 18:42:52 2003 Subject: [Python-Dev] To You as a Nicer Person Message-ID: <417-220039613224254410@runbox.com> Dear python-dev: You may ask yourself > How to make people like you and respect you > How to win friends > How to let your conduct help your health, work, job, career, success, relationships, spirit, mind, well-being, ... > How to make your life smoother and happier > How to do whatever you like without being unpleasant to other people > How to develop good conduct in your children or students > How to make the world peaceful and better You can find all the answers to these questions, and much more, in this great handbook: " Complete Conduct Principles for the 21st Century " by Dr. John Newton It is the best educational GIFT idea for children, friends, relatives, classmates, students, parents, teachers, other educators, ..., particularly at this special time. BENEFITS to each individual reader: Many! -- such as for health, work, job, career, success, self-improvement, education, relationships, spirit, mind, well-being, and much more -- almost all the areas that are important to you in the 21st century. People around you will benefit, too. (Please see the preface of the book for details.) EVERYONE may find this handbook useful and helpful, regardless of age (from children to oldsters), occupation, rank, status, gender, religious belief, race, nationality, country, or region. If you are a parent or a teacher, you can learn how to develop good conduct in your children or students from this handbook. Please advise your children or students to read the book. It will result in great benefits for both you and them. This book is a must for EVERYONE to be better prepared for personal conduct for the rest of the 21st century. The book's content is obvious from its title. The complete useful conduct principles cover not only what we should do, but also what we should not do -- especially those faults people make often and easily. This timely, unique, and very important handbook is designed to suit most people, and is self-contained and user-friendly. The book was also praised as "a compendium of concisely expressed, practical, informative, pertinent, workable advice" by Michael J. Carson, a professional book reviewer. This book is significantly different and better than competitive works. "Unlike most books of this subject, it is NOT a religious book, nor is a collection of old conduct rules." Some of its innovative contents may help solve problems that the Western culture cannot. The book's merit and importance have been recognized and praised by many experts, elected public officials, and world leaders. As a result of popular demand, the book has been a top "Featured Item" in some top on-line bookstores in some subjects, such as "Personal Practical Guides" in "Nonfiction" and "Reference", "Conduct" in "Health, Mind & Body", "Life skills guides", "Ethics & Moral Philosophy", ... "The book will also be effective for violence prevention for the whole society." said some experts. How to make the world peaceful and better --- You can find the solution in the book. Let's work together to make the world peaceful and better! The author, John Newton, holds a Ph.D. from MIT, and does researches at Harvard. His long-term research on "The personal conduct in the human society of the 21st century" resulted in this book. Before the human beings went into the 21st century, the compassionate, merciful, courageous and farsighted Dr. Newton had issued a number of warning predictions, some of which have already been proved in the new century. The book is published by NCWO, headquartered beside Harvard University and MIT, two leading institutes of new knowledge and literature. The publisher has a deep commitment to publishing only the best. NCWO is an educational, non-profit, non-partisan, and honorary organization; it endeavors to make the 21st century nicer than ever before. To accomplish its mission, NCWO is proud to introduce this book. The Web site of NCWO has been chosen as one of "Top Non Profit Sites Chosen by Type Non Profit Editors". The book is available in two types of binding: Hardcover (ISBN 0967370574; case bound, Smyth sewn; with dust jacket) and Paperback (ISBN 0967370582; perfect bound). Both editions are unabridged, and are printed on 60 lb, natural, acid-free, excellent and healthful paper. You can get the book from many fine on-line bookstores and traditional bookstores. For your convenience, (if you wish for an Internet link; otherwise you may skip this section) I herewith provide you with a link directly to the book page of each edition in Half.com by eBay, a popular on-line discount mall: for paperback: http://half.ebay.com/cat/buy/prod.cgi?cpid=2425993 for hardcover: http://half.ebay.com/cat/buy/prod.cgi?cpid=2425992 Please forward this e-mail to people you know -- children, friends, relatives, classmates, students, parents, teachers, other educators, ..., because they can benefit from it, too. This can be a wonderful kindness you provide to them! Sincerely yours, (python-dev, best wishes to you!) Christopher Newell, Ph.D. Cambridge, Massachusetts, USA P.S. Some educational units, ranging from the level of nation or state to individual school or university, have ordered the book as textbook, reference book, gift to students, or as an active action to prevent school violence, to improve education and to benefit students, teachers & parents. To have more people benefit from the book, please consider suggesting to the schools -- your children attend, or you yourself attend, have attended before, teach at, or serve -- that the book be used for fundraising for the schools. The book is an ideal fundraising tool. For example, it may used as a premium or a re-sale product for the fundraising. The successful fundraising will significantly help school education. Better yet, each supporter and his/her family will benefit from the book. Suggesting to the parent-teacher associations (organizations) (PTA/PTO) of the schools is also a good idea. From raymond.hettinger at verizon.net Sat Sep 13 21:05:18 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sat Sep 13 21:27:13 2003 Subject: [Python-Dev] SF #805304: bug or feature? Message-ID: <000201c37a5e$aa6b9e80$ed24c797@oemcomputer> Fred reported that super objects don't respond well to language constructs that bypass attribute lookup. For example, supobj.__setitem__(name,value) works if __setitem__() is defined in the target, but supobj[name]=value will raise a TypeError. www.python.org/sf/805304 Once the fix is approved, there is a question of whether it should be backported. The case against it is that some programs written under Py2.3.1 or Py2.2.4 won't run on Py2.3.0 or Py2.2.3 and the patch can be viewed as an API expansion. The other point of view is that super objects should have always behaved this way and that the patch just fixes buggy behavior. Raymond From skip at mojam.com Sun Sep 14 08:00:34 2003 From: skip at mojam.com (Skip Montanaro) Date: Sun Sep 14 08:00:43 2003 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200309141200.h8EC0Yi25498@manatee.mojam.com> Bug/Patch Summary ----------------- 475 open / 4113 total bugs (+10) 193 open / 2371 total patches (+8) New Bugs -------- cgi.CGIHTTPRequestHandler remembers QUERY_STRING (2003-09-07) http://python.org/sf/801992 Mode argument of dumbdbm does not work (2003-09-07) http://python.org/sf/802128 tkFont may reuse font names (2003-09-08) http://python.org/sf/802310 Bogus use of Tkinter.PhotoImage can give sig11 (2003-09-08) http://python.org/sf/802417 uu.decode prints to stderr (2003-09-09) http://python.org/sf/803413 sgmllib doesn't support hex or Unicode character references (2003-09-09) http://python.org/sf/803422 plat-mac/applesingle.py needs cosmetic changes (2003-09-09) http://python.org/sf/803498 urllib/urllib2(?) timeouts (2003-09-10) http://python.org/sf/803634 Crypto terminology for crypto hash function (2003-09-10) http://python.org/sf/804113 invalid use of setlocale (2003-09-11) http://python.org/sf/804543 PyUnicode_FromEncodedObject (2003-09-12) http://python.org/sf/805015 Inappropriate error received using socket timeout (2003-09-12) http://python.org/sf/805194 _tkinter compilation fails (2003-09-12) http://python.org/sf/805200 super instances don't support item assignment (2003-09-12) http://python.org/sf/805304 IDLE can't save UTF-8 files on Windows (2003-09-13) http://python.org/sf/805728 Minor Python Intro update (2003-09-13) http://python.org/sf/805788 New Patches ----------- Suggested change in how ntpath.expanduser works (2003-09-07) http://python.org/sf/802159 better parser error message for non-EOL following line cont. (2003-09-07) http://python.org/sf/802188 [_ssl.c] SSL_write() called with -1 as size (2003-09-10) http://python.org/sf/803998 A ForwardingHandler for logging (2003-09-10) http://python.org/sf/804180 unittest's main don't handle TestSuite in command line (2003-09-10) http://python.org/sf/804212 Lib/email/Encoders.py iso-2022 is 7bit (2003-09-11) http://python.org/sf/804885 fix --without-threads (2003-09-13) http://python.org/sf/805604 PTH related fixes (2003-09-13) http://python.org/sf/805613 make test_fcntl 64bit clean (2003-09-13) http://python.org/sf/805626 Fix linking with shared libpython (2003-09-13) http://python.org/sf/805678 Highlight builtins (2003-09-13) http://python.org/sf/805830 Closed Bugs ----------- Add docs for 'basestring' (2002-05-24) http://python.org/sf/560286 htmllib.HTMLParser.anchorlist problem (2003-03-28) http://python.org/sf/711632 2 problems with IDLE (2003-07-21) http://python.org/sf/775061 RESET_ERROR is not defined(logging module) (2003-08-21) http://python.org/sf/792649 select module doesn't allow any iterable. (2003-08-31) http://python.org/sf/798046 test_dumbdbm failing (2003-09-04) http://python.org/sf/800824 Making "|" directive from REs a bit clearer (2003-09-04) http://python.org/sf/800899 Bad RE in scanf example (2003-09-05) http://python.org/sf/801306 Closed Patches -------------- Allow os.access to handle Unicode file name (2003-08-17) http://python.org/sf/790000 implement htmllib.HTMLParser.reset (2003-08-22) http://python.org/sf/793021 zipimport 64-bit fix (2003-09-06) http://python.org/sf/801821 From martin at v.loewis.de Sun Sep 14 18:08:57 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sun Sep 14 18:09:10 2003 Subject: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: <007c01c379be$40efa460$b439c797@oemcomputer> References: <001d01c3795d$c2ba1400$30bc958d@oemcomputer> <1063399454.2069.299.camel@localhost.localdomain> <008a01c3798a$f7198b40$b439c797@oemcomputer> <007c01c379be$40efa460$b439c797@oemcomputer> Message-ID: "Raymond Hettinger" writes: > In this particular case, we are (IMHO) much better off adding an > iterable interface than leaving current situation with "db.first()" > and "while 1: db.next()" wrapped in a try/except for an > undocumented exception class -- yuck. As nobody has protested against this specific change (yet?), I guess you can go ahead and apply it. In doing so, you have to accept the burden of taking blame in case this turns out to be a bad idea. > Worse still, the docs for Py2.3.0 cannot be changed and they are > wrong on several counts (the methods return items instead of keys; > EOF is signaled by an exception instead of None; and the example > doesn't run). That would be a documentation bug. However, documentation bugs are generally best fixed by correcting the documentation, not by adjusting the code to match the documentation (atleast, for maintenance releases). Regards, Martin From martin at v.loewis.de Sun Sep 14 18:12:42 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sun Sep 14 18:12:42 2003 Subject: [Python-Dev] SF #805304: bug or feature? In-Reply-To: <000201c37a5e$aa6b9e80$ed24c797@oemcomputer> References: <000201c37a5e$aa6b9e80$ed24c797@oemcomputer> Message-ID: "Raymond Hettinger" writes: > Once the fix is approved, there is a question of whether it should > be backported. The case against it is that some programs written > under Py2.3.1 or Py2.2.4 won't run on Py2.3.0 or Py2.2.3 and the > patch can be viewed as an API expansion. The other point of view > is that super objects should have always behaved this way and that > the patch just fixes buggy behavior. Whether it is a bug or not can be best clarified by looking at the documentation: If it is documented that attributes of super objects always follow MRO, then there is a bug. That might be a doc bug, but Special cases aren't special enough to break the rules. makes clear it wouldn't be a doc bug, but a bug in the code. OTOH, if this is not documented at all, you clearly have a doc bug. Regards, Martin From jeremy at alum.mit.edu Sun Sep 14 18:21:15 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Sun Sep 14 18:21:33 2003 Subject: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: References: <001d01c3795d$c2ba1400$30bc958d@oemcomputer> <1063399454.2069.299.camel@localhost.localdomain> <008a01c3798a$f7198b40$b439c797@oemcomputer> <007c01c379be$40efa460$b439c797@oemcomputer> Message-ID: <1063578075.2093.67.camel@localhost.localdomain> On Sun, 2003-09-14 at 18:08, Martin v. L?wis wrote: > "Raymond Hettinger" writes: > > > In this particular case, we are (IMHO) much better off adding an > > iterable interface than leaving current situation with "db.first()" > > and "while 1: db.next()" wrapped in a try/except for an > > undocumented exception class -- yuck. > > As nobody has protested against this specific change (yet?), I guess > you can go ahead and apply it. In doing so, you have to accept the > burden of taking blame in case this turns out to be a bad idea. Perhaps we could wait to hear from Jack or Just or someone who was concerned about feature creep in maintenance releases. I think we should decide now whether to follow the policy they propose or we should drop it. Right now, it feels like it's in limbo. Jeremy From tim.one at comcast.net Sun Sep 14 18:36:09 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Sep 14 18:37:57 2003 Subject: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: <1063578075.2093.67.camel@localhost.localdomain> Message-ID: [Jeremy] > Perhaps we could wait to hear from Jack or Just or someone who was > concerned about feature creep in maintenance releases. I think we > should decide now whether to follow the policy they propose or we > should drop it. Right now, it feels like it's in limbo. Guido needs to set the direction here. Slamming new features into micro releases became irresistible after the 1.5.2 experience, where a very long time passed without a new Python release (not even a bugfix release). That naturally caused people to want to get every improvement in ASAP. We were able to counteract that successfully in the early days of PythonLabs, because then we pumped out two minor releases per year, and it was much easier to sell "leave the micro release a pure bugfix release -- the next minor release is only a few months away". Then 2.3 stretched out over 18 months, and there's no plan for 2.4 in evidence yet. Given this most recent history, and the unlikelihood that PythonLabs's (or a workalike equivalent's) early days will repeat itself soon, if I had a minor feature I really wanted to get in, I'd try to get it into a 2.3 micro release. OTOH, if Guido decides to go back to a schedule-driven release for 2.4, and it's not terribly far in the future, then it will again be much easier to sell 2.3 micro releases as bugfix-only (and stick to that). We're all pretty creative about what we'll call "a bug", when it suits a feature we believe in <0.7 wink>. From niemeyer at conectiva.com Sun Sep 14 20:06:37 2003 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Sun Sep 14 20:07:14 2003 Subject: [Python-Dev] datetime issues Message-ID: <20030915000637.GC27698@ibook> Greetings! Is there any reason for not having a single keyword parameter for every function accepting a tzinfo instance? I've noticed that some require 'tz', and others 'tzinfo'. Is it too late to fix that? Also, is there any further work going on to improve the datetime module? I'm not suggesting we should mirror mx.DateTime functionalities, but two features I miss from mx.DateTime is the DateFrom(), which does its best to parse a given string, and the RelativeDateTime() functionality. This has probably been raised in the past, so if that's the case, I'm sorry. Marc-Andre, what's your opinion about reusing code from mx.DateTime into datetime? Anyway, reusing code or not, I'll probably put sometime on it in the future, if this looks interesting to everyone (after I finish my current python pendencies, like SRE's recursivity removal). -- Gustavo Niemeyer http://niemeyer.net From tim.one at comcast.net Sun Sep 14 20:35:35 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Sep 14 20:37:26 2003 Subject: [Python-Dev] datetime issues In-Reply-To: <20030915000637.GC27698@ibook> Message-ID: [Gustavo Niemeyer] > Is there any reason for not having a single keyword parameter for > every function accepting a tzinfo instance? Just the lack of anyone pointing it out in the module's 18-month review process . > I've noticed that some require 'tz', and others 'tzinfo'. Is it too > late to fix that? Of course, although it's not too late to add synonyms. It appears that the functions "with a lot of arguments" use tzinfo, and those with only one or two arguments tz now. I doubt anyone would bother to use the tz keyword-arg name for a few-argument function (now(), fromtimestamp(), astimezone()), so I'd rather that those few grow a tzinfo synonym. tz can be deprecated too, if you're bold. > Also, is there any further work going on to improve the datetime > module? Not that I'm aware of. I'm not working on it anymore (other than keeping my out for bug reports, of which there have been blessedly few). > I'm not suggesting we should mirror mx.DateTime functionalities, but > two features I miss from mx.DateTime is the DateFrom(), which does its > best to parse a given string, and the RelativeDateTime() functionality. > > This has probably been raised in the past, so if that's the case, I'm > sorry. Marc-Andre, what's your opinion about reusing code from > mx.DateTime into datetime? > > Anyway, reusing code or not, I'll probably put sometime on it in the > future, if this looks interesting to everyone (after I finish my > current python pendencies, like SRE's recursivity removal). If Guido wanted to stay out of the relatively clear time zone business, I can imagine his interest in trying to guess how to parse date and time strings. Apart from Brett's strptime(), I believe the email module has some useful string->datetime code. It may be good to fold such stuff in this area as Python already supports into datetime somehow. From arigo at tunes.org Mon Sep 15 06:48:36 2003 From: arigo at tunes.org (Armin Rigo) Date: Mon Sep 15 06:50:28 2003 Subject: [Python-Dev] pre-PEP: Resource-Release Support for Generators In-Reply-To: <5.2.1.1.0.20030913160918.02773bc0@pop.bluewin.ch> References: <200308262050.h7QKoqD04780@12-236-84-31.client.attbi.com> <5.2.1.1.0.20030826151628.023f8d68@pop.bluewin.ch> <00a401c36c0f$76952160$c304a044@oemcomputer> <200308262050.h7QKoqD04780@12-236-84-31.client.attbi.com> <5.2.1.1.0.20030913160918.02773bc0@pop.bluewin.ch> Message-ID: <20030915104836.GA1143@vicky.ecs.soton.ac.uk> Hello Samuele, On Sat, Sep 13, 2003 at 04:18:51PM +0200, Samuele Pedroni wrote: > OTOH conflating 'with' and 'for' just for generators seems a rather ad-hoc > breaking of > orthoganility of the two, you could not write anymore code like this: > > g = gen() > for v in g: > ... do something up to a point ... > ... > for v in g: > ... I had thought about this. This occurs when you 'break' out of the first loop. I would say that NOT calling the __exit__() method in this specific case seems quite intuitive, the 'break' meaning 'just exit from the loop now without any further processing, skipping the 'else' part if present'. Armin From sholden at holdenweb.com Mon Sep 15 07:59:36 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Sep 15 08:03:00 2003 Subject: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: Message-ID: > -----Original Message----- > From: python-dev-bounces@python.org > [mailto:python-dev-bounces@python.org]On Behalf Of Tim Peters > Sent: Sunday, September 14, 2003 6:36 PM > To: jeremy@alum.mit.edu; Martin v. L?wis > Cc: Raymond Hettinger; python-dev@python.org > Subject: RE: [Python-Dev] python/dist/src/Lib/bsddb > __init__.py,1.5,1.6 > > > [Jeremy] > > Perhaps we could wait to hear from Jack or Just or someone who was > > concerned about feature creep in maintenance releases. I think we > > should decide now whether to follow the policy they propose or we > > should drop it. Right now, it feels like it's in limbo. > > Guido needs to set the direction here. Slamming new features > into micro > releases became irresistible after the 1.5.2 experience, > where a very long > time passed without a new Python release (not even a bugfix > release). That > naturally caused people to want to get every improvement in ASAP. > > We were able to counteract that successfully in the early days of > PythonLabs, because then we pumped out two minor releases per > year, and it > was much easier to sell "leave the micro release a pure > bugfix release -- > the next minor release is only a few months away". > > Then 2.3 stretched out over 18 months, and there's no plan for 2.4 in > evidence yet. Given this most recent history, and the > unlikelihood that > PythonLabs's (or a workalike equivalent's) early days will > repeat itself > soon, if I had a minor feature I really wanted to get in, I'd > try to get it > into a 2.3 micro release. > > OTOH, if Guido decides to go back to a schedule-driven > release for 2.4, and > it's not terribly far in the future, then it will again be > much easier to > sell 2.3 micro releases as bugfix-only (and stick to that). > > We're all pretty creative about what we'll call "a bug", when > it suits a > feature we believe in <0.7 wink>. > So, what we need is a plan for a 2.4 release, then. How soon would it need to be, though? And who can commit time to making it happen. Seems to me that some formerly active core developers are currently busy in other realms. Is it feasible to have a 2.4 plan, or are we ready to contemplate (gasp) 3.0? What help can such as I, not a practiced C coder and fairly heavily committed and thinly spread, be? regards -- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ From mcherm at mcherm.com Mon Sep 15 08:04:51 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Mon Sep 15 08:04:56 2003 Subject: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6 Message-ID: <1063627491.3f65aae37a3b7@mcherm.com> [Jeremy] > Isn't this just the sort of little feature that was the subject of > recent discussion on the dangers of backporting. It would mean that > someone writing an app against Python 2.3 could never be sure whether > the feature existed. In practice, that would mean developers using > 2.3.x would only find out about problems after deployment when they bump > into a user with the original 2.3. > > I'm not sure how convincing I find this argument, but it's got some > merit. [Martin] > There is no proper balance. If you believe in the policy "no new > features", this policy is absolute - it would be pointless to accept > some new features, but reject others on grounds of the policy. I don't really understand this absolutist position. Suppose (hypothetical example made extreme for retorical purposes) we discover that there was a HUGE bug in release 2.3. Using a variable named "Q" as the argument of a for loop causes the interpreter to seg-fault. This is CLEARLY a bad thing... capital one-letter variable names are uncommon, but not THAT uncommon! So I suppose that 2.3.1 would need to include a fix for this bug. But THEN anyone writing code for 2.3.x could never be SURE whether the bug existed. If they used a capital Q in a for loop, their code would eventually be run by somebody with 2.3.0, causing a crash. The only way to be safe would be to test their code on 2.3.0. How is this "bug fix" any different from a "new feature"? I understand the argument "avoid new features in maintanence releases because there we value stability FAR more than we value features". I understand the argument "avoid new features in maintanence releases because features should be added on the main trunk to encourage the (eventual) use of new versions of Python". But I don't understand the argument "PROHIBIT new features in maintanence releases because then programs written under the maintanence release might not run under the original release", because to me it seems to be logically equivalent to "prohibit maintanence releases". [Martin] > This policy is mutually exclusive with "new features are fine as long > as they don't break anything". You can get opinions, but you cannot > get consensus. If "consensus" is taken to mean "universal agreement", then I'd agree, but if we mean "general agreement or accord" (ie, everyone gets heard then NEARLY everyone agrees and the naysayers decide they can live with it), then we might have a shot. I'd certainly say (and I think most here would agree) that even ONE seriously intended firmly held argument against a particular new feature (in a maint. release) would make me think VERY carefully before allowing that feature. -- Michael Chermside From amk at amk.ca Mon Sep 15 08:42:58 2003 From: amk at amk.ca (A.M. Kuchling) Date: Mon Sep 15 08:43:06 2003 Subject: [Python-Dev] Re: datetime issues References: <20030915000637.GC27698@ibook> Message-ID: On Sun, 14 Sep 2003 20:35:35 -0400, Tim Peters wrote: > If Guido wanted to stay out of the relatively clear time zone > business, I can imagine his interest in trying to guess how to parse date > and time strings. Apart from Brett's strptime(), I believe the email > module > has some useful string->datetime code. The PyXML package includes a parser (xml.utils.iso8601) for the date/time format most commonly used in XML applications. That and a parser for RFC822 dates would meet the needs of many technical applications (though not ones where users can enter dates -- but that problem may be too complicated for the stdlib). I'll happily convert the iso8601 module for addition to the standard library if date parsing is deemed a topic of interest. --amk From amk at amk.ca Mon Sep 15 08:51:02 2003 From: amk at amk.ca (A.M. Kuchling) Date: Mon Sep 15 08:51:19 2003 Subject: [Python-Dev] Re: python/dist/src/Lib/bsddb __init__.py,1.5,1.6 References: Message-ID: On Mon, 15 Sep 2003 07:59:36 -0400, Steve Holden wrote: > other realms. Is it feasible to have a 2.4 plan, or are we ready to > contemplate (gasp) 3.0? I'd be in favor of setting an arbitrary date for a 2.4 release, and when that date rolls around stabilizing and releasing the CVS tree, no matter what's been implemented or not implemented. If we've made major language revisions by that time, fine, but it's worthwhile even if 2.4 is just 2.3 + (various bugfixes deemed too risky for 2.3.1) + (a few new modules). PEP 320 estimates February 2005, a 19-month gap matching that between 2.2 and 2.3. That seems a bit too long, especially if there are no large changes that require testing; aiming for autumn 2004, beginning to freeze in August, seems reasonable. --amk From barry at python.org Mon Sep 15 09:10:22 2003 From: barry at python.org (Barry Warsaw) Date: Mon Sep 15 09:10:33 2003 Subject: [Python-Dev] Re: python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: References: Message-ID: <1063631421.10716.2.camel@anthem> On Mon, 2003-09-15 at 08:51, A.M. Kuchling wrote: > I'd be in favor of setting an arbitrary date for a 2.4 release, and when > that date rolls around stabilizing and releasing the CVS tree, no matter > what's been implemented or not implemented. If we've made major language > revisions by that time, fine, but it's worthwhile even if 2.4 is just 2.3 + > (various bugfixes deemed too risky for 2.3.1) + (a few new modules). > > PEP 320 estimates February 2005, a 19-month gap matching that between 2.2 > and 2.3. That seems a bit too long, especially if there are no large > changes that require testing; aiming for autumn 2004, beginning to freeze > in August, seems reasonable. +1 -Barry From barry at python.org Mon Sep 15 09:14:17 2003 From: barry at python.org (Barry Warsaw) Date: Mon Sep 15 09:14:22 2003 Subject: [Python-Dev] Re: datetime issues In-Reply-To: References: <20030915000637.GC27698@ibook> Message-ID: <1063631656.10716.7.camel@anthem> On Mon, 2003-09-15 at 08:42, A.M. Kuchling wrote: > I believe the email > > module > > has some useful string->datetime code. > > The PyXML package includes a parser (xml.utils.iso8601) for the date/time > format most commonly used in XML applications. That and a parser for > RFC822 dates would meet the needs of many technical applications (though > not ones where users can enter dates -- but that problem may be too > complicated for the stdlib). I'll happily convert the iso8601 module for > addition to the standard library if date parsing is deemed a topic of > interest. Yep, I think email._parseaddr.parsedate{,_tz}() is an RFC 2822 compliant date parser. Note that there's also Unix-from dates which are slightly different. I'd also like to see factored out various date formating functions, so that the module is more or less symmetric. -Barry From nas-python at python.ca Mon Sep 15 10:47:12 2003 From: nas-python at python.ca (Neil Schemenauer) Date: Mon Sep 15 10:39:57 2003 Subject: [Python-Dev] Re: python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: References: Message-ID: <20030915144712.GA7448@mems-exchange.org> On Mon, Sep 15, 2003 at 08:51:02AM -0400, A.M. Kuchling wrote: > I'd be in favor of setting an arbitrary date for a 2.4 release, and when > that date rolls around stabilizing and releasing the CVS tree, no matter > what's been implemented or not implemented. +1 From tim.one at comcast.net Mon Sep 15 10:44:07 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Sep 15 10:44:18 2003 Subject: [Python-Dev] Re: python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: Message-ID: [Andrew Kuchling] > I'd be in favor of setting an arbitrary date for a 2.4 release, and > when that date rolls around stabilizing and releasing the CVS tree, > no matter what's been implemented or not implemented. If we've made > major language revisions by that time, fine, but it's worthwhile even > if 2.4 is just 2.3 + (various bugfixes deemed too risky for 2.3.1) + > (a few new modules). > > PEP 320 estimates February 2005, a 19-month gap matching that between > 2.2 and 2.3. That seems a bit too long, especially if there are no > large changes that require testing; aiming for autumn 2004, beginning > to freeze in August, seems reasonable. I'd like to see 2.4 released no later than 29-Jul-2004. That's a year after 2.3 got released. I don't have nearly as many years left to me as most of you do . From barry at python.org Mon Sep 15 10:55:31 2003 From: barry at python.org (Barry Warsaw) Date: Mon Sep 15 10:55:37 2003 Subject: [Python-Dev] Re: python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: References: Message-ID: <1063637731.10716.25.camel@anthem> On Mon, 2003-09-15 at 10:44, Tim Peters wrote: > I'd like to see 2.4 released no later than 29-Jul-2004. I like the tradition this sets. I can't think of a better birthday present to one of the best Canadian bassists in the world. -Barry From jeremy at alum.mit.edu Mon Sep 15 10:57:02 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Mon Sep 15 10:56:56 2003 Subject: [Python-Dev] Re: python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: References: Message-ID: <1063637821.2093.132.camel@localhost.localdomain> On Mon, 2003-09-15 at 08:51, A.M. Kuchling wrote: > I'd be in favor of setting an arbitrary date for a 2.4 release, and when > that date rolls around stabilizing and releasing the CVS tree, no matter > what's been implemented or not implemented. If we've made major language > revisions by that time, fine, but it's worthwhile even if 2.4 is just 2.3 + > (various bugfixes deemed too risky for 2.3.1) + (a few new modules). > > PEP 320 estimates February 2005, a 19-month gap matching that between 2.2 > and 2.3. That seems a bit too long, especially if there are no large > changes that require testing; aiming for autumn 2004, beginning to freeze > in August, seems reasonable. In the absence of any plans for major new features, I think this is a good plan. A rough schedule might be: 2.4a1 April 2.4a2 late May/early June 2.4b1 July 9 2.4b2 Aug. 6 2.4 f Sep. 3 We should also make a plan for a 2.3 maintenance release. There have been a lot of checkins, but I haven't gotten a sense for how significant the bugs were. Jeremy From neal at metaslash.com Mon Sep 15 11:03:13 2003 From: neal at metaslash.com (Neal Norwitz) Date: Mon Sep 15 11:03:21 2003 Subject: [Python-Dev] Re: python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: <1063637821.2093.132.camel@localhost.localdomain> References: <1063637821.2093.132.camel@localhost.localdomain> Message-ID: <20030915150313.GI1240@epoch.metaslash.com> On Mon, Sep 15, 2003 at 10:57:02AM -0400, Jeremy Hylton wrote: > > We should also make a plan for a 2.3 maintenance release. There have > been a lot of checkins, but I haven't gotten a sense for how significant > the bugs were. I think 2.3 has been great! There were a few memory leaks plugged. I don't even recall any crashes specific to 2.3. I'm very happy with the apparent robustness of 2.3. Maybe nobody is using it? :-) As far as the release schedule for 2.4, I haven't heard any squawking recently about Python changing too fast. But then again, I have no time to read c.l.p anymore. Maybe there's a correlation? :-) Neal From barry at python.org Mon Sep 15 11:08:37 2003 From: barry at python.org (Barry Warsaw) Date: Mon Sep 15 11:08:43 2003 Subject: [Python-Dev] Re: python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: <20030915150313.GI1240@epoch.metaslash.com> References: <1063637821.2093.132.camel@localhost.localdomain> <20030915150313.GI1240@epoch.metaslash.com> Message-ID: <1063638517.10716.32.camel@anthem> On Mon, 2003-09-15 at 11:03, Neal Norwitz wrote: > I think 2.3 has been great! There were a few memory leaks plugged. > I don't even recall any crashes specific to 2.3. I'm very happy with > the apparent robustness of 2.3. Maybe nobody is using it? :-) I'm using it for everything, and I agree! It's a very solid release. I still think we should plan for a 2.3.1 soonish, but the hold up may just be finding a release manager to own it. I'd rather spread the love this time. :) > As far as the release schedule for 2.4, I haven't heard any squawking > recently about Python changing too fast. We can't let that stand, now can we? :) Maybe Guido can post about his big plans for 2.4. There's still time to get PEPpy too. -Barry From anthony at interlink.com.au Mon Sep 15 11:11:35 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Sep 15 11:13:01 2003 Subject: [Python-Dev] Re: python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: <1063638517.10716.32.camel@anthem> Message-ID: <200309151511.h8FFBZ0j029922@localhost.localdomain> >>> Barry Warsaw wrote > I'm using it for everything, and I agree! It's a very solid release. I > still think we should plan for a 2.3.1 soonish, but the hold up may just > be finding a release manager to own it. I'd rather spread the love this > time. :) I've already put my hand up for it, and Raymond's offered to help. I'm thinking about maybe starting this process late next week? (I've already done at least one minor release before, so I've a fair idea of what's involved. Hey, hang on, didn't I write PEP102 for this? ;) Anthony -- Anthony Baxter It's never too late to have a happy childhood. From pje at telecommunity.com Mon Sep 15 11:26:27 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Sep 15 11:26:41 2003 Subject: [Python-Dev] pre-PEP: Resource-Release Support for Generators In-Reply-To: <20030915104836.GA1143@vicky.ecs.soton.ac.uk> References: <5.2.1.1.0.20030913160918.02773bc0@pop.bluewin.ch> <200308262050.h7QKoqD04780@12-236-84-31.client.attbi.com> <5.2.1.1.0.20030826151628.023f8d68@pop.bluewin.ch> <00a401c36c0f$76952160$c304a044@oemcomputer> <200308262050.h7QKoqD04780@12-236-84-31.client.attbi.com> <5.2.1.1.0.20030913160918.02773bc0@pop.bluewin.ch> Message-ID: <5.1.1.6.0.20030915112135.03013ec0@telecommunity.com> At 11:48 AM 9/15/03 +0100, Armin Rigo wrote: >Hello Samuele, > >On Sat, Sep 13, 2003 at 04:18:51PM +0200, Samuele Pedroni wrote: > > OTOH conflating 'with' and 'for' just for generators seems a rather ad-hoc > > breaking of > > orthoganility of the two, you could not write anymore code like this: > > > > g = gen() > > for v in g: > > ... do something up to a point ... > > ... > > for v in g: > > ... > >I had thought about this. This occurs when you 'break' out of the first loop. >I would say that NOT calling the __exit__() method in this specific case seems >quite intuitive, the 'break' meaning 'just exit from the loop now without any >further processing, skipping the 'else' part if present'. Hmmm... You realize this is also going to break the similarity between: i = iter(something) while 1: try: j=i.next() except StopIteration: break and for j in iter(something): pass The former is already complex enough as a mental model. I think it gets altogether too complex if one also has to consider the enter/leave issue. This strikes me as an area like try/except vs. try/finally: it really should be a separate block, just for the sake of explicitness. As much as it would be cool to have the automatic release, I think I'd rather use: with i = iter(something): for j in i: ... And make the resource management very visible. From skip at pobox.com Mon Sep 15 11:32:37 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Sep 15 11:32:49 2003 Subject: [Python-Dev] Re: python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: <1063638517.10716.32.camel@anthem> References: <1063637821.2093.132.camel@localhost.localdomain> <20030915150313.GI1240@epoch.metaslash.com> <1063638517.10716.32.camel@anthem> Message-ID: <16229.56213.235834.239788@montanaro.dyndns.org> Barry> On Mon, 2003-09-15 at 11:03, Neal Norwitz wrote: >> I think 2.3 has been great! There were a few memory leaks plugged. >> I don't even recall any crashes specific to 2.3. I'm very happy with >> the apparent robustness of 2.3. Maybe nobody is using it? :-) Barry> I'm using it for everything, and I agree! Ditto on my development machine, except I just cvs up periodically, so I'm generally running from CVS as of a few days ago, not even 2.3. I do intend to update to 2.3 on the Mojam webserver in a short while. That should provide a nice speed boost over the 2.2.3 I'm currently running there. >> As far as the release schedule for 2.4, I haven't heard any squawking >> recently about Python changing too fast. Barry> We can't let that stand, now can we? :) I suspect that's simply because the focus was on library and performance improvements instead of new language features. Barry> There's still time to get PEPpy too. I would like to get back to PEP 304 (Control of pyc file location). I got stuck when someone raised some good objections about how it would play on Windows with it's multiple rooted file system, but if those problems can be ironed out I think it would be a good addition to the system. Skip From niemeyer at conectiva.com Mon Sep 15 12:26:51 2003 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Mon Sep 15 12:27:32 2003 Subject: [Python-Dev] datetime issues In-Reply-To: References: <20030915000637.GC27698@ibook> Message-ID: <20030915162651.GA30600@ibook.distro.conectiva> > [Gustavo Niemeyer] > > Is there any reason for not having a single keyword parameter for > > every function accepting a tzinfo instance? > > Just the lack of anyone pointing it out in the module's 18-month review > process . Would you belive that I was.. erm.. sleeping!? :-) > > I've noticed that some require 'tz', and others 'tzinfo'. Is it too > > late to fix that? > > Of course, although it's not too late to add synonyms. It appears > that the functions "with a lot of arguments" use tzinfo, and those > with only one or two arguments tz now. I doubt anyone would bother to > use the tz keyword-arg name for a few-argument function (now(), > fromtimestamp(), astimezone()), so I'd rather that those few grow a > tzinfo synonym. tz can be deprecated too, if you're bold. It'd be nice to have these synonyms. I'll put it in my ever growing todo list. > > Also, is there any further work going on to improve the datetime > > module? > > Not that I'm aware of. I'm not working on it anymore (other than > keeping my out for bug reports, of which there have been blessedly > few). > > > I'm not suggesting we should mirror mx.DateTime functionalities, but > > two features I miss from mx.DateTime is the DateFrom(), which does > > its best to parse a given string, and the RelativeDateTime() > > functionality. > > > > This has probably been raised in the past, so if that's the case, I'm > > sorry. Marc-Andre, what's your opinion about reusing code from > > mx.DateTime into datetime? > > > > Anyway, reusing code or not, I'll probably put sometime on it in the > > future, if this looks interesting to everyone (after I finish my > > current python pendencies, like SRE's recursivity removal). > > If Guido wanted to stay out of the relatively clear time zone > business, I can imagine his interest in trying to guess how to parse > date and time strings. Apart from Brett's strptime(), I believe the > email module has some useful string->datetime code. It may be good to > fold such stuff in this area as Python already supports into datetime > somehow. Cool. I'll give it a try as well. I have a working relativedelta implementation now. I'll polish it a little further and publish it somewhere for tests. Thanks! -- Gustavo Niemeyer http://niemeyer.net From amk at amk.ca Mon Sep 15 12:29:11 2003 From: amk at amk.ca (A.M. Kuchling) Date: Mon Sep 15 12:29:18 2003 Subject: [Python-Dev] Re: datetime issues References: <20030915000637.GC27698@ibook> <1063631656.10716.7.camel@anthem> Message-ID: On 15 Sep 2003 09:14:17 -0400, Barry Warsaw wrote: > I'd also like to see factored out various date formating functions, so > that the module is more or less symmetric. Probably this is PEP territory. Unless someone else wants to do it, I'll volunteer to produce a draft PEP for adding parsing and formatting functions. --amk From amk at amk.ca Mon Sep 15 12:44:29 2003 From: amk at amk.ca (A.M. Kuchling) Date: Mon Sep 15 12:44:37 2003 Subject: [Python-Dev] Re: Re: python/dist/src/Lib/bsddb __init__.py,1.5,1.6 References: <1063637821.2093.132.camel@localhost.localdomain> Message-ID: On Mon, 15 Sep 2003 10:57:02 -0400, Jeremy Hylton wrote: > We should also make a plan for a 2.3 maintenance release. There have > been a lot of checkins, but I haven't gotten a sense for how significant > the bugs were. I watch the checkins for the sake of the 2.4 document, and haven't seen any bugfixes that seemed really critical. No one in c.l.py seems to be running into any of the memory leaks that were fixed, and none of the fixed bugs seem likely to silently lose or corrupt data. Is it worth holding 2.3.1 until after Panther ships, on the assumption that its release will turn up more users and more bugs? If not, 2.3.1's timing can probably be left up to the release manager, who can make the release whenever he feels like it. --amk From niemeyer at conectiva.com Mon Sep 15 12:50:26 2003 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Mon Sep 15 12:51:04 2003 Subject: [Python-Dev] Re: datetime issues In-Reply-To: References: <20030915000637.GC27698@ibook> Message-ID: <20030915165026.GB30600@ibook.distro.conectiva> > >If Guido wanted to stay out of the relatively clear time zone > >business, I can imagine his interest in trying to guess how to parse date > >and time strings. Apart from Brett's strptime(), I believe the email > >module > >has some useful string->datetime code. > > The PyXML package includes a parser (xml.utils.iso8601) for the date/time > format most commonly used in XML applications. That and a parser for Cool! Thanks for mentioning it! > RFC822 dates would meet the needs of many technical applications (though > not ones where users can enter dates -- but that problem may be too > complicated for the stdlib). I do not agree. I've done a small research and there seems to be many nice generic implementations of these parsers, which would benefit technical applications *and* hand made dates (some of them below[1]), and it doesn't look that complex. I'll be working to provide trial implementation collecting the best features described in these papers in the near future. Btw, the relativedelta is mostly ready. That's the first step to provide relative dates in the parser. > I'll happily convert the iso8601 module for addition to the standard > library if date parsing is deemed a topic of interest. I'm not yet sure if date parsing and other improvements in the date/time area is a topic of general interest, but at least I am interested. :-) [1] http://www.egenix.com/files/python/mxDateTime.html http://ringmaster.arc.nasa.gov/tools/time_formats.html http://www.thinkage.ca/english/gcos/expl/b/lib/0tosec.html http://search.cpan.org/author/MUIR/Time-modules-2003.0211/lib/Time/ParseDate.pm -- Gustavo Niemeyer http://niemeyer.net From niemeyer at conectiva.com Mon Sep 15 12:55:14 2003 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Mon Sep 15 12:55:51 2003 Subject: [Python-Dev] Re: datetime issues In-Reply-To: References: <20030915000637.GC27698@ibook> <1063631656.10716.7.camel@anthem> Message-ID: <20030915165514.GC30600@ibook.distro.conectiva> > >I'd also like to see factored out various date formating functions, so > >that the module is more or less symmetric. > > Probably this is PEP territory. Unless someone else wants to do it, > I'll volunteer to produce a draft PEP for adding parsing and > formatting functions. Please, don't discard the generic parser in the PEP. -- Gustavo Niemeyer http://niemeyer.net From mwh at python.net Mon Sep 15 14:01:37 2003 From: mwh at python.net (Michael Hudson) Date: Mon Sep 15 14:03:04 2003 Subject: [Python-Dev] refleak status Message-ID: After a week or so of frantically searching for these things, I sort of forgot about the reference leak hunt I started in Python's tests. A week ago I ran my hacked regrtest.py again, made some notes, tried & failed to send them to python-dev, went on holiday, came back, digged a little more and am trying to send again... real leaks i think i've fixed (am syncing up to CVS now, will test overnight). test_class leaked [66, 66, 66, 66] references test_hotshot leaked [111, 111, 111, 111] references real leaks i cant fix: test_compile leaked [45, 45, 45, 45] references lambda a,a:0 worst offender. confused. red herrings: test_codeccallbacks leaked [2, 2, 2, 2] references callback registry test_mutants leaked [-1489, 1520, -755, -183] references randomized test suspected bugs in _Py_RefTotal tracking: test_descr leaked [2, 2, 2, 2] references something to do with resurrection test_gc leaked [18, 18, 18, 18] references test_trashcan ... not a real leak, it seems suspected red herrings: test_cpickle leaked [6, 3, 2, 1] references test_logging leaked [82, 82, 82, 82] references (bet this is registry stuff of one sort or another) test_threadedtempfile leaked [-74, 3, 3, 161] references test_unicode leaked [7, 7, 7, 7] references registries again apparent refleaks that still need digging: test_minidom leaked [1, 1, 1, 1] references test_new leaked [2, 2, 2, 2] references test_pkg leaked [8, 8, 9, 9] references test_pkgimport leaked [5, 6, 6, 6] references test_sax leaked [1762, 1762, 1762, 1762] references Wouldn't like to claim anything about the following tests: 6 tests failed: test_dircache test_doctest test_getopt test_multifile test_warnings test_zipimport 27 tests skipped: test_al test_bsddb test_bsddb3 test_cd test_cl test_curses test_dl test_email_codecs test_gdbm test_gl test_imgfile test_largefile test_linuxaudiodev test_locale test_mpz test_nis test_normalization test_ossaudiodev test_pep277 test_poll test_socket_ssl test_socketserver test_sunaudiodev test_timeout test_urllibnet test_winreg test_winsound Those skips are all expected on darwin. (test_threaded_import deadlocks, so add that to the list) Should investigate why the failing tests fail when run repeatedly and check that they're not hiding ref leaks. Cheers, mwh From bac at OCF.Berkeley.EDU Mon Sep 15 14:20:21 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon Sep 15 14:20:36 2003 Subject: [Python-Dev] Re: python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: <1063637821.2093.132.camel@localhost.localdomain> References: <1063637821.2093.132.camel@localhost.localdomain> Message-ID: <3F6602E5.1040300@ocf.berkeley.edu> Jeremy Hylton wrote: > On Mon, 2003-09-15 at 08:51, A.M. Kuchling wrote: > >>I'd be in favor of setting an arbitrary date for a 2.4 release, and when >>that date rolls around stabilizing and releasing the CVS tree, no matter >>what's been implemented or not implemented. If we've made major language >>revisions by that time, fine, but it's worthwhile even if 2.4 is just 2.3 + >>(various bugfixes deemed too risky for 2.3.1) + (a few new modules). >> >>PEP 320 estimates February 2005, a 19-month gap matching that between 2.2 >>and 2.3. That seems a bit too long, especially if there are no large >>changes that require testing; aiming for autumn 2004, beginning to freeze >>in August, seems reasonable. > > > In the absence of any plans for major new features, I think this is a > good plan. A rough schedule might be: > > 2.4a1 April > 2.4a2 late May/early June > 2.4b1 July 9 > 2.4b2 Aug. 6 > 2.4 f Sep. 3 > +1 We could also try to schedule micro releases two months after the initial release. That seems to have been a decent amount of time to get in bug reports and fix them. Then again, standardizing micro releases might cause people to hold off on using the newest release and thus we might get less testing from users. -Brett From recht at netbsd.org Mon Sep 15 17:07:59 2003 From: recht at netbsd.org (Marc Recht) Date: Mon Sep 15 17:08:15 2003 Subject: [Python-Dev] Python and POSIX Message-ID: <35230000.1063660079@leeloo.intern.geht.de> Hi! Currently Python configure process sets quite a few POSIX/XOPEN macros (eg. _POSIX_C_SOURCE/_XOPEN_SOURCE) to archieve portability. The is a good thing - for the interpreter. The problem with this is that these macros are exported to Python extensions via pyconfig.h. That way every extension which uses pyconfig.h is limited to POSIX/XOPEN functionality and not all library features of the OS (eg. RPC) are available for the extension. For the systems that supported it the Python configure process also sets macros which activate all library functions (_NETBSD_SOURCE, __EXTENSIONS__) and thus avoiding POSIX as a work-around. On all systems I'm aware of this is the same feature-set you get if you don't set any macro at all, but unfortunately not all systems support such a macro. So, why not split pyconfig.h up into an external and internal header? That way the the interpreter could be POSIX confirming (which is indeed a good thing) without enforcing POSIX to the extensions. Regards, Marc From martin at v.loewis.de Mon Sep 15 17:58:14 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Mon Sep 15 18:02:57 2003 Subject: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: <1063627491.3f65aae37a3b7@mcherm.com> References: <1063627491.3f65aae37a3b7@mcherm.com> Message-ID: Michael Chermside writes: > I don't really understand this absolutist position. I think the rationale is that developers don't actively *use* bug fixes, whereas they do use new features. So when you tell them "we have booleans now", they will drop everything else and start putting booleans all over the place. In the middle of that, they think "wait, what about 2.2", and they will test 2.2.1, and find that it works just fine. They run into problems only after they release the software. With bug fixes, it is different: People won't actively change their working code to deliberately formulate it in a way that breaks on older Python versions. Instead, upon reading that the bug has been fixed, they typically think "does not concern me", and proceed with whatever they were doing, anyway. In some cases, they may happen to write new code that relies on a recent bug fix. When they discover that older versions don't work with their code, they just bump the prerequisites, or add a work-around. In addition, there is always the concern that new code may break existing code, so the amount of new code should be kept to the smallest possible minimum. Adding new features is beyond that minimum, so it should not be done (regardless of whether that new code is likely to break existing code, upon visual inspection). Regards, Martin From martin at v.loewis.de Mon Sep 15 18:05:12 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Mon Sep 15 18:05:29 2003 Subject: [Python-Dev] Python and POSIX In-Reply-To: <35230000.1063660079@leeloo.intern.geht.de> References: <35230000.1063660079@leeloo.intern.geht.de> Message-ID: Marc Recht writes: > On all systems I'm aware of this is the same feature-set you get if > you don't set any macro at all, but unfortunately not all systems > support such a macro. You should take a look at Solaris, then. Depending on whether you define XPG macros (I forgot which ones specifically), the socket functions mutate from BSD (e.g. connect(3BSD)) to XPG functions (connect(3XPG)). This is implemented by exposing #pragma redefine_extname connect __xpg_connect or some such in socket.h, if the XPG macro is defined. Likewise, on HP-UX, you select one of the three (!) curses libraries on the system through these macros. > So, why not split pyconfig.h up into an external and internal header? That would be possible if one would a) provide a patch, and b) could demonstrate that, on all relevant systems, lack of such macros cannot possible have an effect on proper operation of extension modules. In particular, don't try to make LFS support "internal": Extensions modules are known to break if they use a different "struct stat" from the one that the core uses. Regards, Martin From python at rcn.com Tue Sep 16 00:46:04 2003 From: python at rcn.com (Raymond Hettinger) Date: Tue Sep 16 00:50:54 2003 Subject: [Python-Dev] Re: Re: python/dist/src/Lib/bsddb __init__.py, 1.5, 1.6 References: <1063637821.2093.132.camel@localhost.localdomain> Message-ID: <004101c37c0d$701be040$0c06a044@oemcomputer> > Is it worth holding 2.3.1 until after Panther ships, on the assumption that > its release will turn up more users and more bugs? If not, 2.3.1's timing > can probably be left up to the release manager, who can make the release > whenever he feels like it. Last month, we worked out a tentantive plan to start the 2.3.1 release process in the third week of September. I think we should stick with that. The initial round of bugfixes is an important step for 2.3 being viewed as stable and to show everyone that development/support has not vanished. If critical bugs surface later, that can be a consideration for 2.3.2. There is no reason to be stingy with the initial bugfix release. Raymond ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From bac at OCF.Berkeley.EDU Tue Sep 16 01:09:07 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Tue Sep 16 01:09:20 2003 Subject: [Python-Dev] Trying to fix time.tzset detection for 2.3.1 Message-ID: <3F669AF3.8050605@ocf.berkeley.edu> Some of you may remember that I was working on closing a bug for Red Hat 6.1 where time.tzset was being detected by 'configure' when it shouldn't have (it's busted on that platform) for 2.3.0 . Unfortunately a working patch never surfaced in time to be thoroughly tested. Well, Kurt Kaiser wrote up a patch ( http://www.python.org/sf/762934 ) that seems sound. I have gotten zero responses from anyone to test it on Red Hat 6.1 and a remember that some people here were helping to test last time (I also remember this led to the creation of a wiki page listing platforms various people could test things on but I can't find the page). I would like to get this in for 2.3.1 (assuming this is considered reasonable; not sure after the discussion that has gone on over what should go into a micro release) if possible. So if anyone is running Red Hat 6.1 or has easy access to one I would appreciate it if you took a look at the patch and gave it a go (don't forget the various steps needed to make the patch work; read the follow-ups to see what needs to be done). -Brett From mal at lemburg.com Tue Sep 16 03:40:59 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Sep 16 03:40:54 2003 Subject: [Python-Dev] datetime issues In-Reply-To: <20030915000637.GC27698@ibook> References: <20030915000637.GC27698@ibook> Message-ID: <3F66BE8B.7060307@lemburg.com> Gustavo Niemeyer wrote: > Also, is there any further work going on to improve the datetime module? > I'm not suggesting we should mirror mx.DateTime functionalities, but two > features I miss from mx.DateTime is the DateFrom(), which does its best > to parse a given string, and the RelativeDateTime() functionality. > > This has probably been raised in the past, so if that's the case, I'm > sorry. Marc-Andre, what's your opinion about reusing code from > mx.DateTime into datetime? The mxDateTime code comes under the eGenix Public License, so reuse is permittted under the terms of the license. That said, I doubt that you want to maintain a date/time parser in the Python core -- I certainly don't :-) > Anyway, reusing code or not, I'll probably put sometime on it in the > future, if this looks interesting to everyone (after I finish my current > python pendencies, like SRE's recursivity removal). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Sep 16 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mwh at python.net Tue Sep 16 04:59:28 2003 From: mwh at python.net (Michael Hudson) Date: Tue Sep 16 04:58:58 2003 Subject: [Python-Dev] Trying to fix time.tzset detection for 2.3.1 In-Reply-To: <3F669AF3.8050605@ocf.berkeley.edu> (Brett C.'s message of "Mon, 15 Sep 2003 22:09:07 -0700") References: <3F669AF3.8050605@ocf.berkeley.edu> Message-ID: <2mznh5az2n.fsf@starship.python.net> "Brett C." writes: > I would like to get this in for 2.3.1 (assuming this is considered > reasonable; not sure after the discussion that has gone on over what > should go into a micro release) if possible. So if anyone is running > Red Hat 6.1 or has easy access to one I would appreciate it if you > took a look at the patch and gave it a go I no longer have access to RH 6.1 (the starship is now Debian testing). Cheers, mwh -- Of course, it obviously is beta hardware so such things are to be expected, but that doesn't mean that you can't point your fingers and generate a nelson style HAHA at a multi billion dollar corporation's expense. -- CmdrTaco on slashdot.org From arigo at tunes.org Tue Sep 16 06:50:04 2003 From: arigo at tunes.org (Armin Rigo) Date: Tue Sep 16 06:52:03 2003 Subject: [Python-Dev] pre-PEP: Resource-Release Support for Generators In-Reply-To: <5.1.1.6.0.20030915112135.03013ec0@telecommunity.com> References: <5.2.1.1.0.20030913160918.02773bc0@pop.bluewin.ch> <200308262050.h7QKoqD04780@12-236-84-31.client.attbi.com> <5.2.1.1.0.20030826151628.023f8d68@pop.bluewin.ch> <00a401c36c0f$76952160$c304a044@oemcomputer> <200308262050.h7QKoqD04780@12-236-84-31.client.attbi.com> <5.2.1.1.0.20030913160918.02773bc0@pop.bluewin.ch> <5.1.1.6.0.20030915112135.03013ec0@telecommunity.com> Message-ID: <20030916105004.GA11247@vicky.ecs.soton.ac.uk> Hello Phillip, On Mon, Sep 15, 2003 at 11:26:27AM -0400, Phillip J. Eby wrote: > i = iter(something) > while 1: > try: > j=i.next() > except StopIteration: > break > > (...) is already complex enough as a mental model. Makes sense. Then let's make sure, if both 'with' and 'yield within try:finally' are accepted, that they are compatible, e.g. by having and __exit__() method on generators (and not just a close()). Armin From recht at netbsd.org Tue Sep 16 06:59:01 2003 From: recht at netbsd.org (Marc Recht) Date: Tue Sep 16 06:59:06 2003 Subject: [Python-Dev] Python and POSIX In-Reply-To: References: <35230000.1063660079@leeloo.intern.geht.de> Message-ID: <46570000.1063709941@leeloo.intern.geht.de> > You should take a look at Solaris, then. Depending on whether you > define XPG macros (I forgot which ones specifically), the socket > functions mutate from BSD (e.g. connect(3BSD)) to XPG functions > (connect(3XPG)). This is implemented by exposing > ># pragma redefine_extname connect __xpg_connect > > or some such in socket.h, if the XPG macro is defined. I had a quick glance at the Solaris 8 headers and it seems that's influenced by __EXTENSIONS__, too. So it _should_ be the same as without the macros. > Likewise, on HP-UX, you select one of the three (!) curses libraries > on the system through these macros. Oh. Didn't know about HP-UX. That's weird. >> So, why not split pyconfig.h up into an external and internal header? > > That would be possible if one would > a) provide a patch, and OK. I justed wanted to verify if there's enough interest (and no general rejection) in such a patch and if anyone else is already working on it. > b) could demonstrate that, on all relevant systems, lack of such macros > cannot possible have an effect on proper operation of extension > modules. That's a bit harder since I don't have access to all "relevant" systems. I could test on the open-source x86 platforms, though. And IIRC CompileFarm has Solaris, too. And I _believe_ that - since these changes came up with Python 2.3 (with the exception of _POSIX_1_SOURCE and _POSIX_SOURCE) - that it will break not much or anything at all if at least _XOPEN_SOURCE are moved _XOPEN_SOURCE_EXTENDED to an internal header (or an internal part of pyconfig.h). (As a first step.) But, you're right there shouldn't be a rash decision and this needs much testing. I think I'll just go on and provide a patch as basis for further discussion. Regards, Marc From neal at metaslash.com Tue Sep 16 08:40:17 2003 From: neal at metaslash.com (Neal Norwitz) Date: Tue Sep 16 08:40:21 2003 Subject: [Python-Dev] Trying to fix time.tzset detection for 2.3.1 In-Reply-To: <3F669AF3.8050605@ocf.berkeley.edu> References: <3F669AF3.8050605@ocf.berkeley.edu> Message-ID: <20030916124017.GB13879@epoch.metaslash.com> On Mon, Sep 15, 2003 at 10:09:07PM -0700, Brett C. wrote: > > Well, Kurt Kaiser wrote up a patch ( http://www.python.org/sf/762934 ) > that seems sound. I have gotten zero responses from anyone to test it > on Red Hat 6.1 and a remember that some people here were helping to test > last time (I also remember this led to the creation of a wiki page > listing platforms various people could test things on but I can't find > the page). http://www.python.org/cgi-bin/moinmoin/PythonTesters From niemeyer at conectiva.com Tue Sep 16 09:54:36 2003 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue Sep 16 09:55:27 2003 Subject: [Python-Dev] datetime issues In-Reply-To: <3F66BE8B.7060307@lemburg.com> References: <20030915000637.GC27698@ibook> <3F66BE8B.7060307@lemburg.com> Message-ID: <20030916135436.GA2509@ibook.distro.conectiva> > The mxDateTime code comes under the eGenix Public License, so > reuse is permittted under the terms of the license. That said, > I doubt that you want to maintain a date/time parser in the > Python core -- I certainly don't :-) Thanks. I've got the relativedelta type ready. It's based on your specification (thanks for it), but uses no code from your module (you'll notice that clearly when looking at the code). Indeed, it has even a different algorithm, which solves some of the current "problems" with your implementation (like getting 03/03 when adding one month to 31/01). You're free to adopt it, if you feel like so. I'll be releasing that code under the PSF license. I'll probably work on a generic parser, once I get some more time. Depending on the results, I'll submit it for review and possible inclusion as well. Here is the documentation of relativedelta, FWIW: """ The relativedelta type is based on the specification of the excelent work done by M.-A. Lemburg in his mx.DateTime extension. However, notice that this type does *NOT* implement the same algorithm as his work. Do *NOT* expect it to behave like mx.DateTime's counterpart. There's two different ways to build a relativedelta instance. The first one is passing it two date/datetime classes: relativedelta(datetime1, datetime2) And the other way is to use the following keyword arguments: year, month, day, hour, minute, seconds, microseconds: Absolute information. years, months, weeks, days, hours, minutes, seconds, microseconds: Relative information, may be negative. weekday: Tuple with (wday, nth), specifying the nth relative weekday. Here is the behavior of operations with relativedelta: 1) Calculate the absolute year, using the 'year' argument, or the original datetime year, if the argument is not present. 2) Add the relative 'years' argument to the absolute year. 3) Do steps 1 and 2 for month/months. 4) Calculate the absolute day, using the 'day' argument, or the original datetime day, if the argument is not present. Then, subtract from the day until it fits in the year and month found after their operations. 5) Add the relative 'days' argument to the absolute day. Notice that the 'weeks' argument is multiplied by 7 and added to 'days'. 6) Do steps 1 and 2 for hour/hours, minute/minutes, second/seconds, microsecond/microseconds. 7) If the 'weekday' argument is present, calculate the weekday, with the given (wday, nth) tuple. wday is the index of the weekday (0-6, 0=Mon), and nth is the number of weeks to add forward or backward, depending on its signal. Notice that if the calculated date is already Monday, for example, using (0, 1) or (0, -1) won't change the day. """ And here is a little demo: >>> from dtutil import *; from datetime import * >>> relativedelta(datetime.now(),datetime(1983,9,10,10,00)) relativedelta(years=+20, days=+6, minutes=+25, seconds=+4, microseconds=+231402) >>> datetime(1983,9,10,10,00)+relativedelta(years=+20, days=+6, minutes=+25, seconds=+4, microseconds=+231402) datetime.datetime(2003, 9, 16, 10, 25, 4, 231402) >>> relativedelta(datetime(2000,2,29),datetime(2001,3,29)) relativedelta(years=-1, months=-1) >>> datetime(2001,3,29)+relativedelta(years=-1, months=-1) datetime.datetime(2000, 2, 29, 0, 0) >>> relativedelta(datetime(2000,2,29),datetime(2001,3,30)) relativedelta(years=-1, months=-1) >>> datetime(2001,3,30)+relativedelta(years=-1, months=-1) datetime.datetime(2000, 2, 29, 0, 0) >>> relativedelta(datetime(2000,2,29),datetime(2001,2,28)) relativedelta(months=-11, days=-28) >>> relativedelta(datetime(2000,2,28),datetime(2001,2,28)) relativedelta(years=-1) >>> relativedelta(datetime(2000,2,27),datetime(2001,2,28)) relativedelta(years=-1, days=-1) >>> relativedelta(datetime(2001,2,28),datetime(2000,2,29)) relativedelta(years=+1) >>> relativedelta(datetime(2001,2,28),datetime(2000,2,28)) relativedelta(years=+1) >>> relativedelta(datetime(2001,2,28),datetime(2000,2,27)) relativedelta(years=+1, days=+1) >>> import calendar >>> date.today()+relativedelta(weekday=(calendar.MONDAY, +2)) datetime.date(2003, 9, 29) >>> date.today()+relativedelta(weekday=(calendar.MONDAY, -1)) datetime.date(2003, 9, 15) -- Gustavo Niemeyer http://niemeyer.net From skip at pobox.com Tue Sep 16 10:17:54 2003 From: skip at pobox.com (Skip Montanaro) Date: Tue Sep 16 10:18:06 2003 Subject: [Python-Dev] Trying to fix time.tzset detection for 2.3.1 In-Reply-To: <3F669AF3.8050605@ocf.berkeley.edu> References: <3F669AF3.8050605@ocf.berkeley.edu> Message-ID: <16231.7058.987290.575924@montanaro.dyndns.org> Brett> Well, Kurt Kaiser wrote up a patch ( Brett> http://www.python.org/sf/762934 ) that seems sound. I have Brett> gotten zero responses from anyone to test it on Red Hat 6.1 and a Brett> remember that some people here were helping to test last time (I Brett> also remember this led to the creation of a wiki page listing Brett> platforms various people could test things on but I can't find Brett> the page). That would be: http://www.python.org/cgi-bin/moinmoin/PythonTesters I see Kurt Kaiser as the only person listed who can test for RH < 6.2. Perhaps you should broadcast a call for testers to c.l.py, referring to the above page and asking in particular for RH 6.1 folks to chime in. Skip From tim at zope.com Tue Sep 16 10:34:23 2003 From: tim at zope.com (Tim Peters) Date: Tue Sep 16 10:34:49 2003 Subject: [Python-Dev] RE: [Zope-Coders] core dump in Zope 2.7 test suite In-Reply-To: <1063684862.2093.193.camel@localhost.localdomain> Message-ID: [Tim] >> $9 = {_ob_next = 0x0, _ob_prev = 0x0, ob_refcnt = 0, >> ob_type = 0x8130160, length = 1, str = 0x41903130, >> hash = -1, defenc = 0x0} >> >> that certainly means the object has been released already. [Jeremy] > This is code that is trying to revive an object from the freelist, so > it isn't a big surprise that the object looks like it has been freed. Doh -- yup, of course. > Now here's a very strange thing. If I apply the following patch, the > problem goes away. The patch doesn't make a lot of sense to me, so > I'm guessing that it's just a lucky patch -- avoiding the problem > without fixing the cause. We should move this to python-dev. OK, I just did . Another clue about the cause coming up. > The thing that is curious is that we change the test against > unicode->str[0], which is probably an unsigned int, Why? Are you doing a UCS4 build? The expansion of Py_UNICODE (unicode->str is an array of Py_UNICODE thingies) can't be guessed from where I sit (it expands to "unsigned short" on Windows, but that's because that's hardcoded in PC\pyconfig.h; I'm not sure where you get its expansion from). > so that we also test that it is > 0. In the specific case that > crashes, the debugger says the value is -875836469. That's probably a great clus! In hex, that's 0xcbcbcbcb, which is the special byte pattern the debug pymalloc sprays into freshly-allocated memory. IOW, unicode->str contains (in a release build) uninitialized heap trash at this point, and so there's nothing it can be tested against that makes any sense. This suggests a deeper bug in the Unicode support. > Does that make any sense? Do we actually have a signed value here? You have to figure out what Py_UNICODE expands to in your build to answer that one. > Or does it get converted to a signed value when the comparison occurs? If gdb showed you -875836469 in response to "unicode->str[0]" in isolation, I have to guess Py_UNICODE is expanding to a signed 4-byte type in your build. > Jeremy > > Index: unicodeobject.c > =================================================================== > RCS file: /cvsroot/python/python/dist/src/Objects/unicodeobject.c,v > retrieving revision 2.196 > diff -c -r2.196 unicodeobject.c > *** unicodeobject.c 16 Sep 2003 03:41:45 -0000 2.196 > --- unicodeobject.c 16 Sep 2003 03:55:53 -0000 > *************** > *** 130,138 **** > /* Resizing shared object (unicode_empty or single character > objects) in-place is not allowed. Use PyUnicode_Resize() > instead ! */ > if (unicode == unicode_empty || > (unicode->length == 1 && > ! unicode->str[0] < 256 && > unicode_latin1[unicode->str[0]] == unicode)) { > PyErr_SetString(PyExc_SystemError, > "can't resize shared unicode objects"); > --- 130,142 ---- > /* Resizing shared object (unicode_empty or single character > objects) in-place is not allowed. Use PyUnicode_Resize() > instead ! */ > if (unicode == unicode_empty || > (unicode->length == 1 && > ! unicode->str[0] < 256 && unicode->str[0] > 0 && > unicode_latin1[unicode->str[0]] == unicode)) { > PyErr_SetString(PyExc_SystemError, > "can't resize shared unicode objects"); Belaboring the obvious, if unicode->str[0] is really heap trash at this point, there's no safe test. From jeremy at zope.com Tue Sep 16 11:05:19 2003 From: jeremy at zope.com (Jeremy Hylton) Date: Tue Sep 16 11:05:50 2003 Subject: [Python-Dev] RE: [Zope-Coders] core dump in Zope 2.7 test suite In-Reply-To: References: Message-ID: <1063724718.4868.2.camel@localhost.localdomain> I disabled the KEEPALIVE_SIZE_LIMIT, as you suggested, and that also fixed the problem. The comment describe the code is really unnerving: /* Limit for the Unicode object free list stay alive optimization. ... Note: This is an experimental feature ! If you get core dumps when using Unicode objects, turn this feature off. */ Why was this experimental feature released with Python if it causes core dumps? Jeremy From tim at zope.com Tue Sep 16 12:12:26 2003 From: tim at zope.com (Tim Peters) Date: Tue Sep 16 12:12:48 2003 Subject: [Python-Dev] RE: [Zope-Coders] core dump in Zope 2.7 test suite In-Reply-To: <1063724718.4868.2.camel@localhost.localdomain> Message-ID: [Jeremy Hylton] > I disabled the KEEPALIVE_SIZE_LIMIT, as you suggested, and that also > fixed the problem. We don't know why, though, nor do we have a small test case, right? > The comment describe the code is really unnerving: > > /* Limit for the Unicode object free list stay alive optimization. > > ... > > Note: This is an experimental feature ! If you get core dumps when > using Unicode objects, turn this feature off. > > */ > > Why was this experimental feature released with Python if it causes > core dumps? Good question . Going back to an earlier msg: > Index: unicodeobject.c > =================================================================== > RCS file: /cvsroot/python/python/dist/src/Objects/unicodeobject.c,v > retrieving revision 2.196 > diff -c -r2.196 unicodeobject.c > *** unicodeobject.c 16 Sep 2003 03:41:45 -0000 2.196 > --- unicodeobject.c 16 Sep 2003 03:55:53 -0000 > *************** > *** 130,138 **** > /* Resizing shared object (unicode_empty or single character > objects) in-place is not allowed. Use PyUnicode_Resize() > instead ! */ > if (unicode == unicode_empty || > (unicode->length == 1 && > ! unicode->str[0] < 256 && > unicode_latin1[unicode->str[0]] == unicode)) { > PyErr_SetString(PyExc_SystemError, > "can't resize shared unicode objects"); > --- 130,142 ---- > /* Resizing shared object (unicode_empty or single character > objects) in-place is not allowed. Use PyUnicode_Resize() > instead ! */ > if (unicode == unicode_empty || > (unicode->length == 1 && > ! unicode->str[0] < 256 && unicode->str[0] > 0 && > unicode_latin1[unicode->str[0]] == unicode)) { > PyErr_SetString(PyExc_SystemError, > "can't resize shared unicode objects"); > Belaboring the obvious, if unicode->str[0] is really heap trash at > this point, there's no safe test. On second thought, the test could be correct even if it does read up heap trash: the unicode_latin1[unicode->str[0]] == unicode part is a very strong condition. If str[0] does contain heap trash, it can't pass, but it could cause Purify (etc) to whine about reading uninitialized memory. If we don't care about that, unicode->length == 1 && (unsigned int)unicode->str[0] < 256U && unicode_latin1[unicode->str[0]] == unicode would be a safe and correct guard. Still, it doesn't look to me like the code *expected* that str could contain uninitialized heap trash at this point, so I'd like someone who thinks they understand this code to think about how that could happen (it apparently is happening, so "beats me" isn't good enough ). From patmiller at llnl.gov Tue Sep 16 12:24:45 2003 From: patmiller at llnl.gov (Pat Miller) Date: Tue Sep 16 12:24:50 2003 Subject: [Python-Dev] Making python C-API thread safe (try 2) In-Reply-To: <200309120311.h8C3BSe01781@oma.cosc.canterbury.ac.nz> References: <200309120311.h8C3BSe01781@oma.cosc.canterbury.ac.nz> Message-ID: <3F67394D.305@llnl.gov> I used to work on the SISAL compiler which was a fine-grained parallel functional language. It reference counted its allocated data. Updating the reference count is a critical section. We were spending 40% of our time blocked for reference counting on only 4 threads. We eventually taught the compiler to do reference count optimizations (e.g. put all readers of an object before the writers), but these optimizations are not suitable to the Python interpreter. With this same reasoning, you can infer why C++ smart pointers are also a bad idea for threaded code. Pat -- Patrick Miller | (925) 423-0309 | http://www.llnl.gov/CASC/people/pmiller All you need in this life is ignorance and confidence, and then success is sure. -- Mark Twain From jeremy at zope.com Tue Sep 16 12:31:20 2003 From: jeremy at zope.com (Jeremy Hylton) Date: Tue Sep 16 12:31:45 2003 Subject: [Python-Dev] RE: [Zope-Coders] core dump in Zope 2.7 test suite In-Reply-To: References: Message-ID: <1063729880.4868.8.camel@localhost.localdomain> On Tue, 2003-09-16 at 12:12, Tim Peters wrote: > [Jeremy Hylton] > > I disabled the KEEPALIVE_SIZE_LIMIT, as you suggested, and that also > > fixed the problem. > > We don't know why, though, nor do we have a small test case, right? Not a small test case, no, but I have boiled it down a bit. If you run the tests from a Zope checkout on the Zope-2_7-branch, you can provoke the crash with "test.py Template." > Going back to an earlier msg: > > > Index: unicodeobject.c > > =================================================================== > > RCS file: /cvsroot/python/python/dist/src/Objects/unicodeobject.c,v > > retrieving revision 2.196 > > diff -c -r2.196 unicodeobject.c > > *** unicodeobject.c 16 Sep 2003 03:41:45 -0000 2.196 > > --- unicodeobject.c 16 Sep 2003 03:55:53 -0000 > > *************** > > *** 130,138 **** > > /* Resizing shared object (unicode_empty or single character > > objects) in-place is not allowed. Use PyUnicode_Resize() > > instead ! */ > > if (unicode == unicode_empty || > > (unicode->length == 1 && > > ! unicode->str[0] < 256 && > > unicode_latin1[unicode->str[0]] == unicode)) { > > PyErr_SetString(PyExc_SystemError, > > "can't resize shared unicode objects"); > > --- 130,142 ---- > > /* Resizing shared object (unicode_empty or single character > > objects) in-place is not allowed. Use PyUnicode_Resize() > > instead ! */ > > if (unicode == unicode_empty || > > (unicode->length == 1 && > > ! unicode->str[0] < 256 && unicode->str[0] > 0 && > > unicode_latin1[unicode->str[0]] == unicode)) { > > PyErr_SetString(PyExc_SystemError, > > "can't resize shared unicode objects"); > > > Belaboring the obvious, if unicode->str[0] is really heap trash at > > this point, there's no safe test. > > On second thought, the test could be correct even if it does read up heap > trash: the > > unicode_latin1[unicode->str[0]] == unicode > > part is a very strong condition. If str[0] does contain heap trash, it > can't pass, but it could cause Purify (etc) to whine about reading > uninitialized memory. If we don't care about that, It could also cause a segmentation violation depending on the value of the stack trash it reads. At least, that's what appears to be going wrong. > > unicode->length == 1 && > (unsigned int)unicode->str[0] < 256U && > unicode_latin1[unicode->str[0]] == unicode > > would be a safe and correct guard. That sounds better. > Still, it doesn't look to me like the code *expected* that str could contain > uninitialized heap trash at this point, so I'd like someone who thinks they > understand this code to think about how that could happen (it apparently is > happening, so "beats me" isn't good enough ). I certainly don't understand the code yet. Jeremy From bac at OCF.Berkeley.EDU Tue Sep 16 12:53:41 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Tue Sep 16 12:54:02 2003 Subject: [Python-Dev] Trying to fix time.tzset detection for 2.3.1 In-Reply-To: <16231.7058.987290.575924@montanaro.dyndns.org> References: <3F669AF3.8050605@ocf.berkeley.edu> <16231.7058.987290.575924@montanaro.dyndns.org> Message-ID: <3F674015.8090801@ocf.berkeley.edu> Skip Montanaro wrote: > Brett> Well, Kurt Kaiser wrote up a patch ( > Brett> http://www.python.org/sf/762934 ) that seems sound. I have > Brett> gotten zero responses from anyone to test it on Red Hat 6.1 and a > Brett> remember that some people here were helping to test last time (I > Brett> also remember this led to the creation of a wiki page listing > Brett> platforms various people could test things on but I can't find > Brett> the page). > > That would be: > > http://www.python.org/cgi-bin/moinmoin/PythonTesters > > I see Kurt Kaiser as the only person listed who can test for RH < 6.2. > Perhaps you should broadcast a call for testers to c.l.py, referring to the > above page and asking in particular for RH 6.1 folks to chime in. > Jordan Krushen pointed out that the patch is actually opened against RH 6.2, not 6.1 like I originally said. But I don't think it is that big of an issue since there is a bug report for this against SuSE. I think the key thing here is an older version of glibc since that would be what defines the tzset function. But I will consider sending a note out to c.l.py about this. Thanks for the suggestion, Skip. -Brett From tim at zope.com Tue Sep 16 12:58:21 2003 From: tim at zope.com (Tim Peters) Date: Tue Sep 16 12:58:45 2003 Subject: [Python-Dev] RE: [Zope-Coders] core dump in Zope 2.7 test suite In-Reply-To: <1063729880.4868.8.camel@localhost.localdomain> Message-ID: [Jeremy Hylton] >>> I disabled the KEEPALIVE_SIZE_LIMIT, as you suggested, and that also >>> fixed the problem. [Tim >> We don't know why, though, nor do we have a small test case, right? [Jeremy] > Not a small test case, no, but I have boiled it down a bit. If you > run the tests from a Zope checkout on the Zope-2_7-branch, you can > provoke the crash with "test.py Template." Sorry, I cannot. I tried a fresh Zope-2_7-branch, on Windows, with released Python 2.3, release build current CVS Python, release build current CVS Python, debug build "test.py Template" gave the same result in all cases: 77 tests run, 1 failure (AssertionError: HTML Output Changed) in checkStringExpressions. But note that Py_UNICODE is unsigned short on this box, and reading a trash one of those as an index isn't nearly as likely to reach into unmapped memory as on a box where you get 4 bytes of trash. >> ... >> On second thought, the test could be correct even if it does read up >> heap trash: the >> >> unicode_latin1[unicode->str[0]] == unicode >> >> part is a very strong condition. If str[0] does contain heap trash, >> it can't pass, but it could cause Purify (etc) to whine about reading >> uninitialized memory. If we don't care about that, > It could also cause a segmentation violation depending on the value of > the stack trash it reads. At least, that's what appears to be going > wrong. Right, I meant the condition is very strong *if* the unicode->str[0] part is known to lie in range(256). unicode_latin1 is statically declared to contain 256 elements, so if the trash index is in range(256) then the index resolves to legitimate memory. >> unicode->length == 1 && >> (unsigned int)unicode->str[0] < 256U && >> unicode_latin1[unicode->str[0]] == unicode >> would be a safe and correct guard. > That sounds better. To worm around something that shouldn't happen at all, sure . >> Still, it doesn't look to me like the code *expected* that str could >> contain uninitialized heap trash at this point, so I'd like someone >> who thinks they understand this code to think about how that could >> happen (it apparently is happening, so "beats me" isn't good enough >> ). > I certainly don't understand the code yet. Me neither, but offhand didn't see anything glaringly wrong. If some internal Unicode operation decided to allocate a short Unicode string, but freed it before filling in any of the string bytes, I suppose the keepalive optimization would retain a Unicode object with uninitialized str space on the unicode free list. A subsequent _PyUnicode_New could grab that and try to boost its ->str size. That could explain it. From jeremy at alum.mit.edu Tue Sep 16 12:58:43 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Tue Sep 16 12:58:47 2003 Subject: [Python-Dev] RE: [Zope-Coders] core dump in Zope 2.7 test suite In-Reply-To: <1063729880.4868.8.camel@localhost.localdomain> References: <1063729880.4868.8.camel@localhost.localdomain> Message-ID: <1063731523.4868.11.camel@localhost.localdomain> On Tue, 2003-09-16 at 12:31, Jeremy Hylton wrote: > On Tue, 2003-09-16 at 12:12, Tim Peters wrote: > > [Jeremy Hylton] > > > I disabled the KEEPALIVE_SIZE_LIMIT, as you suggested, and that also > > > fixed the problem. > > > > We don't know why, though, nor do we have a small test case, right? > > Not a small test case, no, but I have boiled it down a bit. If you run > the tests from a Zope checkout on the Zope-2_7-branch, you can provoke > the crash with "test.py Template." Okay. I have a smaller test case, though I don't know what I couldn't find it earlier. If you run "test.py DocumentTemplate UnicodeStr" by itself, you get the crash. The DocumentTemplate implementation has C and Python implementations of some utilities. It uses the C by default and falls back to Python if the C extension can't be imported. If I change the code so that it always uses the Python implementation, the crash goes away. Jeremy From tim at zope.com Tue Sep 16 13:11:13 2003 From: tim at zope.com (Tim Peters) Date: Tue Sep 16 13:12:00 2003 Subject: [Python-Dev] RE: [Zope-Coders] core dump in Zope 2.7 test suite In-Reply-To: Message-ID: [Jeremy] >> Not a small test case, no, but I have boiled it down a bit. If you >> run the tests from a Zope checkout on the Zope-2_7-branch, you can >> provoke the crash with "test.py Template." [Tim] > Sorry, I cannot. I tried a fresh Zope-2_7-branch, on Windows, with > > released Python 2.3, release build > current CVS Python, release build > current CVS Python, debug build > > "test.py Template" gave the same result in all cases: 77 tests run, 1 > failure (AssertionError: HTML Output Changed) in > checkStringExpressions. > > But note that Py_UNICODE is unsigned short on this box, and reading a > trash one of those as an index isn't nearly as likely to reach into > unmapped memory as on a box where you get 4 bytes of trash. What I *could* do, though, is add this to unicode_resize(): if (unicode != unicode_empty && unicode->length == 1) { assert(unicode->str[0] != 0xcbcbU); } That checks for the debug pymalloc's "uninitialized memory" byte pattern appropriate for this box, and that assert does indeed trigger when running test.py Template It fires while running checkUnicodeInserts, but checkUnicodeInserts does not fail in isolation. So checkUnicodeInserts is the victim of something an earlier test left on the unicode freelist. From tim at zope.com Tue Sep 16 13:15:53 2003 From: tim at zope.com (Tim Peters) Date: Tue Sep 16 13:16:58 2003 Subject: [Python-Dev] RE: [Zope-Coders] core dump in Zope 2.7 test suite In-Reply-To: <1063731523.4868.11.camel@localhost.localdomain> Message-ID: [Jeremy Hylton] > Okay. I have a smaller test case, though I don't know what I couldn't > find it earlier. If you run "test.py DocumentTemplate UnicodeStr" by > itself, you get the crash. As before, nothing crashes here, but running that test does trigger the debug-build-specific assert I added here (see last msg). Good sleuthing! > The DocumentTemplate implementation has C and Python implementations > of some utilities. It uses the C by default and falls back to Python > if the C extension can't be imported. If I change the code so that it > always uses the Python implementation, the crash goes away. I'm guessing lib/python/DocumentTemplate/cDocumentTemplate.c From theller at python.net Tue Sep 16 13:45:40 2003 From: theller at python.net (Thomas Heller) Date: Tue Sep 16 13:45:22 2003 Subject: [Python-Dev] Re: HTMLHelp for Py2.3.1 In-Reply-To: (Boriz Izaguirre's message of "Sun, 7 Sep 2003 15:52:36 +0200") References: Message-ID: <8yook4or.fsf@python.net> "Boriz Izaguirre" writes: > [Thomas] >>[Fred L. Drake] >>> IDLE currently looks for the index.html file in a few places (which >>> depend on platform); if it can't find it, it uses the documentation on >>> python.org. >>> >>> It should be too hard to change it to load the HTML Help viewer if it >>> finds the .chm file on Windows, and to still fall back to the HTML or >>> the online documentation if the .chm can't be found. >> >>Changing it is trivial, EditorWindow.help_url must point to Python23.chm >>(if it exists). I can do this. > > Beware that for .html you want webbrowser.open(url) and for .chm you > want os.startfile(url) > There used to be a patch in Idle-dev for this. > http://tinyurl.com/mj1s > It seems this patch is already applied in 2.3. >>Even nicer would be context-sensitive keyword help, but it seems IDLE >>doesn't support it, right? > > Standard .chm format already includes useful index data to do > context-sensitive search. A couple of years ago I made an Idle extension > to manage this. It wasn't difficult then, and I think it's even easier > now. The problem I found is that you pass the selected text (the one you > are looking for) to HTMLHelp system by calling a Win32 API. I know this, and how to do the windows side. What I meant was that I don't know enough about IDLE (or Tkinter) to implement the functionality to find the word under the cursor. > You'll need win32all installed. That's the way PythonWin works, by the way. > I guess this makes it a no-no for standard Python, right? > I don't think win32all wraps the HTMLHelp api, does it? OTOH, I'm waiting for a reason to include ctypes with core python ;-) > Regards, > -Hernan > Thomas From theller at python.net Tue Sep 16 13:47:06 2003 From: theller at python.net (Thomas Heller) Date: Tue Sep 16 13:46:46 2003 Subject: [Python-Dev] Windows installer and .pyc files mtime In-Reply-To: (Tim Peters's message of "Sat, 30 Aug 2003 22:29:35 -0400") References: Message-ID: <4qzck4md.fsf@python.net> "Tim Peters" writes: > [Thomas Heller] >> Temporary download url: >> >> http://starship.python.net/crew/theller/python-2.3.1-beta1.exe >> >> This is built from CVS 23maint branch, some hours ago. The uncommited >> changes I made are here, together with additional comments: >> >> http://www.python.org/sf/796919 >> >> Only slightly tested so far. > > Slightly more now, on Win98SE. Cool! I thought something was wrong because > the install on Win98SE went faster than I was used to. Turns out that's > just because we're installing 1,200+ fewer files than we used to (thanks to > shipping all the HTML docs in a single .chm file). > > Everything I tried worked fine. I think the *default* should be not to > compile .pyc files. Regardless of the default, popping up a DOS box during > compilation doesn't bother me at all. Great, I'll change it. But when there's an option to compile .pyc, shouldn't there also be an option for .pyo (and should it use -O or -OO)? Thomas From tim at zope.com Tue Sep 16 13:58:30 2003 From: tim at zope.com (Tim Peters) Date: Tue Sep 16 13:58:44 2003 Subject: [Python-Dev] RE: [Zope-Coders] core dump in Zope 2.7 test suite In-Reply-To: Message-ID: About Jeremy's segfault, we've debugged this in an IM session, and it's a bug in Python's (2.1, 2.2, 2.3, doesn't matter) unicodeobject.c. We'll fix it. You're most likely to see a segfault under a UCS4 build. Just another reason to upgrade to Windows . From tim at zope.com Tue Sep 16 14:13:30 2003 From: tim at zope.com (Tim Peters) Date: Tue Sep 16 14:13:44 2003 Subject: [Python-Dev] RE: [Zope-Coders] core dump in Zope 2.7 test suite In-Reply-To: Message-ID: [Tim] > ... > If some internal Unicode operation decided to allocate a short Unicode > string, but freed it before filling in any of the string bytes, I > suppose the keepalive optimization would retain a Unicode object with > uninitialized str space on the unicode free list. A subsequent > _PyUnicode_New could grab that and try to boost its ->str size. That > could explain it. And that turned out to be the case. One example (there are more) is in PyUnicode_DecodeASCII(): the local PyUnicodeObject *v; gets initialized: v = _PyUnicode_New(size); Suppose size is 1. Suppose the string coming in is "\xc8". The first iteration of the loop sets a "ordinal not in range(128)" error, and jumps to onError: onError: Py_XDECREF(v); unicode_dealloc() then stuffs v on unicode_freelist, and because the length "is small" (size == 1), v->str[] is retained, still holding uninitialized heap trash. A later _PyUnicode_New() grabs this off unicode_freelist, decides to boost the str space via unicode_resize(), and the latter blows up in the unicode_latin1[unicode->str[0]] == unicode)) { check because unicode->str[0] happens to be gigantically negative on Jeremy's box, and the preceding unicode->str[0] < 256 && check is too weak. Or is there an implicit assumption that Py_UNICODE is always an unsigned type (in which case, why isn't the literal 256U?; and in which case, it doesn't seem to be true on Jeremy's box). From tim at zope.com Tue Sep 16 14:47:00 2003 From: tim at zope.com (Tim Peters) Date: Tue Sep 16 14:48:12 2003 Subject: [Python-Dev] Windows installer and .pyc files mtime In-Reply-To: <4qzck4md.fsf@python.net> Message-ID: [Tim] >> Everything I tried worked fine. I think the *default* should be not >> to compile .pyc files. Regardless of the default, popping up a DOS >> box during compilation doesn't bother me at all. [Thomas Heller] > Great, I'll change it. But when there's an option to compile .pyc, > shouldn't there also be an option for .pyo (and should it use -O or > -OO)? Sounds like YAGNI. In the years I did the Windows installer, nobody asked for compilation. I haven't seen anyone here ask for it either, just a few people saying they think other people might want it. Since I wasn't one of them, you're going to have to ask someone who thinks they can channel these oddly unassertive users . From martin at v.loewis.de Tue Sep 16 15:52:26 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Sep 16 15:52:46 2003 Subject: [Python-Dev] Python and POSIX In-Reply-To: <46570000.1063709941@leeloo.intern.geht.de> References: <35230000.1063660079@leeloo.intern.geht.de> <46570000.1063709941@leeloo.intern.geht.de> Message-ID: Marc Recht writes: > I had a quick glance at the Solaris 8 headers and it seems that's > influenced by __EXTENSIONS__, too. So it _should_ be the same as > without the macros. While that is true, I would hope that you come to the conclusion that these macros can potentially do much more than simply hiding library functions. Instead, their *primary* purpose is to request Single Unix behaviour in case a platform implements both the standard behaviour, and the traditional one (regardless of which tradition the system follows). So I would be *very* careful to enable these macros only for a few translation units; I'm pretty sure that the "law" would be on the vendor's side if inconsistent usage of these defines would be found to cause problems, in Irix 27, or ValueBSD 3. > That's a bit harder since I don't have access to all "relevant" > systems. I could test on the open-source x86 platforms, though. And > IIRC CompileFarm has Solaris, too. If the change is only for 2.4, that would be fine - that code hopefully will get testing on several other platforms before 2.4 is released. > And I _believe_ that - since these changes came up with Python 2.3 > (with the exception of _POSIX_1_SOURCE and _POSIX_SOURCE) - that it > will break not much or anything at all if at least _XOPEN_SOURCE are > moved _XOPEN_SOURCE_EXTENDED to an internal header (or an internal > part of pyconfig.h). Believes should not be trusted too much when it comes to portability. If you can think of an obscure interpretation of some standard, it is almost certain that there is some vendor out there who has chosen that interpretation. Regards, Martin From martin at v.loewis.de Tue Sep 16 15:55:46 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Sep 16 15:56:21 2003 Subject: [Python-Dev] RE: [Zope-Coders] core dump in Zope 2.7 test suite In-Reply-To: References: Message-ID: "Tim Peters" writes: > Still, it doesn't look to me like the code *expected* that str could contain > uninitialized heap trash at this point, so I'd like someone who thinks they > understand this code to think about how that could happen (it apparently is > happening, so "beats me" isn't good enough ). I would like to see a stack trace. Regards, Martin From martin at v.loewis.de Tue Sep 16 16:00:27 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Sep 16 16:00:45 2003 Subject: [Python-Dev] RE: [Zope-Coders] core dump in Zope 2.7 test suite In-Reply-To: <1063729880.4868.8.camel@localhost.localdomain> References: <1063729880.4868.8.camel@localhost.localdomain> Message-ID: Jeremy Hylton writes: > > Still, it doesn't look to me like the code *expected* that str could contain > > uninitialized heap trash at this point, so I'd like someone who thinks they > > understand this code to think about how that could happen (it apparently is > > happening, so "beats me" isn't good enough ). > > I certainly don't understand the code yet. The code you quote is about the "singleton" Unicode character (i.e. those with code points < 256). If you create a Unicode object S with len(S)==1, and ord(S) < 256, you don't necessarily get a new object, but you may get an existing (cached) one. If you then apply PyUnicde_Resize to that object, you get an error because resizing that object would require changing all shared copies, which is bad because Unicode object are immutable. I agree with Tim that, in no case, resize should be called for a garbage string - only valid strings (i.e. with truly allocated memory) should ever be resized. Regards, Martin From martin at v.loewis.de Tue Sep 16 16:05:18 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Sep 16 16:05:50 2003 Subject: [Python-Dev] RE: [Zope-Coders] core dump in Zope 2.7 test suite In-Reply-To: References: Message-ID: "Tim Peters" writes: > Or is there an implicit assumption that Py_UNICODE is always an > unsigned type (in which case, why isn't the literal 256U? Likely, the author of the code was subconsciously assuming so. However, that assumption is invalid, and sharing should only happen for ordinals in range(256). Regards, Martin From tim.one at comcast.net Tue Sep 16 16:46:31 2003 From: tim.one at comcast.net (Tim Peters) Date: Tue Sep 16 16:46:42 2003 Subject: [Python-Dev] RE: [Zope-Coders] core dump in Zope 2.7 test suite In-Reply-To: Message-ID: [Martin v. L?wis] > ... > I agree with Tim that, in no case, resize should be called for a > garbage string - only valid strings (i.e. with truly allocated memory) > should ever be resized. unicode->str pointed to properly allocated memory in the failing cases, but to uninitialized allocated memory. So str[0] was just some random collection of bits from the heap, and checking random bits < 256 wasn't enough to weed out the cases where the random bits happened to look like a negative integer. For Jeremy, str[0] was 0xcbcbcbcb (for me it was 0xcbcb), and using that as an index blew up. ... [Tim] >> Or is there an implicit assumption that Py_UNICODE is always an >> unsigned type (in which case, why isn't the literal 256U? [Martin] > Likely, the author of the code was subconsciously assuming > so. However, that assumption is invalid, and sharing should only > happen for ordinals in range(256). I've since patched Jeremy's patch to try to live with that Py_UNICODE may be a signed type too. From mal at lemburg.com Tue Sep 16 17:22:26 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Sep 16 17:22:28 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: References: Message-ID: <3F677F12.80409@lemburg.com> tim_one@users.sourceforge.net wrote: > Update of /cvsroot/python/python/dist/src/Objects > In directory sc8-pr-cvs1:/tmp/cvs-serv17421/Objects > > Modified Files: > unicodeobject.c > Log Message: > On c.l.py, Martin v. L?wis said that Py_UNICODE could be of a signed type, > so fiddle Jeremy's fix to live with that. Also added more comments. Note that the implementation will bomb in several places if Py_UNICODE is a signed type. Py_UNICODE was never intended to be a signed type, so the proper fix would be to add logic so that Py_UNICODE gets forced to be an unsigned type. The only case where Py_UNICODE could become signed is via a compiler that defines wchar_t to be signed -- rather unlikely. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Sep 16 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From tim.one at comcast.net Tue Sep 16 17:39:58 2003 From: tim.one at comcast.net (Tim Peters) Date: Tue Sep 16 17:40:04 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: <3F677F12.80409@lemburg.com> Message-ID: [M.-A. Lemburg] > Note that the implementation will bomb in several places if > Py_UNICODE is a signed type. > > Py_UNICODE was never intended to be a signed type, so the proper > fix would be to add logic so that Py_UNICODE gets forced to be an > unsigned type. Jeremy believed Py_UNICODE was already an unsigned type on his box, and that was the box with the segfaults. I don't know. Comparison with a signed int confused the issues to the point where we gave up and just fixed it . > The only case where Py_UNICODE could become signed is via > a compiler that defines wchar_t to be signed -- rather unlikely. The C standard requires wchar_t to be an integer type, but doesn't constrain it further to an unsigned integer type. I don't like relying on non-standard assumptions in cases where there's little-to-no cost in not relying on them. For example, the cast I put in with this patch is probably a nop on most boxes, just forcing an unsigned comparison (which must have been the original intent, if Py_UNICODE was assumed to be an unsigned type). From gjc at inescporto.pt Tue Sep 16 18:04:42 2003 From: gjc at inescporto.pt (Gustavo J A M Carneiro) Date: Tue Sep 16 18:06:34 2003 Subject: Solutions for LC_NUMERIC, was Re: [Python-Dev] Re: Be Honest about LC_NUMERIC [REPOST] In-Reply-To: References: <20030903134155.GB649@async.com.br> Message-ID: <1063749875.20959.15.camel@emperor.homelinux.net> A Qua, 2003-09-03 ?s 22:32, Martin v. L?wis escreveu: > Christian Reis writes: > > > "On glibc-based systems, Python is safe from runtime changes in the > > LC_NUMERIC locale category. On other platforms, changing LC_NUMERIC > > at runtime may cause float conversion routines to fail. This may > > impact the Python interpreter internally." > > I could accept such a limitation. However, I maintain that the aspects > must be fully understood, and I agree with Guido that they must be > documented in the documentation proper (i.e. as comments in the source > code, or the user documentation). > > So I suggest that you proceed with the PEP and the implementation > as-is. The PEP would need to be enhanced to summarize this discussion > (perhaps listing the alternatives that you just mentioned); the patch > would need to be enhanced to be more simple, readable, and in the > style of the rest of the Python source code, and would need to include > documentation changes and test cases (although some test cases > probably already exist). I have updated the patch[1], with James Henstridge's cleaned up code (and some more cleanup). I am aware it still needs more comments and test cases, but interested parties might want to start looking at it. Please note that, after some discussion, we have concluded it is better not to use glibc locale extensions to implement thread-safe float conversion. The main reasons are: (1) simplicity (no ifdefs); (2) consistency between glibc and non-glibc platforms. We should accept the fact python programs should not be allowed change locale after creating threads, period. Perhaps python itself should raise an exception if this is ever attempted. This is subject to discussion, of course! ;) Regards. [1] http://sourceforge.net/tracker/download.php?group_id=5470&atid=305470&file_id=61688&aid=774665 -- Gustavo J. A. M. Carneiro From lists at webcrunchers.com Tue Sep 16 16:50:09 2003 From: lists at webcrunchers.com (John D.) Date: Tue Sep 16 18:44:29 2003 Subject: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6 In-Reply-To: <005a01c37962$b94a0e20$30bc958d@oemcomputer> References: <001d01c3795d$c2ba1400$30bc958d@oemcomputer> <16226.6197.573407.585746@montanaro.dyndns.org> Message-ID: Raymond writes: >[Raymond] >> >> > IIRC, dbhash and bsddb don't affect the Apple MacIntosh users. >[Skip] >> I don't know what you mean here. Do you mean Mac OS 9 or Mac OS X? I >> use bsddb all the time on Mac OS X. > >Hmm, I'll have to fix the docs too. They currently list bsddb as being >only for Unix and Windows. While you are at it, please add a few more examples of how to use "bsddb3" in particular the "dbtables" section. Like how do you select all from a table. JD From tim at zope.com Tue Sep 16 21:34:50 2003 From: tim at zope.com (Tim Peters) Date: Tue Sep 16 21:35:40 2003 Subject: [Python-Dev] RE: [Zope-CVS] CVS: Products/Ape/lib/apelib/zodb3 - serializers.py:1.3storage.py:1.8 In-Reply-To: <200309162100.h8GL08H01454@cvs.baymountain.com> Message-ID: [Shane Hathaway] > Modified Files: > serializers.py storage.py > Log Message: > Fixed Python 2.3 compatibility. > > - If you append to a list while it is being pickled, Python 2.3 will > pick up the new items. Previous versions of Python did not. How do you grow a list while it's being pickled? Must be another thread doing that, and if so that's mighty dubious. If you're pickling to a genuine file object, then cPickle releases the GIL around its fwrite() calls, so that's one way. But in the context of the patch: + p.persistent_id = lambda ob: None # Stop recording references p.dump(unmanaged) s = outfile.getvalue() event.addUnmanagedPersistentObjects(unmanaged) the use of .getvalue() suggests the pickle target isn't a real file. Then cPickle never releases the GIL (cStringIO doesn't either), and no other thread should dare muck with a Python object (like the Python list getting pickled) without holding the GIL. However this ends up happening, if you instead shrink a list while it's being pickled, Pythons earlier than 2.3 can end up trying to pickle random recycled memory. Their cPickles captured the list length once at the start of pickling the list, and that coding optimization is appropriate only if the GIL is never released. 2.3 uses the iteration protocol instead, so "sees" dynamic changes in the list size. Copying Python-Dev because this may reveal a new mountain of ways to crash the interpreter: mutate objects in devious ways while they're getting cPickled. From bac at OCF.Berkeley.EDU Tue Sep 16 23:09:44 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Tue Sep 16 23:09:49 2003 Subject: [Python-Dev] Trying to fix time.tzset detection for 2.3.1 In-Reply-To: <3F674015.8090801@ocf.berkeley.edu> References: <3F669AF3.8050605@ocf.berkeley.edu> <16231.7058.987290.575924@montanaro.dyndns.org> <3F674015.8090801@ocf.berkeley.edu> Message-ID: <3F67D078.1000106@ocf.berkeley.edu> Brett C. wrote: > Skip Montanaro wrote: > >> Brett> Well, Kurt Kaiser wrote up a patch ( >> Brett> http://www.python.org/sf/762934 ) that seems sound. I have >> Brett> gotten zero responses from anyone to test it on Red Hat 6.1 >> and a >> Brett> remember that some people here were helping to test last >> time (I >> Brett> also remember this led to the creation of a wiki page listing >> Brett> platforms various people could test things on but I can't find >> Brett> the page). >> >> That would be: >> >> http://www.python.org/cgi-bin/moinmoin/PythonTesters >> >> I see Kurt Kaiser as the only person listed who can test for RH < 6.2. >> Perhaps you should broadcast a call for testers to c.l.py, referring >> to the >> above page and asking in particular for RH 6.1 folks to chime in. >> > > Jordan Krushen pointed out that the patch is actually opened against RH > 6.2, not 6.1 like I originally said. But I don't think it is that big > of an issue since there is a bug report for this against SuSE. I think > the key thing here is an older version of glibc since that would be what > defines the tzset function. > Well, Jordan emailed me personally with the result of running RH 6.1 and RH 6.2 under VMWare; both passed (thanks, Jordan)! So, does anyone have any objections to me checking this in, or do people feel this is not enough testing to show the patch does what it needs to? And a question on patching configure.pre.in: is that all I have to patch or do I need to run reconfigure and then also patch configure.in? -Brett From martin at v.loewis.de Wed Sep 17 00:26:57 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Wed Sep 17 00:27:20 2003 Subject: [Python-Dev] Trying to fix time.tzset detection for 2.3.1 In-Reply-To: <3F67D078.1000106@ocf.berkeley.edu> References: <3F669AF3.8050605@ocf.berkeley.edu> <16231.7058.987290.575924@montanaro.dyndns.org> <3F674015.8090801@ocf.berkeley.edu> <3F67D078.1000106@ocf.berkeley.edu> Message-ID: "Brett C." writes: > And a question on patching configure.pre.in: is that all I have to > patch or do I need to run reconfigure and then also patch configure.in? There is no configure.pre.in in Python. Martin From hernan at orgmf.com.ar Wed Sep 17 02:53:18 2003 From: hernan at orgmf.com.ar (=?us-ascii?Q?Hernan_Martinez_Foffani?=) Date: Wed Sep 17 02:55:24 2003 Subject: [Python-Dev] RE: HTMLHelp for Py2.3.1 In-Reply-To: <8yook4or.fsf@python.net> Message-ID: [Thomas] >[Hernan] >> Standard .chm format already includes useful index data to do >> context-sensitive search. A couple of years ago I made an Idle >> extension to manage this. It wasn't difficult then, and I think it's >> even easier now. The problem I found is that you pass the selected >> text (the one you are looking for) to HTMLHelp system by calling a >> Win32 API. > I know this, and how to do the windows side. What I meant was that I > don't know enough about IDLE (or Tkinter) to implement the > functionality to find the word under the cursor. I'm pretty sure that some IDLE extension used to have such functionality. It wasn't that difficult. It was a patch I did a couple of years now and wasn't finish. If I can find it I'll tell you my approach. >> You'll need win32all installed. That's the way PythonWin works, by >> the way. I guess this makes it a no-no for standard Python, right? >> > I don't think win32all wraps the HTMLHelp api, does it? Yes, it does. I think Mark(?) did it for his PythonWin IDE. > OTOH, I'm waiting for a reason to include ctypes with core python ;-) So there is one. ;-) -Hernan From mal at lemburg.com Wed Sep 17 04:08:13 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Sep 17 04:08:23 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: References: Message-ID: <3F68166D.5030206@lemburg.com> Tim Peters wrote: > [M.-A. Lemburg] > >>Note that the implementation will bomb in several places if >>Py_UNICODE is a signed type. >> >>Py_UNICODE was never intended to be a signed type, so the proper >>fix would be to add logic so that Py_UNICODE gets forced to be an >>unsigned type. > > Jeremy believed Py_UNICODE was already an unsigned type on his box, and that > was the box with the segfaults. I don't know. Comparison with a signed int > confused the issues to the point where we gave up and just fixed it . That sounds more like compiler bug to me. What's bothering me is that such compares are done in other places too, so a more general solution would be better. >>The only case where Py_UNICODE could become signed is via >>a compiler that defines wchar_t to be signed -- rather unlikely. > > The C standard requires wchar_t to be an integer type, but doesn't constrain > it further to an unsigned integer type. I don't like relying on > non-standard assumptions in cases where there's little-to-no cost in not > relying on them. For example, the cast I put in with this patch is probably > a nop on most boxes, just forcing an unsigned comparison (which must have > been the original intent, if Py_UNICODE was assumed to be an unsigned type). No question there, but wouldn't it be easier to test such a platform and then fallback to "unigned int" in case wchar_t is found to be a signed value ? -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Sep 17 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From shane at zope.com Wed Sep 17 08:34:50 2003 From: shane at zope.com (Shane Hathaway) Date: Wed Sep 17 08:35:31 2003 Subject: [Python-Dev] RE: [Zope-CVS] CVS: Products/Ape/lib/apelib/zodb3 - serializers.py:1.3storage.py:1.8 In-Reply-To: References: Message-ID: On Tue, 16 Sep 2003, Tim Peters wrote: > [Shane Hathaway] > > Modified Files: > > serializers.py storage.py > > Log Message: > > Fixed Python 2.3 compatibility. > > > > - If you append to a list while it is being pickled, Python 2.3 will > > pick up the new items. Previous versions of Python did not. > > How do you grow a list while it's being pickled? Must be another thread > doing that, and if so that's mighty dubious. If you're pickling to a > genuine file object, then cPickle releases the GIL around its fwrite() > calls, so that's one way. But in the context of the patch: > > + p.persistent_id = lambda ob: None # Stop recording references > p.dump(unmanaged) > s = outfile.getvalue() > event.addUnmanagedPersistentObjects(unmanaged) Actually, the persistent_id() function was appending to the "unmanaged" list on each call. Here is a simplified version of the mistake: unmanaged = [] def persistent_id(ob): unmanaged.append(ob) return None s = StringIO() p = Pickler(s) p.persistent_id = persistent_id p.dump(unmanaged) Python 2.2 apparently records the length of the list being pickled before analyzing its contents. This meant that my mistake went unnoticed. (I did not intend to modify "unmanaged" while it was being pickled! There are two passes in the actual code; the "unmanaged" list is built up in the first pass and pickled in the second pass.) Python 2.3, on the other hand, picks up each new item as it is appended, resulting in a finite but very long loop. > Copying Python-Dev because this may reveal a new mountain of ways to crash > the interpreter: mutate objects in devious ways while they're getting > cPickled. I wouldn't be too concerned. Only people who do crazy things with persistent_id() hooks will be affected. ;-) Shane From neal at metaslash.com Wed Sep 17 09:13:54 2003 From: neal at metaslash.com (Neal Norwitz) Date: Wed Sep 17 09:13:58 2003 Subject: [Python-Dev] Trying to fix time.tzset detection for 2.3.1 In-Reply-To: <3F67D078.1000106@ocf.berkeley.edu> References: <3F669AF3.8050605@ocf.berkeley.edu> <16231.7058.987290.575924@montanaro.dyndns.org> <3F674015.8090801@ocf.berkeley.edu> <3F67D078.1000106@ocf.berkeley.edu> Message-ID: <20030917131354.GC1211@epoch.metaslash.com> On Tue, Sep 16, 2003 at 08:09:44PM -0700, Brett C. wrote: > > And a question on patching configure.pre.in: is that all I have to patch > or do I need to run reconfigure and then also patch configure.in? As Martin said, there is no configure.pre.in. The file is configure.in and you need to regenerate configure and pyconfig.h.in with autoconf and autoheader (version 2.57). Neal From lomax at pumpichank.com Wed Sep 17 09:18:26 2003 From: lomax at pumpichank.com (Frnak "Sealmbcrd Eggs" Lomax) Date: Wed Sep 17 09:18:31 2003 Subject: [Python-Dev] Fsrit Propttoye: SCLABMRE Message-ID: <16232.24354.366003.171152@gargle.gargle.HOWL> I hvae jsut fhiseind rniedag a Shdlaost atrlcie from Mnoday, Sbepemetr 15th, aobut an ingtriniug sdtuy from Cgbdirmae Uinevtrsiy, dciinesrbg how wodrs with thier itienrors selmrbacd can siltl be rdbaleae. Bcaeuse of the tioeznme that I lvie in, an initettrment dfceet in my clbae medom, and the lneiignrg atfer-ecfefts of my lsat "mohsroum salad exntarvzagaa", I am aaylws two days bhneid on the Web. So suentnd was I at the icoinatpimls of this study, taht I cunlqtneesoy decedid to doetve my etrnie lfie and gtelray aeamssd fnrtoue on cunitodcng ftrhuer rasreech and dnpeeelmvot into tihs lifnsaevig tghlocnoey. As a cross-eeyd dxsiyelc, I konw that teshe cilmas of svotailan are niehetr hlroyepbe nor bmobaistc over-daaomtiazitrn. Pnseetred hree for your rveiew, aaovpprl and disiscsuon, is our poyptotre sytsem, caelld SLABCRME for SeCren Rslmnebeig Aacrihc Mnoaaluptiin of BetaoLd Ecuitolon. I have scnie bugen a meosdt sutartp based in Aceicdnt Maarnlyd, and we will soon be reeianlsg an Olutook puglin, anolg with a sftorwae devmoeelpnt kit, and oethr digintoasc tloos. Smoe of our tlgeocnhoy wlil nallaruty be open sruoce, but I do ietnnd and ecpext to frutehr etnxed my hoougmuns wtaleh by sinlelg sftorawe lcseiens and srhink wrapped ratiel pocrudt. Psaele cactnot us for more iomrfaiontn. I'm aslo in enpommelyt duicnssioss wtih the Tmboit; I ecepxt that he can bosot the pmecarrofne of our pooyptrte's itmpeanetolmin salreolefvd, at wcihh piont we will be pthincig emdbeded ssmeyts to the mfntacuuearrs of ruoters, eiltncreoc trprwyeiets, psnreoal dtaiigl astsntsais, eorctinlec cepmutor coltlerond tersotas, maet prbeos, RFC 1149 digtail cemara pheons, and nxet gniaoetern wopehoe csionuhs. hinpog-to-be-deotersyd-by-moroscift-ly y'rs, -fnark -------------- next part -------------- #! /usr/bin/env python # Author: Frank "Topsy Turvy" Lomax """Slmbcrae the letrtes in a word. Tihs was seen on Soalhsdt, so it msut be ture: Ardcnoicg to resraceh at Cgmdrabie Uesnirtivy, it dsoen't matter in what odrer the lettres in a wrod are, the olny iotrmpant tnhig is taht the fsrit and lsat lteter be at the rhigt pcale. The rest can be a toatl mess and you can sltil raed it wuhoitt peolbrm. This is bseacue we do not raed eervy ltteer by it self but the wrod as a wohle. Cireheo. """ import re import sys from random import shuffle EMPTYSTRING = '' def scrambleword(word): if len(word) <= 2: return word middle = list(word[1:-1]) shuffle(middle) return word[0] + EMPTYSTRING.join(middle) + word[-1] def main(): text = sys.stdin.read() words = re.split(r'(\W+)', text) while words: sys.stdout.write(scrambleword(words.pop(0))) if words: sys.stdout.write(words.pop(0)) if __name__ == '__main__': main() From jeremy at alum.mit.edu Wed Sep 17 10:12:37 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Wed Sep 17 10:12:47 2003 Subject: [Python-Dev] RE: [Zope-Coders] core dump in Zope 2.7 test suite In-Reply-To: References: Message-ID: <1063807956.10299.31.camel@localhost.localdomain> I've applied the latest patch to the head, 2.3, and 2.2 branches. It looks like it isn't necessary on the 2.1 branch. While the small-Unicode-keepalive code is still there, the length-one Unicode strings aren't immortal. Since there is no-special case code looking at str[0], it appears unnecessary to initialize it. Jeremy From jepler at unpythonic.net Wed Sep 17 10:59:11 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Wed Sep 17 10:59:26 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: <3F68166D.5030206@lemburg.com> References: <3F68166D.5030206@lemburg.com> Message-ID: <20030917145905.GF29089@unpythonic.net> On Wed, Sep 17, 2003 at 10:08:13AM +0200, M.-A. Lemburg wrote: > No question there, but wouldn't it be easier to test such a platform > and then fallback to "unigned int" in case wchar_t is found to > be a signed value ? This inaccurate comment created by configure [from 2.3b1] implies that sds/2 already expects wchar_t to be unsigned if it is to be usable: if test "$unicode_size" = "$ac_cv_sizeof_wchar_t" then PY_UNICODE_TYPE="wchar_t" AC_DEFINE(HAVE_USABLE_WCHAR_T, 1, [Define if you have a useable wchar_t type defined in wchar.h; useable means wchar_t must be 16-bit unsigned type. (see Include/unicodeobject.h).]) AC_DEFINE(PY_UNICODE_TYPE,wchar_t) ... but configure doesn't actually check for signedness, and the "16-bit" part is inaccurate. (it must be 16 bits for ucs-2, or 32-bits for ucs-4) Jeff From jeremy at alum.mit.edu Wed Sep 17 11:05:27 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Wed Sep 17 11:05:31 2003 Subject: [Python-Dev] pending deprecation warning for Set.update Message-ID: <1063811126.10299.42.camel@localhost.localdomain> It looks like current CVS has grown a pending deprecation warning for the update() method of Sets and that this change has been backported to the 2.3 branch. It seems to me to be bad form to add a warning in a maintenance release. The update() method was new in 2.3 and documented in the 2.3 docs. So people are going to write code using update() and then update to 2.3.1 and get warnings? I'm also not clear on the rationale for the warning. update() seems to match the semantics of dict.update(), and it less verbose than the suggested alternative -- union_update(). Jeremy From skip at pobox.com Wed Sep 17 11:51:04 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Sep 17 11:51:16 2003 Subject: [Python-Dev] pending deprecation warning for Set.update In-Reply-To: <1063811126.10299.42.camel@localhost.localdomain> References: <1063811126.10299.42.camel@localhost.localdomain> Message-ID: <16232.33512.613600.989386@montanaro.dyndns.org> Jeremy> It looks like current CVS has grown a pending deprecation Jeremy> warning for the update() method of Sets ... +1 on everything Jeremy wrote. Skip From python at rcn.com Wed Sep 17 12:02:00 2003 From: python at rcn.com (Raymond Hettinger) Date: Wed Sep 17 12:02:09 2003 Subject: [Python-Dev] pending deprecation warning for Set.update References: <1063811126.10299.42.camel@localhost.localdomain> Message-ID: <003401c37d35$0784fac0$9c3dc797@oemcomputer> > It looks like current CVS has grown a pending deprecation warning for > the update() method of Sets and that this change has been backported to > the 2.3 branch. It seems to me to be bad form to add a warning in a > maintenance release. To deprecate would be bad form. To provide as much advanced warning as possible is good form. That is much better than having 2.3.1 users spending the next year developing code with update() and then having it deprecated in 2.4. They might as well know right now. > I'm also not clear on the rationale for the warning. update() seems to > match the semantics of dict.update(), and it less verbose than the > suggested alternative -- union_update(). Part of the reason for initially implementing sets in Python instead of C and for not having it as a builtin was was that the API experimental. This change was based in part upon user feedback solicited from the newsgroup. The verbosity issue weighs less because there is the __ior__ operator alternative: a |= b. That is succinct as you can get. And the long form is nothing compared to Set.symmetric_difference_update() ;-) Raymond ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From python at rcn.com Wed Sep 17 12:18:15 2003 From: python at rcn.com (Raymond Hettinger) Date: Wed Sep 17 12:18:21 2003 Subject: [Python-Dev] pending deprecation warning for Set.update References: <1063811126.10299.42.camel@localhost.localdomain> <16232.33512.613600.989386@montanaro.dyndns.org> Message-ID: <004e01c37d37$4cd979a0$9c3dc797@oemcomputer> > Jeremy> It looks like current CVS has grown a pending deprecation > Jeremy> warning for the update() method of Sets ... [Skip] > +1 on everything Jeremy wrote. One other thought: having a PendingDeprecationWarning is somewhat innocuous. By default, those warnings are filtered out (set to Ignore). You have to ask to see them. It would be much different for a DeprecationWarning. It is pretty conservative to issue a pending warning now, a deprecation warning in 2.4, and remove the duplicate method in 2.5. This is doubly true for a brand new module that has only a minimal existing code base. Not bringing it up for 12 to18 months is, IMHO, much more impactful. Raymond From nas-python at python.ca Wed Sep 17 12:44:35 2003 From: nas-python at python.ca (Neil Schemenauer) Date: Wed Sep 17 12:37:16 2003 Subject: [Python-Dev] pending deprecation warning for Set.update In-Reply-To: <003401c37d35$0784fac0$9c3dc797@oemcomputer> References: <1063811126.10299.42.camel@localhost.localdomain> <003401c37d35$0784fac0$9c3dc797@oemcomputer> Message-ID: <20030917164435.GA19007@mems-exchange.org> On Wed, Sep 17, 2003 at 12:02:00PM -0400, Raymond Hettinger wrote: > The verbosity issue weighs less because there is the __ior__ operator > alternative: a |= b. That is succinct as you can get. >>> from sets import Set >>> s = Set() >>> s |= [1, 2, 3] Traceback (most recent call last): File "", line 1, in ? File "/www/plat/python2.3/lib/python2.3/sets.py", line 420, in __ior__ self._binary_sanity_check(other) File "/www/plat/python2.3/lib/python2.3/sets.py", line 313, in _binary_sanity_check raise TypeError, "Binary operation only permitted between sets" TypeError: Binary operation only permitted between sets >>> s.update([1, 2, 3]) >>> s Set([1, 2, 3]) >>> I quite a few instances of update() that will have to be changed to union_update(). Neil From raymond.hettinger at verizon.net Wed Sep 17 13:46:46 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Wed Sep 17 13:46:55 2003 Subject: [Python-Dev] Py2.3.1 Message-ID: <00ce01c37d43$aa879940$9c3dc797@oemcomputer> Are we on for doing a micro release this week? Raymond Hettinger From jepler at unpythonic.net Wed Sep 17 15:40:29 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Wed Sep 17 15:40:33 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: <20030917145905.GF29089@unpythonic.net> References: <3F68166D.5030206@lemburg.com> <20030917145905.GF29089@unpythonic.net> Message-ID: <20030917194029.GI29089@unpythonic.net> On Wed, Sep 17, 2003 at 09:59:11AM -0500, Jeff Epler wrote: > This inaccurate comment created by configure [from 2.3b1] implies that > _____ already expects wchar_t to be unsigned if it is to be usable: urm, I meant to refer to Python, not a software product most of you are lucky enough to have never heard of... Jeff From martin at v.loewis.de Wed Sep 17 17:39:32 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Wed Sep 17 17:39:53 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: <3F68166D.5030206@lemburg.com> References: <3F68166D.5030206@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > No question there, but wouldn't it be easier to test such a platform > and then fallback to "unigned int" in case wchar_t is found to > be a signed value ? Why is it important that Py_UNICODE is unsigned? Regards, Martin From vinay_sajip at yahoo.co.uk Wed Sep 17 17:43:58 2003 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed Sep 17 17:42:13 2003 Subject: [Python-Dev] Consistent logging in the standard library Message-ID: <005101c37d64$d37270c0$652b6992@alpha> A recent thread has discussed integrating logging with the rest of the standard library. Among other things, a question has arisen about what logging channel names standard library modules should use, and possible clashes with channel names used by applications. I'd like to propose that some portion of the namespace be reserved for use by logging in the standard library. If acceptable, all that would need to be done would be to update the documentation to indicate the namespace reservation. The actual use of the namespace could be implemented as and when each module is changed to use the logging module. The scheme of having all standard library logging under one umbrella allows applications to be configured so that all library-level messages can be routed to specific handlers, filtered simply at one point, etc. My preference for namespace reservation is that any logger name beginning with "python." should be reserved for use by the standard distribution. I think that each module should use an eponymous logging channel name rooted in the hierarchy for the Python library - e.g. asyncore would use "python.asyncore", etc. Matthew Barnes, who kicked off the original thread, prefers "stdlib" as the prefix to use, rather than "python". Comments, please! Vinay Sajip From martin at v.loewis.de Wed Sep 17 17:45:14 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Wed Sep 17 17:45:28 2003 Subject: [Python-Dev] pending deprecation warning for Set.update In-Reply-To: <004e01c37d37$4cd979a0$9c3dc797@oemcomputer> References: <1063811126.10299.42.camel@localhost.localdomain> <16232.33512.613600.989386@montanaro.dyndns.org> <004e01c37d37$4cd979a0$9c3dc797@oemcomputer> Message-ID: "Raymond Hettinger" writes: > It is pretty conservative to issue a pending warning now, a deprecation > warning in 2.4, and remove the duplicate method in 2.5. No, it is *not* conservative to add a warning in a maintenance branch, pending or not. Even changing the documentation only would not be conservative. It might be sensible (which I think it is not), but it is not conservative. Regards, Martin From barry at python.org Wed Sep 17 17:56:50 2003 From: barry at python.org (Barry Warsaw) Date: Wed Sep 17 17:56:41 2003 Subject: [Python-Dev] Consistent logging in the standard library In-Reply-To: <005101c37d64$d37270c0$652b6992@alpha> References: <005101c37d64$d37270c0$652b6992@alpha> Message-ID: <1063835810.14316.20.camel@geddy> On Wed, 2003-09-17 at 17:43, Vinay Sajip wrote: > Matthew Barnes, who kicked off the original thread, prefers "stdlib" as the > prefix to use, rather than "python". What about _ (underscore)? One of my half-baked totally unoriginal PEP ideas for Python 2.4 would be a syntax for forcing imports from the standard library. E.g. say I have a foo package, and inside there I've got a foo/logging.py module. Say there's also a foo/bar.py module and it wants to import the global logging package. How can it do that? "import logging" does a relative import. So the idea would be to be able to write "import _.logging" and definitely get the global logging package. Using underscore in the logger would mirror this mnemonic for globalness. -Barry From azaidi at vsnl.com Wed Sep 17 18:11:35 2003 From: azaidi at vsnl.com (Arsalan Zaidi) Date: Wed Sep 17 18:12:53 2003 Subject: [Python-Dev] Consistent logging in the standard library References: <005101c37d64$d37270c0$652b6992@alpha> Message-ID: <016801c37d68$b2169100$31469cca@LocalHost> Vinay Sajip wrote: > I'd like to propose that some portion of the namespace be reserved > for use by logging in the standard library. If acceptable, all that > would need to be done would be to update the documentation to > indicate the namespace reservation. The actual use of the namespace > could be implemented as and when each module is changed to use the > logging module. > > The scheme of having all standard library logging under one umbrella > allows applications to be configured so that all library-level > messages can be routed to specific handlers, filtered simply at one > point, etc. I agree with this and was infact about to propose this to Matthew Barnes a short while ago. Good thing I read the list first. :-) > My preference for namespace reservation is that any logger name > beginning with "python." should be reserved for use by the standard > distribution. I think that each module should use an eponymous > logging channel name rooted in the hierarchy for the Python library - > e.g. asyncore would use "python.asyncore", etc. > > Matthew Barnes, who kicked off the original thread, prefers "stdlib" > as the prefix to use, rather than "python". No strong preference either way, but if someone were to hold a gun to my head, I'd scream... And then express my preference for 'python' as the root. But that's just the Java talking :-) --Arsalan From azaidi at vsnl.com Wed Sep 17 18:14:18 2003 From: azaidi at vsnl.com (Arsalan Zaidi) Date: Wed Sep 17 18:15:32 2003 Subject: [Python-Dev] Consistent logging in the standard library References: <005101c37d64$d37270c0$652b6992@alpha> Message-ID: <017201c37d69$14898220$31469cca@LocalHost> Vinay Sajip wrote: > I'd like to propose that some portion of the namespace be reserved > for use by logging in the standard library. If acceptable, all that > would need to be done would be to update the documentation to > indicate the namespace reservation. The actual use of the namespace > could be implemented as and when each module is changed to use the > logging module. > > The scheme of having all standard library logging under one umbrella > allows applications to be configured so that all library-level > messages can be routed to specific handlers, filtered simply at one > point, etc. I agree with this and was infact about to propose this to Matthew Barnes a short while ago. Good thing I read the list first. :-) > My preference for namespace reservation is that any logger name > beginning with "python." should be reserved for use by the standard > distribution. I think that each module should use an eponymous > logging channel name rooted in the hierarchy for the Python library - > e.g. asyncore would use "python.asyncore", etc. > > Matthew Barnes, who kicked off the original thread, prefers "stdlib" > as the prefix to use, rather than "python". No strong preference either way, but if someone were to hold a gun to my head, I'd scream... And then express my preference for 'python' as the root. But that's just the Java talking :-) Someone proposed an '_'. I don't know. My gut feeling is that it's too cryptic and Perl like. If you must have something, call it 'base' or 'python' or even 'stdlib'. But what do I know... --Arsalan From azaidi at vsnl.com Wed Sep 17 18:24:26 2003 From: azaidi at vsnl.com (Arsalan Zaidi) Date: Wed Sep 17 18:25:33 2003 Subject: [Python-Dev] Consistent logging in the standard library References: <005101c37d64$d37270c0$652b6992@alpha> <016801c37d68$b2169100$31469cca@LocalHost> Message-ID: <019501c37d6a$7bd57d20$31469cca@LocalHost> I've accidently sent two slightly different emails on the same topic... Read the one timestamped 6:15 am. I know; my bad. No cookies for me for the next 24 hours... --Arsalan From skip at pobox.com Wed Sep 17 18:29:45 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Sep 17 18:29:59 2003 Subject: [Python-Dev] Consistent logging in the standard library In-Reply-To: <1063835810.14316.20.camel@geddy> References: <005101c37d64$d37270c0$652b6992@alpha> <1063835810.14316.20.camel@geddy> Message-ID: <16232.57433.388130.573539@montanaro.dyndns.org> Barry> So the idea would be to be able to write "import _.logging" and Barry> definitely get the global logging package. Using underscore in Barry> the logger would mirror this mnemonic for globalness. The connection seems tenuous, at best. If I log messages to syslog and and that file is later viewed or summarized by a third party, "python" seems a lot more useful than "_" or even "stdlib". Skip From cs1spw at bath.ac.uk Wed Sep 17 19:57:43 2003 From: cs1spw at bath.ac.uk (Simon Willison) Date: Wed Sep 17 19:56:58 2003 Subject: [Python-Dev] Improving the CGI module Message-ID: <3F68F4F7.6020601@bath.ac.uk> Hi all, I've recently started using Python for server side web development, and I've been running in to some limitations in the CGI module. The most critical of these is that the module provides no method of determining whether form data originated in a query string (GET data) or a POSTed form. The module also provides very little in the way of support for common CGI related operations such as sending common headers and parsing and sending cookies; the Cookie module provides parsing capabilities but requires additional glue code to extract cookie data from the CGI environment. I have previous experience with PHP, which has a number of extremely convenient mechanisms for dealing with form input and cookies that I sorely miss when working with Python. I have started work on porting some of these capabilities, but it struck me that several of the ideas represented would make valuable additions to the Python standard library itself. I have been working with Python for over a year, but I have no real experience of the Python development process. Is this list the right place to discuss potential improvements to the CGI module, or should I work independently and construct a PEP? I noticed that PEP 222 already exists relating to "web library enhancements", but it has a 'Deferred' status and appears not to have been modified since December 2000. Best regards, Simon Willison http://simon.incutio.com/categories/python/ From janssen at parc.com Wed Sep 17 20:15:57 2003 From: janssen at parc.com (Bill Janssen) Date: Wed Sep 17 20:16:25 2003 Subject: [Python-Dev] Improving the CGI module In-Reply-To: Your message of "Wed, 17 Sep 2003 16:57:43 PDT." <3F68F4F7.6020601@bath.ac.uk> Message-ID: <03Sep17.171602pdt."58611"@synergy1.parc.xerox.com> Simon, I sent the following out last week, and I'm currently working on a Web page and sample charter for a Web-SIG devoted to improving the Python Web support. Let's get together and do it! Bill ------------------------------------------ Bill Janssen (650) 812-4763 FAX: (650) 812-4258 Palo Alto Research Center, Inc., 3333 Coyote Hill Rd, Palo Alto, CA 94304 (From last week:) Subject: Another try at a Python Web SIG? From: Bill Janssen Date: Thu, 11 Sep 2003 13:53:48 PDT To: meta-sig@python.org I know this was just discussed a few months ago, but I'm very interested in getting a Web SIG together or reinvigorated or whatever it takes. The current Web support in Python is somewhat embarassingly patchy, given the neatness of the rest of the system. My specific thinking is about three things: 1) Add SSL server-side support to the "socket" module. This will include some way of managing certificates. Probably modelled strongly after the way we did it in ILU, if I do it :-). 2) Update the "httplib" module to include all of the client-side functionality available in the Linux "curl" program (a reasonable checklist, I think, but am open to better suggestions). 3) Replace/extend the BaseHTTPServer and SimpleHTTPServer modules with an extended webserver module which provides at least the functionality in Medusa. I'd be happy to work in #1 (starting with a PEP, I think) and provide help with #2 and #3, if others would like to pitch in. I don't want to collide with others who are working on this already, and think a SIG would be a good way to coordinate efforts. Bill From ping at zesty.ca Wed Sep 17 20:21:59 2003 From: ping at zesty.ca (Ping) Date: Wed Sep 17 20:22:09 2003 Subject: [Python-Dev] Improving the CGI module In-Reply-To: <3F68F4F7.6020601@bath.ac.uk> Message-ID: On Thu, 18 Sep 2003, Simon Willison wrote: > I've recently started using Python for server side web development, and > I've been running in to some limitations in the CGI module. [...] > The module also provides very little in the way of support for > common CGI related operations ... I think that improvements to the CGI module would be wonderful, and we could do a lot of useful things ranging from the most basic (e.g. friendly access to the CGI environment variables) to more sophisticated things (e.g. constructing URLs that submit forms). (I have recently been forced to use PHP instead of Python for some major projects and constantly feel the pain of using PHP -- in my opinion, it is full of inconsistencies and workarounds, and really has the feel of a language that was not carefully designed. It would be nice for Python to be more competitive with PHP in its support for simple Web scripting.) I would be interested in helping with the CGI module project. -- ?!ng From tim.one at comcast.net Wed Sep 17 20:22:13 2003 From: tim.one at comcast.net (Tim Peters) Date: Wed Sep 17 20:22:22 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: <3F68166D.5030206@lemburg.com> Message-ID: [Tim] >> Jeremy believed Py_UNICODE was already an unsigned type on his box, >> and that was the box with the segfaults. I don't know. Comparison >> with a signed int confused the issues to the point where we gave up >> and just fixed it . [M.-A. Lemburg] > That sounds more like compiler bug to me. It could be, but if so Jeremy is running on a mainstream Linux+gcc platform and then it's something we can't really wish away. Jeremy, can you tell us what Py_UNICODE resolves to on your box, or give enough details so someone else can figure it out? As I read the C standard, unsigned_int < 256 has to use unsigned comparison, so it's a compiler bug, or I'm misreading the standard, or Jeremy was mistaken in believing Py_UNICODE resolves to an unsigned thingie on his box (we know for sure that the bit pattern 0xcdcdcdcd compared less than 256 on his box; that's obviously what it should do if Py_UNICODE resolves to a signed 4-byte thing on his box, but not otherwise). > What's bothering me is that such compares are done in other places too, > so a more general solution would be better. I'd like to figure out what Jeremy's true problem was first -- we've got a solution to his symptom now, but don't really know why it was necessary. From mfb at lotusland.dyndns.org Wed Sep 17 20:22:27 2003 From: mfb at lotusland.dyndns.org (Matthew F. Barnes) Date: Wed Sep 17 20:36:43 2003 Subject: [Python-Dev] Consistent logging in the standard library In-Reply-To: <16232.57433.388130.573539@montanaro.dyndns.org> References: <005101c37d64$d37270c0$652b6992@alpha><1063835810.14316.20.camel@geddy > <16232.57433.388130.573539@montanaro.dyndns.org> Message-ID: <57798.130.76.32.20.1063844547.squirrel@server.lotusland.dyndns.org> Skip Montanaro said: > > Barry> So the idea would be to be able to write "import _.logging" and > Barry> definitely get the global logging package. Using underscore in > Barry> the logger would mirror this mnemonic for globalness. > > The connection seems tenuous, at best. If I log messages to syslog and > and > that file is later viewed or summarized by a third party, "python" seems a > lot more useful than "_" or even "stdlib". > I agree with Barry that a name should be given to the standard library "package" and that the namespace reserved in the logging hierarchy for standard library modules should match this name. On the other hand, "_" is making me have unpleasant Perl flashbacks. *shiver* So I also agree with Skip that something more verbose would be more useful. I'm okay with waiting to see what becomes of Barry's idea before deciding on a logging namespace. Barry: Not to get too far off-topic, but I'm curious what this statement would do: from _ import * And was this notation ever considered? import .logging (Not that that would help us much with the logging namespace issue. :-) Matthew Barnes From jeremy at zope.com Wed Sep 17 21:19:33 2003 From: jeremy at zope.com (Jeremy Hylton) Date: Wed Sep 17 21:21:10 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: References: Message-ID: <1063847973.10299.76.camel@localhost.localdomain> On Wed, 2003-09-17 at 20:22, Tim Peters wrote: > It could be, but if so Jeremy is running on a mainstream Linux+gcc platform > and then it's something we can't really wish away. Jeremy, can you tell us > what Py_UNICODE resolves to on your box, or give enough details so someone > else can figure it out? > > As I read the C standard, > > unsigned_int < 256 > > has to use unsigned comparison, so it's a compiler bug, or I'm misreading > the standard, or Jeremy was mistaken in believing Py_UNICODE resolves to an > unsigned thingie on his box (we know for sure that the bit pattern > 0xcdcdcdcd compared less than 256 on his box; that's obviously what it > should do if Py_UNICODE resolves to a signed 4-byte thing on his box, but > not otherwise). I was a little confused by the various UNICODE macros. (Is there a comment block somewhere that explains what they are for?) gcc -E tells me: typedef unsigned int Py_UCS4; typedef wchar_t Py_UNICODE; typedef long int wchar_t; (not necessarily in that order) I got Py_UCS4 and Py_UNICODE confused. The detailed output confirms that Py_UNICODE is a signed long int. Jeremy From sholden at holdenweb.com Wed Sep 17 21:22:29 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Sep 17 21:26:07 2003 Subject: [Python-Dev] Improving the CGI module In-Reply-To: Message-ID: [Ping] > On Thu, 18 Sep 2003, Simon Willison wrote: > > I've recently started using Python for server side web > development, and > > I've been running in to some limitations in the CGI module. [...] > > The module also provides very little in the way of support for > > common CGI related operations ... > > I think that improvements to the CGI module would be wonderful, > and we could do a lot of useful things ranging from the most basic > (e.g. friendly access to the CGI environment variables) to more > sophisticated things (e.g. constructing URLs that submit forms). > I'd agree that there's lots of room for new web functionality, but having looked in some detail at the CGI module, and with backward compatibility in mind, I'd question whether changes to that module are the best way to do this. > (I have recently been forced to use PHP instead of Python for some > major projects and constantly feel the pain of using PHP -- in my > opinion, it is full of inconsistencies and workarounds, and really > has the feel of a language that was not carefully designed. It > would be nice for Python to be more competitive with PHP in its > support for simple Web scripting.) > Unfortunately the CGI modules itself appears to have "just growed" in a rather organic way. It could be that my analytic skills simply aren't up to a task which others will breeze through, but I suspect that it might be better to start with a different framework, relieving the project of the need to make the same mistakes that CGI did. New mistakes are always more interesting ;-) > I would be interested in helping with the CGI module project. > My time is pretty limited, but I'd like to monitor such a project and snipe from the sidelines from time to time. regards -- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ From tim.one at comcast.net Wed Sep 17 22:55:35 2003 From: tim.one at comcast.net (Tim Peters) Date: Wed Sep 17 22:55:41 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: <1063847973.10299.76.camel@localhost.localdomain> Message-ID: [Jeremy Hylton] > I was a little confused by the various UNICODE macros. (Is there a > comment block somewhere that explains what they are for?) Not that I've found. If someone writes one, don't forget the intended difference between PY_UNICODE_TYPE and Py_UNICODE (hint: there isn't a difference ). > gcc -E tells me: > > typedef unsigned int Py_UCS4; > typedef wchar_t Py_UNICODE; > typedef long int wchar_t; > > (not necessarily in that order) > > I got Py_UCS4 and Py_UNICODE confused. The detailed output confirms > that Py_UNICODE is a signed long int. So that puts an end to the claim that it's unlikely wchar_t will resolve to a signed type. Strangely, while char is a signed type under MSVC, wchar_t is an unsigned type. I expect both differ under gcc, then. At least it's consistent . Anyway, everywhere the code may be doing a_Py_UNICODE comparison a_(signed)_int is doing something unintended now on your box. "The rules" for mixed-signedness comparison are pretty much a nightmare, especially when you're not sure how many bits are involved on both sides: http://yarchive.net/comp/ansic_broken_unsigned.html MAL's idea of forcing PY_UNICODE_TYPE to resolve to an unsigned type may be the easiest way out. From barry at python.org Thu Sep 18 00:52:02 2003 From: barry at python.org (Barry Warsaw) Date: Thu Sep 18 00:52:10 2003 Subject: [Python-Dev] Consistent logging in the standard library In-Reply-To: <57798.130.76.32.20.1063844547.squirrel@server.lotusland.dyndns.org> References: <005101c37d64$d37270c0$652b6992@alpha> <1063835810.14316.20.camel@geddy > <16232.57433.388130.573539@montanaro.dyndns.org> <57798.130.76.32.20.1063844547.squirrel@server.lotusland.dyndns.org> Message-ID: <1063860722.9847.2.camel@anthem> On Wed, 2003-09-17 at 20:22, Matthew F. Barnes wrote: > Not to get too far off-topic, but I'm curious what this statement would do: > > from _ import * I don't know. Perhaps Lib should have an __init__.py that would export an __all__? > And was this notation ever considered? > > import .logging It just was! :) > (Not that that would help us much with the logging namespace issue. :-) No, but maybe they shouldn't be tied. Unless they should . I just wanted to bring it up because it seemed somewhat related and I've been meaning to write a PEP on forced global imports for Python 2.4. -Barry From barry at python.org Thu Sep 18 00:58:59 2003 From: barry at python.org (Barry Warsaw) Date: Thu Sep 18 00:59:09 2003 Subject: [Python-Dev] Improving the CGI module In-Reply-To: References: Message-ID: <1063861139.9847.10.camel@anthem> On Wed, 2003-09-17 at 21:22, Steve Holden wrote: > > > Unfortunately the CGI modules itself appears to have "just growed" in a > rather organic way. It could be that my analytic skills simply aren't up > to a task which others will breeze through, but I suspect that it might > be better to start with a different framework, relieving the project of > the need to make the same mistakes that CGI did. New mistakes are always > more interesting ;-) +1! -Barry From martin at v.loewis.de Thu Sep 18 01:19:06 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Sep 18 01:19:21 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: References: Message-ID: "Tim Peters" writes: > So that puts an end to the claim that it's unlikely wchar_t will resolve to > a signed type. Strangely, while char is a signed type under MSVC, wchar_t > is an unsigned type. I think people have learned, over time, that negative characters are evil. So MS chose to make wchar_t unsigned, as, for 2-byte Unicode, you might otherwise run into negative characters. For gcc, people probably also agree that negative characters are evil, but using long int causes no harm here: All assigned characters are still positive, as Unicode goes only up to 2**21. Regards, Martin From anthony at interlink.com.au Thu Sep 18 01:23:52 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Sep 18 01:25:32 2003 Subject: [Python-Dev] Py2.3.1 In-Reply-To: <00ce01c37d43$aa879940$9c3dc797@oemcomputer> Message-ID: <200309180523.h8I5Nru7031988@localhost.localdomain> >>> "Raymond Hettinger" wrote > Are we on for doing a micro release this week? I was planning on doing it mid-next week. (I sent an email about this a couple of days ago - it doesn't seem to have gone through?) Anthony From bac at OCF.Berkeley.EDU Thu Sep 18 02:52:47 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Thu Sep 18 02:52:59 2003 Subject: [Python-Dev] Trying to fix time.tzset detection for 2.3.1 In-Reply-To: <20030917131354.GC1211@epoch.metaslash.com> References: <3F669AF3.8050605@ocf.berkeley.edu> <16231.7058.987290.575924@montanaro.dyndns.org> <3F674015.8090801@ocf.berkeley.edu> <3F67D078.1000106@ocf.berkeley.edu> <20030917131354.GC1211@epoch.metaslash.com> Message-ID: <3F69563F.6090209@ocf.berkeley.edu> Neal Norwitz wrote: > On Tue, Sep 16, 2003 at 08:09:44PM -0700, Brett C. wrote: > >>And a question on patching configure.pre.in: is that all I have to patch >>or do I need to run reconfigure and then also patch configure.in? > > > As Martin said, there is no configure.pre.in. The file is configure.in > and you need to regenerate configure and pyconfig.h.in with autoconf > and autoheader (version 2.57). > OK. Well, I ran autoreconf and autoheader. Both configure.in and configure look fine and ready to be committed. But pyconfig.h.in is another thing entirely. I just ran ``cvs diff`` on it and the diff I get is (on both HEAD and release23-maint): +/* Define this if you have tcl and TCL_UTF_MAX==6 */ +#undef HAVE_UCS4_TCL + Uh, why is that new? Is this from Martin's patch to detect RH's custom Tcl? Could this be from autoheader not being run after that patch (or someone else's)? -Brett From mal at lemburg.com Thu Sep 18 03:23:44 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Sep 18 03:23:42 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: References: <3F68166D.5030206@lemburg.com> Message-ID: <3F695D80.6020804@lemburg.com> Martin v. L?wis wrote: > "M.-A. Lemburg" writes: > >>No question there, but wouldn't it be easier to test such a platform >>and then fallback to "unigned int" in case wchar_t is found to >>be a signed value ? > > Why is it important that Py_UNICODE is unsigned? Because that's what was used as basis in the type implementation as well as the codecs (internal and external). Comparisons simply work differently when you're using a signed type which is also why most compilers warn about this -- but you know that. An signed type also doesn't make much sense for things like character storage -- the sign information is useless and you lose a bit for each character. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Sep 18 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From vinay_sajip at yahoo.co.uk Thu Sep 18 03:45:30 2003 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu Sep 18 03:43:42 2003 Subject: [Python-Dev] Consistent logging in the standard library References: <005101c37d64$d37270c0$652b6992@alpha> <1063835810.14316.20.camel@geddy> <16232.57433.388130.573539@montanaro.dyndns.org> Message-ID: <00ad01c37db8$d8bc9d40$652b6992@alpha> > Barry> So the idea would be to be able to write "import _.logging" and > Barry> definitely get the global logging package. Using underscore in > Barry> the logger would mirror this mnemonic for globalness. > Skip> The connection seems tenuous, at best. If I log messages to syslog and and Skip> that file is later viewed or summarized by a third party, "python" seems a Skip> lot more useful than "_" or even "stdlib". I agree. Readability in logs is the most important consideration here, and I think a scheme such as "python." + makes it very clear where an event originated. Of course this would just be a prefix; a module could log to any name under this namespace, e.g. "python.asyncore.socket" for socket-related events in asyncore. Vinay From mwh at python.net Thu Sep 18 06:59:03 2003 From: mwh at python.net (Michael Hudson) Date: Thu Sep 18 06:58:37 2003 Subject: [Python-Dev] Trying to fix time.tzset detection for 2.3.1 In-Reply-To: <3F69563F.6090209@ocf.berkeley.edu> (Brett C.'s message of "Wed, 17 Sep 2003 23:52:47 -0700") References: <3F669AF3.8050605@ocf.berkeley.edu> <16231.7058.987290.575924@montanaro.dyndns.org> <3F674015.8090801@ocf.berkeley.edu> <3F67D078.1000106@ocf.berkeley.edu> <20030917131354.GC1211@epoch.metaslash.com> <3F69563F.6090209@ocf.berkeley.edu> Message-ID: <2misnq8irs.fsf@starship.python.net> "Brett C." writes: > But pyconfig.h.in is another thing entirely. I just ran ``cvs diff`` > on it and the diff I get is (on both HEAD and release23-maint): > > +/* Define this if you have tcl and TCL_UTF_MAX==6 */ > +#undef HAVE_UCS4_TCL > + > > Uh, why is that new? Is this from Martin's patch to detect RH's > custom Tcl? Could this be from autoheader not being run after that > patch (or someone else's)? Looks like it (as in, I see that here too, and I don't have any local changes to configure.in). Cheers, mwh -- > I'm a little confused. That's because you're Australian! So all the blood flows to your head, away from the organ most normal guys think with. -- Mark Hammond & Tim Peters, comp.lang.python From mfb at lotusland.dyndns.org Thu Sep 18 08:04:57 2003 From: mfb at lotusland.dyndns.org (Matthew F. Barnes) Date: Thu Sep 18 08:19:06 2003 Subject: [Python-Dev] Consistent logging in the standard library In-Reply-To: <00ad01c37db8$d8bc9d40$652b6992@alpha> References: <005101c37d64$d37270c0$652b6992@alpha><1063835810.14316.20.camel@geddy ><16232.57433.388130.573539@montanaro.dyndns.org> <00ad01c37db8$d8bc9d40$652b6992@alpha> Message-ID: <3122.192.168.1.1.1063886697.squirrel@server.lotusland.dyndns.org> Vinay Sajip said: > I agree. Readability in logs is the most important consideration here, and > I think a scheme such as "python." + makes it very clear where > an event originated. Of course this would just be a prefix; a module could > log to any name under this namespace, e.g. "python.asyncore.socket" for > socket-related events in asyncore. > I'm convinced. I'll work the "python" prefix into the proposal. Matthew Barnes From martin at v.loewis.de Thu Sep 18 11:13:12 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Sep 18 11:13:24 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: <3F695D80.6020804@lemburg.com> References: <3F68166D.5030206@lemburg.com> <3F695D80.6020804@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > Because that's what was used as basis in the type implementation > as well as the codecs (internal and external). Comparisons simply > work differently when you're using a signed type which is also > why most compilers warn about this -- but you know that. Yes and no. It appears that you are assuming that Py_UNICODE is always unsigned, however, it is (AFAICT) nowhere documented, and I don't see any strong reason ("it used to be that way" is not a strong reason - code does change over time). > An signed type also doesn't make much sense for things like > character storage -- the sign information is useless and you > lose a bit for each character. OTOH, using wchar_t where possible is also valuable. On all systems with a signed wchar_t that we know of, that wchar_t has 32 bits - losing the sign bit does not mean to lose character code points, as Unicode has less than 2**17 code points, anyway. Regards, Martin From tim.one at comcast.net Thu Sep 18 11:18:43 2003 From: tim.one at comcast.net (Tim Peters) Date: Thu Sep 18 11:18:47 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: Message-ID: [martin@v.loewis.de] > I think people have learned, over time, that negative characters are > evil. So MS chose to make wchar_t unsigned, as, for 2-byte Unicode, > you might otherwise run into negative characters. For gcc, people > probably also agree that negative characters are evil, but using long > int causes no harm here: All assigned characters are still positive, > as Unicode goes only up to 2**21. That's fine for gcc then, so long as we don't branch on the contents of uninitialized memory, and we just fixed a bug of that sort. If Py_UNICODE ever resolves to a signed 2-byte type, though, the sign bit is still in play for legit contents. From nick at nick.uklinux.net Thu Sep 18 14:04:00 2003 From: nick at nick.uklinux.net (Nick Roberts) Date: Thu Sep 18 14:09:32 2003 Subject: [Python-Dev] Enhancement to pdb in gud.el Message-ID: <16233.62352.331254.344092@nick.uklinux.net> Hi, I've written a mode (gdb-ui.el) which is in the Emacs CVS repository (http://savannah.gnu.org/projects/emacs) that displays breakpoint icons in the display margin of Emacs when debugging with GDB. I can almost do the same thing for pdb (the Emacs interface to the python debugger in gud.el). Breakpoints in GUD can be set/cleared using the toolbar and it is easy to write a lisp function to do that. However, they can also be set/cleared from the GUD buffer with break/clear and gud-pdb-marker-filter is the best place to detect this. If I'm using pdb.py to debug fib.py and I type: (Pdb) b 4 I get the message: Breakpoint 1 at /home/nick/python/fib.py:4 and I can use gud-pdb-marker-filter to detect this string and insert a breakpoint icon at the appropriate line in the source buffer. Typing 'clear 1' gives Deleted breakpoint 1' and typing 'clear fib.py:4' gives no message and can't be detected. If, in both cases, the the message is modified to somthing like Deleted breakpoint 1 at /home/nick/python/fib.py:4 then I would have enough information. Anyone interested? Nick From martin at v.loewis.de Thu Sep 18 14:54:25 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Sep 18 14:54:41 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: References: Message-ID: "Tim Peters" writes: > That's fine for gcc then, so long as we don't branch on the contents of > uninitialized memory, and we just fixed a bug of that sort. > > If Py_UNICODE ever resolves to a signed 2-byte type, though, the sign bit is > still in play for legit contents. Right. I agree that we don't want to use a signed 2-byte type. Given that only MS uses a two-byte wchar_t (to my knowledge), and that theirs is unsigned, having an autoconf test to detect that this configuration is not supported is more than enough. Regards, Martin From recht at netbsd.org Thu Sep 18 18:44:24 2003 From: recht at netbsd.org (Marc Recht) Date: Thu Sep 18 18:44:31 2003 Subject: [Python-Dev] Python and POSIX In-Reply-To: References: <35230000.1063660079@leeloo.intern.geht.de> <46570000.1063709941@leeloo.intern.geht.de> Message-ID: <48950000.1063925064@leeloo.intern.geht.de> > While that is true, I would hope that you come to the conclusion that > these macros can potentially do much more than simply hiding library > functions. Instead, their *primary* purpose is to request Single Unix > behaviour in case a platform implements both the standard behaviour, > and the traditional one (regardless of which tradition the system > follows). I'm fully aware of it. I say it's sometimes a PITA, too. At least if it comes to 3rd party modules. > So I would be *very* careful to enable these macros only for a few > translation units; I'm pretty sure that the "law" would be on the > vendor's side if inconsistent usage of these defines would be found to > cause problems, in Irix 27, or ValueBSD 3. That's why I'm totaly in favor of "stricly POSIX conforming" for Python itself. It's IMHO an other situation when it comes to modules or extensions. (Also, the ones in the Python distribution.) IMHO the module implementors should define for themself which standard they need/support (if any). As you've pointed out in your last mail there are (major?) technical problems with stat (eg. struct stat) which need to be solved. > If the change is only for 2.4, that would be fine - that code > hopefully will get testing on several other platforms before 2.4 is > released. Totally agree. That would be a way to big change for 2.3.x and it needs much testing, too. > Believes should not be trusted too much when it comes to > portability. If you can think of an obscure interpretation of some > standard, it is almost certain that there is some vendor out there who > has chosen that interpretation. And sometimes the standard itself is the problem... Regards, Marc From bac at OCF.Berkeley.EDU Thu Sep 18 20:38:57 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Thu Sep 18 20:39:12 2003 Subject: [Python-Dev] Trying to fix time.tzset detection for 2.3.1 In-Reply-To: <2misnq8irs.fsf@starship.python.net> References: <3F669AF3.8050605@ocf.berkeley.edu> <16231.7058.987290.575924@montanaro.dyndns.org> <3F674015.8090801@ocf.berkeley.edu> <3F67D078.1000106@ocf.berkeley.edu> <20030917131354.GC1211@epoch.metaslash.com> <3F69563F.6090209@ocf.berkeley.edu> <2misnq8irs.fsf@starship.python.net> Message-ID: <3F6A5021.1080100@ocf.berkeley.edu> Michael Hudson wrote: > "Brett C." writes: > > >>But pyconfig.h.in is another thing entirely. I just ran ``cvs diff`` >>on it and the diff I get is (on both HEAD and release23-maint): >> >>+/* Define this if you have tcl and TCL_UTF_MAX==6 */ >>+#undef HAVE_UCS4_TCL >>+ >> >>Uh, why is that new? Is this from Martin's patch to detect RH's >>custom Tcl? Could this be from autoheader not being run after that >>patch (or someone else's)? > > > Looks like it (as in, I see that here too, and I don't have any local > changes to configure.in). > OK, cool. Thanks for the info, Michael. I am going to go ahead and patch HEAD and release23-maint for configure.in, configure, and pyconfig.h.in . -Brett From tim.one at comcast.net Thu Sep 18 23:06:01 2003 From: tim.one at comcast.net (Tim Peters) Date: Thu Sep 18 23:06:22 2003 Subject: [Python-Dev] Fun with 2.3 shutdown Message-ID: When the Zope3 tests are run under Python 2.3, after the test runner ends we usually get treated to a long string of these things: """ Unhandled exception in thread started by Error in sys.excepthook: Original exception was: """ All that output is from Python, but it isn't particularly helpful . Here's what I know: For Zope readers: these are indeed due to the _Checkpoint threads that are left behind, sitting in self._event.wait(self._interval) self._interval is 120 seconds (i.e., a relatively long time). For Python readers: Event.wait() with a timeout ends up in _Condition.wait(), where a lazy busy loop wakes up about 20 times per second to see whether it can proceed. For some reason an exception is getting raised in the wait() code. I'm not exactly sure what or why yet, but that will come soon enough. For this message, it's enough to know that it happens (an educated guess comes later in this msg). Python is in Py_Finalize, in its (new in 2.3) second call of PyGC_Collect(): /* Collect garbage. This may call finalizers; it's nice to call these before all modules are destroyed. */ PyGC_Collect(); /* Destroy all modules */ PyImport_Cleanup(); /* Collect final garbage. This disposes of cycles created by new-style class definitions, for example. */ PyGC_Collect(); WE'RE IN THIS ONE *Presumably* the _Condition.wait() code wakes up after modules have been destroyed, and raises an exception due to referencing a None'ed-out threading.py global. A fact is that PyGC_Collect has called a __del__ method when the problem occurs, and that's why some other thread was able to get the GIL and proceed (gc.collect() gave control back to Python code, via a Python __del__ method). *Presumably* this only causes a visible problem when running "at least most of" the Zope3 tests because the Zope3 tests create a ton of containers and gc takes significant time then. Presumably if just a few tests are run, the second call to PyGC_Collect() goes so fast that it's unlikely the _Condition.wait() code will wake up and blow up before Python exits. Back to for-sure stuff: the thread does raise an exception, and we wind up back in threadmodule.c t_bootstrap: if (res == NULL) { IT IS NULL if (PyErr_ExceptionMatches(PyExc_SystemExit)) PyErr_Clear(); else { PyObject *file; PySys_WriteStderr( "Unhandled exception in thread started by "); file = PySys_GetObject("stderr"); if (file) PyFile_WriteObject(boot->func, file, 0); else PyObject_Print(boot->func, stderr, 0); PySys_WriteStderr("\n"); PyErr_PrintEx(0); } } It's not PyExc_SystemExit, so we write Unhandled exception in thread started by to C's stderr and then get sys.stderr. Heh. sys has been wiped by this point, and file gets is Py_None. Brrrrrrrr. Everything after this is a comedy of unintended errors. file is true (so far as C is concerned), so we do PyFile_WriteObject(boot->func, file, 0); That tries some tests that don't pass, then does writer = PyObject_GetAttrString(f, "write"); None.write doesn't exist, so the exception we started with is lost and replaced by an unwanted AttributeError. PyFile_WriteObject returns -1 then, but t_bootstrap doesn't check for that. Instead it writes a newline, and goes on to PyErr_PrintEx(0); That eventually does hook = PySys_GetObject("excepthook"); and for a now-familiar reason, that also returns None. Again hook looks true to C, and PyObject *result = PyEval_CallObject(hook, args); overwrites the unwanted AttributeError with a different unwanted exception. PyErr_PrintEx notices that the call fails, and prints "Error in sys.excepthook:\n" to C's stderr. It then tries PyErr_Display(exception2, v2, tb2); That starts with PyObject *f = PySys_GetObject("stderr"); and *again* that returns None from the ruined sys module, and again f looks true to C. It becomes something of a mess then, so suffice it to say that PyErr_Display's attempts to write to file None via PyFile_WriteString(buf, f) don't accomplish anything, and then PyErr_Display() ends with /* If an error happened here, don't show it. XXX This is wrong, but too many callers rely on this behavior. */ if (err != 0) PyErr_Clear(); So we come back with no exception set, and PyErr_PrintEx proceeds to write "\nOriginal exception was:\n" to C's stderr. Then it tries PyErr_Display(exception, v, tb); and that goes thru the long dance of retrieving None from sys.stderr (etc) all over again. The best thing for the Zope3 tests would be to find ways not to leave useless daemon threads hanging around. I'm not sure what it says about Python's shutdown strategy. If I find and fix a None-module-global problem in _Condition.wait(), that's fine and probably helpful (in 2.3.1), but in the context of Zope3 the run() methods of these threads are going to trigger None-module-global problems of their own. That can also be fixed. So a combination of changes to threading.py and Zope3 code can probably hide the shutdown glitches. Somehow, though, it just doesn't feel entirely satisfactory <0.9 wink>. seemingly-out-of-gordian-knot-cutters-for-this-week-ly y'rs - tim From martin at v.loewis.de Fri Sep 19 00:59:00 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Fri Sep 19 00:59:18 2003 Subject: [Python-Dev] Python and POSIX In-Reply-To: <48950000.1063925064@leeloo.intern.geht.de> References: <35230000.1063660079@leeloo.intern.geht.de> <46570000.1063709941@leeloo.intern.geht.de> <48950000.1063925064@leeloo.intern.geht.de> Message-ID: Marc Recht writes: > That's why I'm totaly in favor of "stricly POSIX conforming" for > Python itself. It's IMHO an other situation when it comes to modules > or extensions. (Also, the ones in the Python distribution.) IMHO the > module implementors should define for themself which standard they > need/support (if any). As you've pointed out in your last mail there > are (major?) technical problems with stat (eg. struct stat) which need > to be solved. Unfortunately, it might just be that this specific problem cannot be solved, in a general way. Instead, the author of the code would have to take the risk of breaking the in-process ABI. For example, if the #defines would cause different libraries to be linked (rather than different symbols of the same library being used), you might find that mapping both libraries into the same process is just not supported, and that the extension module would still get the POSIX version of the function at run-time, and that the system vendor provides no support for "mixed" processes. So if you implement such a change, you should also make it easy for a developer to avoid the danger of such undeliberate breakage, by having a mode where extensions compile with the exact same settings as the main interpreter. Perhaps that should even be the default, and the do-it-yourself configuration would be activated only on explicit request (e.g. a define before including Python.h). Regards, Martin From anthony at interlink.com.au Fri Sep 19 01:37:49 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Sep 19 01:39:26 2003 Subject: [Python-Dev] Fun with 2.3 shutdown In-Reply-To: Message-ID: <200309190537.h8J5bnIG015051@localhost.localdomain> >>> "Tim Peters" wrote > When the Zope3 tests are run under Python 2.3, after the test runner ends we > usually get treated to a long string of these things: > > """ > Unhandled exception in thread started by > Error in sys.excepthook: > Original exception was: > > """ Yep. I remember mentioning these at the Melbourne sprint. > I'm not sure what it says about Python's shutdown strategy. If I find and > fix a None-module-global problem in _Condition.wait(), that's fine and > probably helpful (in 2.3.1), but in the context of Zope3 the run() methods > of these threads are going to trigger None-module-global problems of their > own. That can also be fixed. Hm. Is this serious enough a problem to hold off 2.3.1 until it's fixed? My feeling is "no", because it only affects Z3 developers, and they're hardened to random pain by now, and it's also just a shutdown glitch. On the other hand, making the Z3 shutdown code os.kill(os.getpid(), 9) would also make the messages go away ;) Anthony -- Anthony Baxter It's never too late to have a happy childhood. From bac at OCF.Berkeley.EDU Fri Sep 19 02:35:01 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Sep 19 02:35:02 2003 Subject: [Python-Dev] python-dev Summary for 2003-09-01 through 2003-09-15 [draft] Message-ID: <3F6AA395.9040500@ocf.berkeley.edu> Not planning on rushing this summary out the door; most likely will send it out on Sunday. So have a read and show me why I should not pass the writing test I have to take next month that is supposed to prove I am a competent writer. --------------- python-dev Summary for 2003-09-01 through 2003-09-15 ++++++++++++++++++++++++++++++++++++++++++++++++++++ This is a summary of traffic on the `python-dev mailing list`_ from September 1, 2003 through September 15, 2003. It is intended to inform the wider Python community of on-going developments on the list. To comment on anything mentioned here, just post to `comp.lang.python`_ (or email python-list@python.org which is a gateway to the newsgroup) with a subject line mentioning what you are discussing. All python-dev members are interested in seeing ideas discussed by the community, so don't hesitate to take a stance on something. And if all of this really interests you then get involved and join `python-dev`_! This is the twenty-fifth summary written by Brett Cannon (with school looming on the horizon). All summaries are archived at http://www.python.org/dev/summary/ . Please note that this summary is written using reStructuredText_ which can be found at http://docutils.sf.net/rst.html . Any unfamiliar punctuation is probably markup for reST_ (otherwise it is probably regular expression syntax or a typo =); you can safely ignore it, although I suggest learning reST; its simple and is accepted for `PEP markup`_ and gives some perks for the HTML output. Also, because of the wonders of programs that like to reformat text, I cannot guarantee you will be able to run the text version of this summary through Docutils_ as-is unless it is from the original text file. .. _PEP Markup: http://www.python.org/peps/pep-0012.html The in-development version of the documentation for Python can be found at http://www.python.org/dev/doc/devel/ and should be used when looking up any documentation on something mentioned here. PEPs (Python Enhancement Proposals) are located at http://www.python.org/peps/ . To view files in the Python CVS online, go to http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/ . Reported bugs and suggested patches can be found at the SourceForge_ project page. .. _python-dev: http://www.python.org/dev/ .. _SourceForge: http://sourceforge.net/tracker/?group_id=5470 .. _python-dev mailing list: http://mail.python.org/mailman/listinfo/python-dev .. _comp.lang.python: http://groups.google.com/groups?q=comp.lang.python .. _Docutils: http://docutils.sf.net/ .. _reST: .. _reStructuredText: http://docutils.sf.net/rst.html .. contents:: .. _last summary: http://www.python.org/dev/summary/2003-08-16_2003-08-31.html ===================== Summary Announcements ===================== Summary is nice and short this time. More people went on vacation (although Martin came back and so that helped to pick up the traffic a little). I skipped covering some threads that are due for PEPs since those will be more in-depth then anything I write. I should give fair warning that starting on the 22nd of September I will begin school. I am hoping that it will not be difficult and I will have a carefree time in school. But if my workload becomes a burden the Summaries might suffer. I hope they don't, though, and I am going to do my best to make sure that doesn't happen. ========= Summaries ========= --------------------------------------- If It Isn't Documented, Is It a Secret? --------------------------------------- Some undocumented methods in readline were discovered by Philip Eby. He asked if this was on purpose and whether he should submit a bug report on the lack of documentation. Guido told him to go ahead and submit the bug report. If you ever find something undocumented in the Python source code and there is nothing suggesting the code is meant to be hidden from the user (it's for internal use, a comment says it is experimental, etc.), then please file a bug report! Contributing threads: - `Undocumented functions in 'readline' module `__ ----------------------------------------------------------------------------------------- "You are more trouble than you are worth!", screamed the micro release to the new feature ----------------------------------------------------------------------------------------- The topic of what should be backported for micro releases (with Python 2.3.1 looming on the horizon) came up. Originally minor improvements were allowed in if they didn't break backwards compatibility. And of course bugfixes have been as well as long as they didn't break code that came to depend on the buggy behavior (really depends on how long the bugginess has been there and how well-known it is). But then the point of OS X 10.3 possibly becoming the largest install base of Python of any version (it will be 2.3) came up. With rough estimates being thrown around of 5 million installs in about a year's time, the point that making it difficult to run Python 2.3.x code on that size of an install base would be bad. For instance, if some small new feature was added to 2.3.1 then any code using that feature would not be able to run on a virgin install of OS X 10.3 (and considering that this is Mac it should not be expected that most users will want to, let alone know how, to add a secondary install of Python since the original is used by the OS and thus should not be overwritten). It looks like a more conservative patching scheme will be taken with micro releases. Bytecode will also continue to work between releases. Guido has always unofficially held that position but now it has been vocalized. Contributing threads: - `Documenting branch policy `__ ---------------------------------------------- PyPy "Berlin" sprint (29th Sep - 4th Oct 2003) ---------------------------------------------- Just what the title says: there is going to be a sprint_ for PyPy_ in Berlin from September 29 to October 4. Read the email for more specific details on goals and such. .. _sprint: http://www.python.org/cgi-bin/moinmoin/SprintPlan .. _PyPy: http://codespeak.net/pypy/index.cgi?home Contributing threads: - `PyPy "Berlin" sprint (29th Sep - 4th Oct 2003) `__ -------------------------------- Away with you ambiguous imports! -------------------------------- A discussion on how to create a hierarchy of loggers in the standard library led to the idea of having a way to cause an import to come directly from the standard library and thus remove any ambiguity from a relative import grabbing a similarly named module from the local package. Barry Warsaw he was thinking of writing a PEP on this idea. Contributing threads: - `Consistent logging in the standard library `__ --------------------------------------------------- Quick: want to give a Python talk at Linuxworld NY? --------------------------------------------------- If you are interested in what the title asks, then read the email for some details. Contributing threads: - `Quick: want to give a Python talk at Linuxworld NY? `__ ------------------------------------------- Be careful about saying something is "easy" ------------------------------------------- A word of advice about saying something is "easy" on python-dev: if you say that you should make sure you can back up that claim because otherwise you will be told to write the "easy" patch and that will be the end of the discussion. Contributing threads: - `Making python C-API thread safe (try 2) `__ -------------------------- Parsing dates for datetime -------------------------- There was some talk about coming up with some code to parse dates for the datetime module. It was kind of hand-wavy since there are multiples chunks of code that can do that in the standard library. If you have an opinion on this (or anything for that matter), make sure to make a post to `comp.lang.python`_ about it. Contributing threads: - `datetime issues `__ From jafo-python-dev at tummy.com Fri Sep 19 04:33:46 2003 From: jafo-python-dev at tummy.com (Sean Reifschneider) Date: Fri Sep 19 04:33:55 2003 Subject: [Python-Dev] Discussion on adding rsplit() for strings and unicode objects. Message-ID: <20030919083346.GA11541@tummy.com> A few weeks ago I submitted a patch to add an rsplit() method to string and unicode objects: >Adding rsplit() to string and unicode objects. (2003-09-06) > http://python.org/sf/801847 loewis responded with: >[...] I feel that there are already too many string methods, so I >won't accept that patch. I'm not rejecting it, either, because I see >that other maintainers might have a different opinion. In short, you >should propose your change to python-dev, finding out what "a majority" >of the maintainers thinks [...] My understanding of why the patch was not accepted is that users can implement it themselves. My reason for spending the time implementing the patch was that I noticed it was missing in some coding I was doing. Instead of having this tool available, I had to interrupt the flow of my coding and build and test an rsplit implementation. Not a huge undertaking, of course, but still something that had to be dealt with. I guess at least some maintainers wish for string objects to have fewer methods. I haven't heard of any alternative location to place it other than placing a Python implementation in the Cookbook. That alternative seems strange to me -- requiring a user locate and maintain an alternative implementation in their own code, likely at a fairly big performance penalty (my rsplit is an adaptation of the existing split() C code). Obviously, I felt this was enough of a lack in the library that I spent most of a Sunday afternoon tracking down and implementing it. One of Python's strengths is it's rich standard library (he says after spending a few hours unsuccessfully trying to package a dozen Perl modules), and I'd like to see it get even better. I'd like to open this discussion, as loewis suggests, to the python-dev folks. Thoughts? Thanks, Sean -- Money is the root of all evil! Man needs roots... Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995. Qmail, Python, SysAdmin Back off man. I'm a scientist. http://HackingSociety.org/ From fincher.8 at osu.edu Fri Sep 19 06:12:15 2003 From: fincher.8 at osu.edu (Jeremy Fincher) Date: Fri Sep 19 05:13:20 2003 Subject: [Python-Dev] Discussion on adding rsplit() for strings and unicode objects. In-Reply-To: <20030919083346.GA11541@tummy.com> References: <20030919083346.GA11541@tummy.com> Message-ID: <200309190612.15965.fincher.8@osu.edu> On Friday 19 September 2003 04:33 am, Sean Reifschneider wrote: > A few weeks ago I submitted a patch to add an rsplit() method to > > string and unicode objects: > >Adding rsplit() to string and unicode objects. (2003-09-06) > > http://python.org/sf/801847 I'm all for it. I've had to implement rsplit on my own as well, and always lamented (but never did anything, unfortunately) about its lack in the standard library. That strings have rfind, rindex, rstrip, and rjust, but not rsplit always seemed somewhat less than complete to me. > loewis responded with: > >[...] I feel that there are already too many string methods, so I > >won't accept that patch. Python's string methods (and the fact that I can easily iterate over the characters in a string) enable me to write heavily string-manipulating code without resorting to the cryptic regular expressions my old Perl code was laden with. As long as they're named appropriately, I can't see the harm in adding useful methods, especially one such as rsplit that completes a set of front-to-back/back-to-front methods. Jeremy From mal at lemburg.com Fri Sep 19 07:10:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Sep 19 07:10:20 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: References: Message-ID: <3F6AE417.2060301@lemburg.com> Martin v. L?wis wrote: > "Tim Peters" writes: > >>That's fine for gcc then, so long as we don't branch on the contents of >>uninitialized memory, and we just fixed a bug of that sort. >> >>If Py_UNICODE ever resolves to a signed 2-byte type, though, the sign bit is >>still in play for legit contents. > > Right. I agree that we don't want to use a signed 2-byte type. Given > that only MS uses a two-byte wchar_t (to my knowledge), and that > theirs is unsigned, having an autoconf test to detect that this > configuration is not supported is more than enough. Since wchar_t is the only case where a signed type can pop up, why not extend the autoconf test to check for signedness and then reject signed wchar_t value as not-usable (ie. undefine HAVE_USABLE_WCHAR_T). It looks to me as if this would resolve the problem once and for all. Signed values simply cause too many problems for this kind of application. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Sep 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From arigo at tunes.org Fri Sep 19 08:01:00 2003 From: arigo at tunes.org (Armin Rigo) Date: Fri Sep 19 08:06:55 2003 Subject: [Python-Dev] Fun with 2.3 shutdown In-Reply-To: References: Message-ID: <20030919120100.GB23317@vicky.ecs.soton.ac.uk> Hello Tim, The behavior of Python regarding whether threads should continue to run after the main one exited has never been too clear. Wouldn't it make sense to try to control this aspect more precisely ? I have also experienced this kind of shutdown glitches. It definitely needs more thinking, but what about this one: Python would only shutdown after all threads are finished; but exiting the main thread would by default send a SystemExit exception to all current threads, if that can be done cleanly (oh well, this problem again). Another question, what was the rationale for setting module globals to None instead of simply deleting them (and getting the expected AttributeErrors both at C and at Python level) ? Armin From python-kbutler at sabaydi.com Fri Sep 19 10:24:58 2003 From: python-kbutler at sabaydi.com (Kevin J. Butler) Date: Fri Sep 19 10:25:22 2003 Subject: [Python-Dev] Enhancement to pdb in gud.el In-Reply-To: References: Message-ID: <3F6B11BA.6000008@sabaydi.com> From: Nick Roberts > Typing 'clear 1' gives Deleted breakpoint 1' and typing 'clear fib.py:4' gives > no message and can't be detected. If, in both cases, the the message is > modified to somthing like > > Deleted breakpoint 1 at /home/nick/python/fib.py:4 > > then I would have enough information. The attached modifications do it for me. A couple of issues: - I completely replaced the Bdb.get_break method. It was returning a boolean indicating if the breakpoint existed or not, which appeared to be an error (get_breaks returned the list of breakpoints). The method was never used in bdb.py or pdb.py, so I replaced it with what I thought it should be. :-) That is, it now returns the breakpoint instance requested. If there is a reason, I can easily make this a new method (get_breakpoint/get_breakpointinstance) rather than replacing get_break. - I've duplicated logic from Bdb.clear_bpbynumber into a new method get_bpbynumber. The logic differs only in the return error messages or None. This seemed the simplest way to preserve the return value semantics convention of the Bdb class. - I'm also calling 'get', then 'clear' which does the work of 'get' twice. I did this to preserve the return value conventions of Bdb and make the breakpoint info available to the UI. Shouldn't be a performance issue, right? ;-) Suggestions before I submit these as a patch to SF? Life-doesn't-get-any-better-than-contributing-to-emacs-in-Python-ly y'rs, kb Sorry for the non-CVS diffs - I don't have a copy of the 2.3 repository right now: *** bdb.py.1.42 Fri Sep 19 08:08:17 2003 --- bdb.py Fri Sep 19 08:00:52 2003 *************** *** 266,274 **** self.breaks = {} def get_break(self, filename, lineno): ! filename = self.canonic(filename) ! return filename in self.breaks and \ ! lineno in self.breaks[filename] def get_breaks(self, filename, lineno): filename = self.canonic(filename) --- 266,275 ---- self.breaks = {} def get_break(self, filename, lineno): ! bplist = self.get_breaks( filename, lineno ) ! if len( bplist ): ! return bplist[-1] ! return None def get_breaks(self, filename, lineno): filename = self.canonic(filename) *************** *** 276,281 **** --- 277,294 ---- lineno in self.breaks[filename] and \ Breakpoint.bplist[filename, lineno] or [] + def get_bpbynumber(self, arg): + # duplicates logic from clear_bpbynumber + try: + number = int(arg) + except: + return None + try: + bp = Breakpoint.bpbynumber[number] + except IndexError: + return None + return bp + def get_file_breaks(self, filename): filename = self.canonic(filename) if filename in self.breaks: *** pdb.py.1.66 Fri Sep 19 08:06:01 2003 --- pdb.py Fri Sep 19 08:07:23 2003 *************** *** 467,482 **** except: err = "Invalid line number (%s)" % arg else: err = self.clear_break(filename, lineno) if err: print '***', err return numberlist = arg.split() for i in numberlist: err = self.clear_bpbynumber(i) if err: print '***', err else: ! print 'Deleted breakpoint %s ' % (i,) do_cl = do_clear # 'c' is already an abbreviation for 'continue' def do_where(self, arg): --- 467,486 ---- except: err = "Invalid line number (%s)" % arg else: + bp = self.get_break(filename, lineno) err = self.clear_break(filename, lineno) if err: print '***', err + else: + print 'Deleted breakpoint %s at %s:%s' % (i, bp.file, bp.line) return numberlist = arg.split() for i in numberlist: + bp = self.get_bpbynumber(i) err = self.clear_bpbynumber(i) if err: print '***', err else: ! print 'Deleted breakpoint %s at %s:%s' % (i, bp.file, bp.line) do_cl = do_clear # 'c' is already an abbreviation for 'continue' def do_where(self, arg): From hpk at trillke.net Fri Sep 19 11:42:40 2003 From: hpk at trillke.net (Holger Krekel) Date: Fri Sep 19 11:42:45 2003 Subject: [Python-Dev] pep 310 (reliable acquisition/release pairs) Message-ID: <20030919174240.C31102@prim.han.de> hello, admittedly i only followed Brett's very nice summaries (thanks!) in the last month so i may have missed some details. But here are a few comments regarding "PEP 310 Reliable Acquisition/Release Pairs". The PEP actually is about interacting with the execution of a code block. It allows to define (one-shot) interception points for entering and leaving a code block. Now there are at least two interesting cases which the PEP does (quite explicitely) not cover: - what to do with exceptions - what to do with yield IMHO introducing a new block statement at this stage in language development warrants an effort to tackle these cases (and maybe more like e.g. allowing the handler to trigger looping). This is probably best done with trying to directly design a protocol between the "interpreter-loop" and the - what i'd call - the "execution handler". And here it seems much more important to provide a concise definition of such a protocol (enter/leave/except/whatever) than to try to find an exact equivalence to plain python code. "Execution events" are really occuring at several points/bytecodes in the eval-loop and a plain python construct can only go so far. for example, you *could* say something like __enter__ called every time execution of the code block resumes (including when it is entered the first time) but this is not easily expressed in plain Python code although it seems implementable (and useful for resource-control when yielding). So while i am not strictly against the proposal i'd humbly ask for not hurrying into accepting the PEP as is. Python 2.4 is not closeby so i hope there is still some time to discuss this. I have done some practical experiments ("x-python") which actually incorporate XML-syntax into Python using a concept of an execution handler (tags are execution handlers and can additionally "catch" unassigned/dropped values that would otherwise be POP_TOPed). While this is impossible to ever appear in official Python i'd like to think about and report some experiences - but not before Novemeber because the next PyPy sprint and other stuff is keeping me far too busy. thanks & regards, holger From gherron at islandtraining.com Fri Sep 19 11:50:47 2003 From: gherron at islandtraining.com (Gary Herron) Date: Fri Sep 19 11:54:06 2003 Subject: [Python-Dev] Re: Fsrit Propttoye: SCLABMRE In-Reply-To: <16232.24354.366003.171152@gargle.gargle.HOWL> References: <16232.24354.366003.171152@gargle.gargle.HOWL> Message-ID: <200309190850.47841.gherron@islandtraining.com> Wow! This otavbeiorsn is both aimznag and eaittnnnerig. I was trohugh the Sjceubt: line and into the fisrt senencte brefoe I even rleeiazd annhtiyg was odd. Tahkns, Gary Herron On Wednesday 17 September 2003 06:18 am, Frnak "Sealmbcrd Eggs" Lomax wrote: > I hvae jsut fhiseind rniedag a Shdlaost atrlcie from Mnoday, Sbepemetr > 15th, aobut an ingtriniug sdtuy from Cgbdirmae Uinevtrsiy, dciinesrbg > how wodrs with thier itienrors selmrbacd can siltl be rdbaleae. > Bcaeuse of the tioeznme that I lvie in, an initettrment dfceet in my > clbae medom, and the lneiignrg atfer-ecfefts of my lsat "mohsroum > salad exntarvzagaa", I am aaylws two days bhneid on the Web. > > So suentnd was I at the icoinatpimls of this study, taht I > cunlqtneesoy decedid to doetve my etrnie lfie and gtelray aeamssd > fnrtoue on cunitodcng ftrhuer rasreech and dnpeeelmvot into tihs > lifnsaevig tghlocnoey. As a cross-eeyd dxsiyelc, I konw that teshe > cilmas of svotailan are niehetr hlroyepbe nor bmobaistc > over-daaomtiazitrn. > > Pnseetred hree for your rveiew, aaovpprl and disiscsuon, is our > poyptotre sytsem, caelld SLABCRME for SeCren Rslmnebeig Aacrihc > Mnoaaluptiin of BetaoLd Ecuitolon. I have scnie bugen a meosdt > sutartp based in Aceicdnt Maarnlyd, and we will soon be reeianlsg an > Olutook puglin, anolg with a sftorwae devmoeelpnt kit, and oethr > digintoasc tloos. Smoe of our tlgeocnhoy wlil nallaruty be open > sruoce, but I do ietnnd and ecpext to frutehr etnxed my hoougmuns > wtaleh by sinlelg sftorawe lcseiens and srhink wrapped ratiel pocrudt. > Psaele cactnot us for more iomrfaiontn. > > I'm aslo in enpommelyt duicnssioss wtih the Tmboit; I ecepxt that he > can bosot the pmecarrofne of our pooyptrte's itmpeanetolmin > salreolefvd, at wcihh piont we will be pthincig emdbeded ssmeyts to > the mfntacuuearrs of ruoters, eiltncreoc trprwyeiets, psnreoal dtaiigl > astsntsais, eorctinlec cepmutor coltlerond tersotas, maet prbeos, RFC > 1149 digtail cemara pheons, and nxet gniaoetern wopehoe csionuhs. > > hinpog-to-be-deotersyd-by-moroscift-ly y'rs, > -fnark From python-kbutler at sabaydi.com Fri Sep 19 12:43:38 2003 From: python-kbutler at sabaydi.com (Kevin J. Butler) Date: Fri Sep 19 12:43:48 2003 Subject: [Python-Dev] Fsrit Propttoye: SCLABMRE In-Reply-To: References: Message-ID: <3F6B323A.4070108@sabaydi.com> I agilzoope bceasue I msiesd rnaiedg your msagese unitl I saw a rlepy. I'm delgehitd taht you are cientarg a cnompay to exiplot tihs marouvles pnmenoehon. If you are looikng for aiidndotal team mebmers for yuor work snrlibcmag txet, I simbut this, the paorrgm I wtore when I haerd of this ireesttnnig and ueufsl (or mbaye jsut ireeitsnntg) eefcft. I hope it metes wtih yuor approval, tuhgoh wehn tarteed wtih ilestf (and a maoterde aomunt of uesr ivtintorenen, it lsot sangtcniilfiy more rlaaitebdiy than the visoern wtriten in the other 'P' luangage. Stetrad wtih more, lsot more, raetenid mroe...). In-serbmclad-txet-ly y'rs kb """ Srbmlcae by Kiven Beultr 2003 pthyon-kbuletr@sdabayi.com Rlaseeed into pibulc dmioan wtih no wrtanray Use at yuor own rsik """ import sys import random import re def srbmlcae( wrod ): ieiedsns = list( wrod[1:-1] ) random.shuffle( ieiedsns ) return wrod[0] + ''.join( ieiedsns ) + wrod[-1] def scRbaelmrx( match ): return srbmlcae( match.group() ) for lnie in sys.stdin: print re.sub( "\w{4,}", scRbaelmrx, lnie )[:-1] From tjreedy at udel.edu Fri Sep 19 13:44:09 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Fri Sep 19 13:44:17 2003 Subject: [Python-Dev] Re: python-dev Summary for 2003-09-01 through 2003-09-15[draft] References: <3F6AA395.9040500@ocf.berkeley.edu> Message-ID: "Brett C." wrote in message > But then the point of OS X 10.3 possibly becoming the largest install > base of Python of any version (it will be 2.3) came up. I an somehow amused at the prospect of Apple OX (Uni)X releases influencing the timing of Python releases. > With rough estimates being thrown around of 5 million installs > in about a year's > time, the point that making it difficult to run Python 2.3.x code on > that size of an install base would be bad. I've begun work on a 2.3 based project. If I were to get users who have received rather than installed Python, I would be more hesitant to simply say 'upgrade', especially for minor features in a 2.3.x. > that this is Mac it should not be expected that most users will want to, > let alone know how, to add a secondary install of Python since the > original is used by the OS and thus should not be overwritten). I presume a Mac installer would know how to add rather than overwrite. I am curious whether anyone has ever suggested to PC makers that Python + PyWin would be a useful addition to their Windows package -- both for setup/admin scripts (as done by RedHat. ..., and now Apple -- I don't know the specifics) and as a freebie addon (Basic for the 21st Century)? Terry J. Reedy From bac at OCF.Berkeley.EDU Fri Sep 19 13:59:07 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Sep 19 13:59:14 2003 Subject: [Python-Dev] Re: python-dev Summary for 2003-09-01 through 2003-09-15[draft] In-Reply-To: References: <3F6AA395.9040500@ocf.berkeley.edu> Message-ID: <3F6B43EB.1080308@ocf.berkeley.edu> Terry Reedy wrote: > "Brett C." wrote in message > > >>But then the point of OS X 10.3 possibly becoming the largest > > install > >>base of Python of any version (it will be 2.3) came up. > > > I an somehow amused at the prospect of Apple OX (Uni)X releases > influencing the timing of Python releases. > I personally find it cool, but I am an OSX user, so I am biased beyond belief. > >>With rough estimates being thrown around of 5 million installs >>in about a year's >>time, the point that making it difficult to run Python 2.3.x code on >>that size of an install base would be bad. > > > I've begun work on a 2.3 based project. If I were to get users who > have received rather than installed Python, I would be more hesitant > to simply say 'upgrade', especially for minor features in a 2.3.x. > > >>that this is Mac it should not be expected that most users will want > > to, > >>let alone know how, to add a secondary install of Python since the >>original is used by the OS and thus should not be overwritten). > > > I presume a Mac installer would know how to add rather than overwrite. > Depends on how the installer is set up. Most OS X apps are fully self-contained within a directory (when you see an application icon it actually is just a directory that happens to follow a specific naming scheme and layout). So Python apps that needs a different version of the interpreter could include their own copy with their app (perk of there only being one CPU family is you only need to compile once). But for a site-wide install, that gets a little trickier. It is definitely doable, but there is not much of a standard (I would assume it would be in either /Applications or ~/Applications , but no guarantees; could specify at install and store it somewhere, I guess) and there is always the UNIX way but that is not as much of a worry since if you are mucking with the UNIX side you should know how to handle this situation. > I am curious whether anyone has ever suggested to PC makers that > Python + PyWin would be a useful addition to their Windows package -- > both for setup/admin scripts (as done by RedHat. ..., and now > Apple -- I don't know the specifics) and as a freebie addon (Basic for > the 21st Century)? > HP/Compaq uses Python for something. webmaster at python.org gets email every so often about an HP/Compaq user wanting to know what Python is and whether they can uninstall it safely. As for OS X, they are using it apparently for driving PDF generation (or at least that was my understanding). And I don't think MS wants competition for VB.NET in the form of Python. But it would be nice. -Brett From nick at nick.uklinux.net Fri Sep 19 14:11:28 2003 From: nick at nick.uklinux.net (Nick Roberts) Date: Fri Sep 19 14:17:58 2003 Subject: [Python-Dev] Enhancement to pdb in gud.el In-Reply-To: <3F6B11BA.6000008@sabaydi.com> References: <3F6B11BA.6000008@sabaydi.com> Message-ID: <16235.18128.422738.929390@nick.uklinux.net> Kevin J. Butler writes: > > A couple of issues: > > ... I know very little about python (!). I'm trying to extend what I have done in Emacs to other languages. What's the relationship between bdb.py and pdb.py? What message would your patch generate when a breakpoint is set/cleared? Nick From python-kbutler at sabaydi.com Fri Sep 19 15:04:30 2003 From: python-kbutler at sabaydi.com (Kevin J. Butler) Date: Fri Sep 19 15:04:56 2003 Subject: [Python-Dev] Enhancement to pdb in gud.el In-Reply-To: <16235.18128.422738.929390@nick.uklinux.net> References: <3F6B11BA.6000008@sabaydi.com> <16235.18128.422738.929390@nick.uklinux.net> Message-ID: <3F6B533E.5000403@sabaydi.com> Nick Roberts wrote: >Kevin J. Butler writes: > > > > A couple of issues: > > > > ... > >I know very little about python (!). I'm trying to extend what I have done in >Emacs to other languages. What's the relationship between bdb.py and pdb.py? >What message would your patch generate when a breakpoint is set/cleared? > > Sorry, bdb.py provides back-end features used by UI code in pdb.py - part of the fix went into each. After my patch: (Pdb) cl 3 Deleted breakpoint 3 at z:\work\wm\wm.py:136 (Pdb) cl wm.py:17 Deleted breakpoint 5 at z:\work\wm\wm.py:17 kb From tim.one at comcast.net Fri Sep 19 15:11:14 2003 From: tim.one at comcast.net (Tim Peters) Date: Fri Sep 19 15:11:25 2003 Subject: [Python-Dev] Fun with 2.3 shutdown In-Reply-To: Message-ID: [Tim] > When the Zope3 tests are run under Python 2.3, after the test runner > ends we usually get treated to a long string of these things: > > """ > Unhandled exception in thread started by > Error in sys.excepthook: > Original exception was: > > """ [bunch of analysis deleted] > ... > Event.wait() with a timeout ends up in _Condition.wait(), where a > lazy busy loop wakes up about 20 times per second to see whether it > can proceed. > > For some reason an exception is getting raised in the wait() code. > I'm not exactly sure what or why yet, but that will come soon enough. It didn't. The primary effect of adding some vanilla debugging prints to threading.py's _Condition.wait() was to make Python die with segfaults at shutdown time instead. If Python's *second* call to PyGC_collect() in Py_Finalize() is commented out (the call that occurs after /* Destroy all modules */ PyImport_Cleanup(); ), all the problems go away, including the nonsense errors sprayed out at the end of Zope3 test runs. Recall that the nonsense errors are caused by a dozen stale daemon threads trying to execute Python code after the interpreter has been severely torn down, and they get the *chance* to do this because the second PyGC_collect() finds trash with Python __del__ methods (so PyGC_collect() loses the GIL when calling the __del__ methods, and all the daemon threads can proceed then). Note that this isn't a problem with code *in* __del__ methods! That makes it a different kind of shutdown glitch than we've usually wrestled with. It's a problem with Python code that has nothing to do with __del__; __del__'s only contribution is to release the GIL. Because the second PyGC_collect() *is* finding addtional finalizers to run, it's unattractive to stop calling it (getting more user-defined finalizers to run was the purpose of adding these PyGC_collect() shutdown calls to 2.3). OTOH, any __del__ method that runs after PyImport_Cleanup() will be (AFAICT) just as vulnerable to producing nonsense errors and segfaults as the code in _Condition.wait() has proven to be (sys is useless by that point, and all the Python internal code sucking basic objects out of sys isn't expecting to get None back). Maybe we should remove the second PyGC_collect() call before more apps run into these mysteries. Maybe we should delay tearing down sys as a special case (even more of a special case than it is now). Maybe the Zope3 tests should stop leaving an ever-growing number of daemon threads around (which appears to be the only solution so long as they're run under Python 2.3). From martin at v.loewis.de Fri Sep 19 16:43:46 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Fri Sep 19 16:44:10 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: <3F6AE417.2060301@lemburg.com> References: <3F6AE417.2060301@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > Since wchar_t is the only case where a signed type can pop > up, why not extend the autoconf test to check for signedness > and then reject signed wchar_t value as not-usable (ie. > undefine HAVE_USABLE_WCHAR_T). Because that would exclude a number of relevant systems where wchar_t would be usable. Regards, Martin From jvb at prairienet.org Fri Sep 19 17:26:08 2003 From: jvb at prairienet.org (John Belmonte) Date: Fri Sep 19 17:26:16 2003 Subject: [Python-Dev] Improving the CGI module In-Reply-To: <3F68F4F7.6020601@bath.ac.uk> References: <3F68F4F7.6020601@bath.ac.uk> Message-ID: <3F6B7470.1060407@prairienet.org> Simon Willison wrote: > I've recently started using Python for server side web development, and > I've been running in to some limitations in the CGI module. ... > I have previous experience with PHP, which has a number of extremely > convenient mechanisms for dealing with form input and cookies that I > sorely miss when working with Python. I have started work on porting > some of these capabilities, but it struck me that several of the ideas > represented would make valuable additions to the Python standard library > itself. Recently I was also thinking there is a lot of room for improvement. Although the Python libraries have objects that represent HTTP request and responses on the client side, there is no such thing for the server side to use from a CGI script. (There is a little of this functionality mixed in with the HTTP server modules however.) I would like to see modules such as Quixote's http_request and http_response (which were taken from Zope) improved and incorporated into the standard library. They do the essentials like manage headers, status, encoding, and cookies. Something I found useful was that they can even parse headers such as Accept and give you a dictionary of MIME types that the client will accept. The mod_python modules might also offer some ideas. Also, in any improvements that are made, I would like to see things kept general enough to support the REST architecture style [1]. In other words, don't assume that the client is always a web browser, or that the body of a POST is always a form, etc. HTTP and CGI are for more than just serving web pages. Regards, -John Belmonte [1] http://internet.conveyor.com/RESTwiki/ -- http:// if le.o / From ianb at colorstudy.com Fri Sep 19 17:48:55 2003 From: ianb at colorstudy.com (Ian Bicking) Date: Fri Sep 19 17:51:47 2003 Subject: [Python-Dev] Improving the CGI module In-Reply-To: <3F6B7470.1060407@prairienet.org> Message-ID: <11097042-EAEB-11D7-A692-000393C2D67E@colorstudy.com> On Friday, September 19, 2003, at 04:26 PM, John Belmonte wrote: > Simon Willison wrote: >> I've recently started using Python for server side web development, >> and I've been running in to some limitations in the CGI module. > ... >> I have previous experience with PHP, which has a number of extremely >> convenient mechanisms for dealing with form input and cookies that I >> sorely miss when working with Python. I have started work on porting >> some of these capabilities, but it struck me that several of the >> ideas represented would make valuable additions to the Python >> standard library itself. > > Recently I was also thinking there is a lot of room for improvement. > Although the Python libraries have objects that represent HTTP request > and responses on the client side, there is no such thing for the > server side to use from a CGI script. (There is a little of this > functionality mixed in with the HTTP server modules however.) > > I would like to see modules such as Quixote's http_request and > http_response (which were taken from Zope) improved and incorporated > into the standard library. They do the essentials like manage > headers, status, encoding, and cookies. Something I found useful was > that they can even parse headers such as Accept and give you a > dictionary of MIME types that the client will accept. The mod_python > modules might also offer some ideas. I think this would be an excellent idea -- providing some (*any*) basic request and response implementation/interface would (hopefully) provide some sort of basic vocabulary for generic Python web packages (i.e., packages that aren't tied specifically to one framework). Right now there isn't even a lowest common denominator, even though the differences in the implementation of these interfaces isn't that great. If all a cgi (or web or whatever) module did was provide a request and response object, that'd be a big improvement. With certain criteria, I guess: it should be easy to create from a CGI context, but not difficult from other contexts (i.e., given any file-like input and output, and a dictionary of CGI-environment-like variables). It should also be general enough for most purposes, like allowing streamed output, you mentioned accepting any POST body, probably some other stuff. But none of that is particular difficult to do. > Also, in any improvements that are made, I would like to see things > kept general enough to support the REST architecture style [1]. In > other words, don't assume that the client is always a web browser, or > that the body of a POST is always a form, etc. HTTP and CGI are for > more than just serving web pages. From nick at nick.uklinux.net Sat Sep 20 11:20:50 2003 From: nick at nick.uklinux.net (Nick Roberts) Date: Sat Sep 20 11:27:38 2003 Subject: [Python-Dev] Enhancement to pdb in gud.el In-Reply-To: <3F6B533E.5000403@sabaydi.com> References: <3F6B11BA.6000008@sabaydi.com> <16235.18128.422738.929390@nick.uklinux.net> <3F6B533E.5000403@sabaydi.com> Message-ID: <16236.28754.105629.565153@nick.uklinux.net> Kevin J. Butler writes: > ... > > After my patch: > (Pdb) cl 3 > Deleted breakpoint 3 at z:\work\wm\wm.py:136 > (Pdb) cl wm.py:17 > Deleted breakpoint 5 at z:\work\wm\wm.py:17 I've applied your patch manually to my files bdb.py and pdb.py (I have 2.2) and it appears to work. If you load the file below in Emacs (M-x eval-buffer or M-x load-file), then type M-x pdb, the debugger (with your patch) should work with breakpoint icons in the display margin. Emacs must be a version from the CVS repository to work straight away. This will also give you buttons for debugging if you turn the toolbar on (M-x tool-bar-mode). I've tested it on GNU/Linux but it should work on Windows too if I've defined the regexp correctly. I've just noticed that you can enable/disable breakpoints in python. In my mode for gdb, the breakpoints are greyed out when the breakpoints become disabled. I could do this for python too if these commands were changed to emit a message. If your patches are accepted (by SF?) then I will integrate my changes into gud.el in the Emacs CVS repository. Nick ----------------------------------------------------- (require 'gud) (defun gud-sentinel (proc msg) (cond ((null (buffer-name (process-buffer proc))) ;; buffer killed ;; Stop displaying an arrow in a source file. (setq overlay-arrow-position nil) (set-process-buffer proc nil) (if (memq gud-minor-mode-type '(gdba pdb)) (gdb-reset) (gud-reset))) ((memq (process-status proc) '(signal exit)) ;; Stop displaying an arrow in a source file. (setq overlay-arrow-position nil) (with-current-buffer gud-comint-buffer (if (memq gud-minor-mode '(gdba pdb)) (gdb-reset) (gud-reset))) (let* ((obuf (current-buffer))) ;; save-excursion isn't the right thing if ;; process-buffer is current-buffer (unwind-protect (progn ;; Write something in *compilation* and hack its mode line, (set-buffer (process-buffer proc)) ;; Fix the mode line. (setq mode-line-process (concat ":" (symbol-name (process-status proc)))) (force-mode-line-update) (if (eobp) (insert ?\n mode-name " " msg) (save-excursion (goto-char (point-max)) (insert ?\n mode-name " " msg))) ;; If buffer and mode line will show that the process ;; is dead, we can delete it now. Otherwise it ;; will stay around until M-x list-processes. (delete-process proc)) ;; Restore old buffer, but don't restore old point ;; if obuf is the gud buffer. (set-buffer obuf)))))) (defun gdb-reset () "Exit a debugging session cleanly by killing the gdb buffers and resetting the source buffers." (dolist (buffer (buffer-list)) (if (not (eq buffer gud-comint-buffer)) (with-current-buffer buffer (if (memq gud-minor-mode '(gdba pdb)) (if (string-match "^\*.+*$" (buffer-name)) (kill-buffer nil) (if (display-images-p) (remove-images (point-min) (point-max)) (gdb-remove-strings (point-min) (point-max))) (setq left-margin-width 0) (setq gud-minor-mode nil) (kill-local-variable 'tool-bar-map) (setq gud-running nil) (if (get-buffer-window (current-buffer)) (set-window-margins (get-buffer-window (current-buffer)) left-margin-width right-margin-width)))))))) (defun gdb-put-string (putstring pos) "Put string PUTSTRING in front of POS in the current buffer. PUTSTRING is displayed by putting an overlay into the current buffer with a `before-string' STRING that has a `display' property whose value is PUTSTRING." (let ((gdb-string "x") (buffer (current-buffer))) (let ((overlay (make-overlay pos pos buffer)) (prop (list (list 'margin 'left-margin) putstring))) (put-text-property 0 (length gdb-string) 'display prop gdb-string) (overlay-put overlay 'put-break t) (overlay-put overlay 'before-string gdb-string)))) (defun gdb-remove-strings (start end &optional buffer) "Remove strings between START and END in BUFFER. Remove only strings that were put in BUFFER with calls to `put-string'. BUFFER nil or omitted means use the current buffer." (unless buffer (setq buffer (current-buffer))) (let ((overlays (overlays-in start end))) (while overlays (let ((overlay (car overlays))) (when (overlay-get overlay 'put-break) (delete-overlay overlay))) (setq overlays (cdr overlays))))) (defconst breakpoint-xpm-data "/* XPM */ static char *magick[] = { /* columns rows colors chars-per-pixel */ \"12 12 2 1\", \" c red\", \"+ c None\", /* pixels */ \"++++++++++++\", \"+++ +++\", \"++ ++\", \"+ +\", \"+ +\", \"+ +\", \"+ +\", \"+ +\", \"+ +\", \"++ ++\", \"+++ +++\", \"++++++++++++\" };" "XPM data used for breakpoint icon.") (defconst breakpoint-enabled-pbm-data "P1 12 12\", 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0" "PBM data used for enabled breakpoint icon.") (defvar breakpoint-enabled-icon (find-image `((:type xpm :data ,breakpoint-xpm-data) (:type pbm :data ,breakpoint-enabled-pbm-data))) "Icon for enabled breakpoint in display margin") ;; ====================================================================== ;; pdb (Python debugger) functions ;; History of argument lists passed to pdb. (defvar gud-pdb-history nil) ;; Last group is for return value, e.g. "> test.py(2)foo()->None" ;; Either file or function name may be omitted: "> (0)?()" (defvar gud-pdb-marker-regexp "^> \\([-a-zA-Z0-9_/.:\\]*\\|\\)(\\([0-9]+\\))\\([a-zA-Z0-9_]*\\|\\?\\)()\\(->[^\n]*\\)?\n") (defvar gud-pdb-marker-regexp-file-group 1) (defvar gud-pdb-marker-regexp-line-group 2) (defvar gud-pdb-marker-regexp-fnname-group 3) (defvar gud-pdb-marker-regexp-start "^> ") (defvar gud-pdb-marker-regexp-breakpoint "reakpoint [0-9]+ at \\(\\([a-zA-Z]:\\)?[^:\n]*\\):\\([0-9]*\\)\n") ;; There's no guarantee that Emacs will hand the filter the entire ;; marker at once; it could be broken up across several strings. We ;; might even receive a big chunk with several markers in it. If we ;; receive a chunk of text which looks like it might contain the ;; beginning of a marker, we save it here between calls to the ;; filter. (defun gud-pdb-marker-filter (string) (setq gud-marker-acc (concat gud-marker-acc string)) (let ((output "")) ;; Process all the complete markers in this chunk. (while (string-match gud-pdb-marker-regexp gud-marker-acc) (setq ;; Extract the frame position from the marker. gud-last-frame (let ((file (match-string gud-pdb-marker-regexp-file-group gud-marker-acc)) (line (string-to-int (match-string gud-pdb-marker-regexp-line-group gud-marker-acc)))) (if (string-equal file "") gud-last-frame (cons file line))) ;; Output everything instead of the below output (concat output (substring gud-marker-acc 0 (match-end 0))) ;; ;; Append any text before the marker to the output we're going ;; ;; to return - we don't include the marker in this text. ;; output (concat output ;; (substring gud-marker-acc 0 (match-beginning 0))) ;; Set the accumulator to the remaining text. gud-marker-acc (substring gud-marker-acc (match-end 0)))) (if (string-match (concat "B" gud-pdb-marker-regexp-breakpoint) gud-marker-acc) (let ((file (match-string 1 gud-marker-acc)) (line (match-string 3 gud-marker-acc))) (gud-pdb-insert-breakpoint file line))) (if (string-match (concat "Deleted b" gud-pdb-marker-regexp-breakpoint) gud-marker-acc) (let ((file (match-string 1 gud-marker-acc)) (line (match-string 3 gud-marker-acc))) (gud-pdb-remove-breakpoint file line))) ;; Does the remaining text look like it might end with the ;; beginning of another marker? If it does, then keep it in ;; gud-marker-acc until we receive the rest of it. Since we ;; know the full marker regexp above failed, it's pretty simple to ;; test for marker starts. (if (string-match gud-pdb-marker-regexp-start gud-marker-acc) (progn ;; Everything before the potential marker start can be output. (setq output (concat output (substring gud-marker-acc 0 (match-beginning 0)))) ;; Everything after, we save, to combine with later input. (setq gud-marker-acc (substring gud-marker-acc (match-beginning 0)))) (setq output (concat output gud-marker-acc) gud-marker-acc "")) output)) (defun gud-pdb-insert-breakpoint (file line) (with-current-buffer (find-file-noselect file) (save-current-buffer (set (make-local-variable 'gud-minor-mode) 'pdb) (set (make-local-variable 'tool-bar-map) gud-tool-bar-map) (setq left-margin-width 2) (if (get-buffer-window (current-buffer)) (set-window-margins (get-buffer-window (current-buffer)) left-margin-width right-margin-width))) (save-excursion (goto-line (string-to-number line)) (let ((start (progn (beginning-of-line) (- (point) 1))) (end (progn (end-of-line) (+ (point) 1)))) (if (display-images-p) (progn (remove-images start end) (put-image breakpoint-enabled-icon (+ start 1) "breakpoint icon enabled" 'left-margin)) (gdb-remove-strings start end) (gdb-put-string "B" (+ start 1))))))) (defun gud-pdb-remove-breakpoint (file line) (with-current-buffer (find-file-noselect file) (save-excursion (goto-line (string-to-number line)) (let ((start (progn (beginning-of-line) (- (point) 1))) (end (progn (end-of-line) (+ (point) 1)))) (if (display-images-p) (remove-images start end) (gdb-remove-strings start end)))))) (defcustom gud-pdb-command-name "pdb" "File name for executing the Python debugger. This should be an executable on your path, or an absolute file name." :type 'string :group 'gud) (defun pdb (command-line) "Run pdb on program FILE in buffer `*gud-FILE*'. The directory containing FILE becomes the initial working directory and source-file directory for your debugger." (interactive (list (gud-query-cmdline 'pdb))) (gud-common-init command-line nil 'gud-pdb-marker-filter) (set (make-local-variable 'gud-minor-mode) 'pdb) (gud-def gud-break "break %l" "\C-b" "Set breakpoint at current line.") (gud-def gud-remove "clear %f:%l" "\C-d" "Remove breakpoint at current line") (gud-def gud-step "step" "\C-s" "Step one source line with display.") (gud-def gud-next "next" "\C-n" "Step one line (skip functions).") (gud-def gud-cont "continue" "\C-r" "Continue with display.") (gud-def gud-finish "return" "\C-f" "Finish executing current function.") (gud-def gud-up "up" "<" "Up one stack frame.") (gud-def gud-down "down" ">" "Down one stack frame.") (gud-def gud-print "p %e" "\C-p" "Evaluate Python expression at point.") ;; Is this right? (gud-def gud-statement "! %e" "\C-e" "Execute Python statement at point.") ;; (setq comint-prompt-regexp "^(.*pdb[+]?) *") (setq comint-prompt-regexp "^(Pdb) *") (setq paragraph-start comint-prompt-regexp) (run-hooks 'pdb-mode-hook)) From public-kbutler at sabaydi.com Sat Sep 20 12:06:33 2003 From: public-kbutler at sabaydi.com (Kevin J. Butler) Date: Sat Sep 20 12:06:59 2003 Subject: [Python-Dev] Enhancement to pdb in gud.el In-Reply-To: <16236.28754.105629.565153@nick.uklinux.net> References: <3F6B11BA.6000008@sabaydi.com> <16235.18128.422738.929390@nick.uklinux.net> <3F6B533E.5000403@sabaydi.com> <16236.28754.105629.565153@nick.uklinux.net> Message-ID: <3F6C7B09.5060608@sabaydi.com> Nick Roberts wrote: > I've just noticed that you can enable/disable breakpoints in python. In my > mode for gdb, the breakpoints are greyed out when the breakpoints become > disabled. I could do this for python too if these commands were changed to > emit a message. Two additional lines will add that - see below. > If your patches are accepted (by SF?) then I will integrate my changes into > gud.el in the Emacs CVS repository. OK, I'll go ahead and submit them... However, what is the fallback behavior when emacs with the new gud.el encounters a pdb.py that doesn't have the patch? That is, will users of existing versions of python lose any functionality? kb *** pdb.py.p1 Sat Sep 20 09:54:20 2003 --- pdb.py Sat Sep 20 09:58:22 2003 *************** *** 388,393 **** --- 388,394 ---- bp = bdb.Breakpoint.bpbynumber[i] if bp: bp.enable() + print 'Enabled breakpoint %s at %s:%s' % (i, bp.file, bp.line) def do_disable(self, arg): args = arg.split() *************** *** 405,410 **** --- 406,412 ---- bp = bdb.Breakpoint.bpbynumber[i] if bp: bp.disable() + print 'Disabled breakpoint %s at %s:%s' % (i, bp.file, bp.line) def do_condition(self, arg): # arg is breakpoint number and condition From anthony at interlink.com.au Sat Sep 20 12:16:35 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat Sep 20 12:18:15 2003 Subject: [Python-Dev] Py2.3.1 Message-ID: <200309201616.h8KGGaf5021370@localhost.localdomain> I'm planning on starting the dance of the seven release tarballs Tuesday morning AU time, which is more or less 0:00 Tuesday UTC. Assuming that no-one objects to this timeframe, could people be _extremely_ conservative with checkins on the branch for the 24 hours before this? I'm going to be starting to cut test release tarballs probably some time tomorrow and building it on as many platforms as I can ssh to easily - I can make this available to other people if they want to help test, or if they've got some sort of strange version of Unix (Irix and HP/UX are always good ones to check for platform oddities). Anthony -- Anthony Baxter It's never too late to have a happy childhood. From nick at nick.uklinux.net Sat Sep 20 17:33:56 2003 From: nick at nick.uklinux.net (Nick Roberts) Date: Sat Sep 20 17:45:44 2003 Subject: [Python-Dev] Enhancement to pdb in gud.el In-Reply-To: <3F6C7B09.5060608@sabaydi.com> References: <3F6B11BA.6000008@sabaydi.com> <16235.18128.422738.929390@nick.uklinux.net> <3F6B533E.5000403@sabaydi.com> <16236.28754.105629.565153@nick.uklinux.net> <3F6C7B09.5060608@sabaydi.com> Message-ID: <16236.51140.284372.526221@nick.uklinux.net> > However, what is the fallback behavior when emacs with the new gud.el > encounters a pdb.py that doesn't have the patch? That is, will users of > existing versions of python lose any functionality? The messages from clear, enable and disable will be new, so the existing behaviour will hold for existing versions of python. However, the message from break (and tbreak) will not have changed, but the way gud.el acts on it will. So, break will place a breakpoint icon that clear can't remove. However, these changes will go on the main trunk of the CVS version of Emacs which is unlikely to be released this year. Emacs development is relatively slow and the last non-bugfix release was Sepember 2001 (21.1). How often are new versions of Python released? Providing the next release of Python is before that of Emacs, I dont think there's a problem. Nick From tim.one at comcast.net Sat Sep 20 18:49:53 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Sep 20 18:49:58 2003 Subject: [Python-Dev] Py2.3.1 In-Reply-To: <200309201616.h8KGGaf5021370@localhost.localdomain> Message-ID: [Anthony Baxter] > I'm planning on starting the dance of the seven release tarballs > Tuesday morning AU time, which is more or less 0:00 Tuesday UTC. > Assuming that no-one objects to this timeframe, could people be > _extremely_ conservative with checkins on the branch for the 24 hours > before this? Since CVS doesn't store metadata about branches, whenever mentioning a branch it's always helpful to name the exact branch tag you're talking about. Are you talking about the release23-maint branch here? If so, please note that I just made a few checkins to repair Windows breakage in the test suite, on the HEAD, but I don't expect to have time to backport that stuff (it's not hard, I've just got too much else to do now and don't even have a release23-maint checkout yet). Good luck! From tim.one at comcast.net Sat Sep 20 19:15:08 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Sep 20 19:15:15 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: Message-ID: [M.-A. Lemburg] >> Since wchar_t is the only case where a signed type can pop >> up, why not extend the autoconf test to check for signedness >> and then reject signed wchar_t value as not-usable (ie. >> undefine HAVE_USABLE_WCHAR_T). [martin@v.loewis.de] > Because that would exclude a number of relevant systems where wchar_t > would be usable. So what if MAL ammened his suggestion to reject signed 2-byte wchar_t value as not-usable +++++++ ? From bac at OCF.Berkeley.EDU Sat Sep 20 20:27:34 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sat Sep 20 20:27:37 2003 Subject: [Python-Dev] Enhancement to pdb in gud.el In-Reply-To: <16236.51140.284372.526221@nick.uklinux.net> References: <3F6B11BA.6000008@sabaydi.com> <16235.18128.422738.929390@nick.uklinux.net> <3F6B533E.5000403@sabaydi.com> <16236.28754.105629.565153@nick.uklinux.net> <3F6C7B09.5060608@sabaydi.com> <16236.51140.284372.526221@nick.uklinux.net> Message-ID: <3F6CF076.30409@ocf.berkeley.edu> Nick Roberts wrote: > > However, what is the fallback behavior when emacs with the new gud.el > > encounters a pdb.py that doesn't have the patch? That is, will users of > > existing versions of python lose any functionality? > > The messages from clear, enable and disable will be new, so the existing > behaviour will hold for existing versions of python. However, the message from > break (and tbreak) will not have changed, but the way gud.el acts on it will. > So, break will place a breakpoint icon that clear can't remove. > > However, these changes will go on the main trunk of the CVS version of Emacs > which is unlikely to be released this year. Emacs development is relatively > slow and the last non-bugfix release was Sepember 2001 (21.1). How often are > new versions of Python released? Providing the next release of Python is > before that of Emacs, I dont think there's a problem. > The frequency of Python releases is actually being debated as of this moment. Some have suggested having a release once a year (which seems to be winning unanimous support). If we stick with the old way (I couldn't count on it) then who knows; probably year to a year and a half. As for this thread, is this the proper place for this? Wouldn't on the python-mode mailing list (http://mail.python.org/mailman/listinfo/python-mode) be a better place for Emacs-related stuff? Or is my Vim bias influencing me too much? -Brett From theller at python.net Fri Sep 19 15:35:31 2003 From: theller at python.net (Thomas Heller) Date: Sat Sep 20 21:27:39 2003 Subject: [Python-Dev] 2.3.1 windows binaries Message-ID: For the 2.3.1 Windows binaries, I've followed the instructions in PCBuild\readme.txt and downloaded the sources for the additional packages. I have bzip2-1.0.2.tar.gz db-4.1.25.NC.zip tcl843-src.zip tk843-src.zip openssl-0.9.7b.tar.gz and zlib-1.1.4. All these versions are exactly those the readme file mentions, with the exception of openssl. Quote from the PCBuild\readme.txt: Get the latest source code for OpenSSL from http://www.openssl.org You (probably) don't want the "engine" code. For example, get openssl-0.9.6g.tar.gz not openssl-engine-0.9.6g.tar.gz Unpack into the "dist" directory, retaining the folder name from the archive - for example, the latest stable OpenSSL will install as dist/openssl-0.9.6g You can (theoretically) use any version of OpenSSL you like - the build process will automatically select the latest version. Is it important to use the same openssl version that Python 2.3.0 used, or is the readme file correct in saying that the latest version is the one to use? And does this have to be mentioned somewhere (Misc/NEWS, the readme file for windows)? Thomas From tim.one at comcast.net Sat Sep 20 21:35:27 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Sep 20 21:35:30 2003 Subject: [Python-Dev] 2.3.1 windows binaries In-Reply-To: Message-ID: [Thomas Heller] > For the 2.3.1 Windows binaries, I've followed the instructions in > PCBuild\readme.txt and downloaded the sources for the additional > packages. I have > > bzip2-1.0.2.tar.gz > db-4.1.25.NC.zip > tcl843-src.zip > tk843-src.zip > openssl-0.9.7b.tar.gz > > and zlib-1.1.4. > > All these versions are exactly those the readme file mentions, with > the exception of openssl. Quote from the PCBuild\readme.txt: > > Get the latest source code for OpenSSL from > http://www.openssl.org > > You (probably) don't want the "engine" code. For example, get > openssl-0.9.6g.tar.gz > not > openssl-engine-0.9.6g.tar.gz > > Unpack into the "dist" directory, retaining the folder name from > the archive - for example, the latest stable OpenSSL will install > as dist/openssl-0.9.6g > > You can (theoretically) use any version of OpenSSL you like - the > build process will automatically select the latest version. > > Is it important to use the same openssl version that Python 2.3.0 > used, or is the readme file correct in saying that the latest version > is the one to use? Nobody knows, because nobody (AFAIK) has ever tried building the Windows Python with an OpenSSL release other than the one mentioned in README.txt. Try it and see whether it works? That would be a happier suggestion if the Python test suite exercised more of the SSL code. > And does this have to be mentioned somewhere (Misc/NEWS, the readme > file for windows)? PCbuild\readme.txt in a given release is intended to describe exactly how that release was built, in detail sufficient so that someone other than the person who built the release stands a good chance of reproducing the whole bit. I've also added a NEWS entry when moving to a different release of any of the 3rd-party packages. That's just full disclosure . From public-kbutler at sabaydi.com Sat Sep 20 23:27:51 2003 From: public-kbutler at sabaydi.com (Kevin J. Butler) Date: Sat Sep 20 23:28:07 2003 Subject: [Python-Dev] Enhancement to pdb in gud.el In-Reply-To: <3F6CF076.30409@ocf.berkeley.edu> References: <3F6B11BA.6000008@sabaydi.com> <16235.18128.422738.929390@nick.uklinux.net> <3F6B533E.5000403@sabaydi.com> <16236.28754.105629.565153@nick.uklinux.net> <3F6C7B09.5060608@sabaydi.com> <16236.51140.284372.526221@nick.uklinux.net> <3F6CF076.30409@ocf.berkeley.edu> Message-ID: <3F6D1AB7.1050608@sabaydi.com> > Nick Roberts wrote: > >> > However, what is the fallback behavior when emacs with the new >> gud.el > encounters a pdb.py that doesn't have the patch? That is, >> will users of > existing versions of python lose any functionality? >> >> The messages from clear, enable and disable will be new, so the existing >> behaviour will hold for existing versions of python. However, the >> message from >> break (and tbreak) will not have changed, but the way gud.el acts on >> it will. >> So, break will place a breakpoint icon that clear can't remove. >> However, these changes will go on the main trunk of the CVS version of >> Emacs >> which is unlikely to be released this year. Emacs development is >> relatively >> slow and the last non-bugfix release was Sepember 2001 (21.1). How >> often are >> new versions of Python released? Providing the next release of Python is >> before that of Emacs, I dont think there's a problem. Well, older versions of Python tend to remain in usage long after newer versions are available - for instance, 1.5.2 is still the standard in some places, and there is virtually no chance of patching 1.5.2 for something like this. It sounds like the emacs-side misbehavior will be cosmetic only, so probably not too bad. Brett C. wrote: > As for this thread, is this the proper place for this? Wouldn't on the > python-mode mailing list > (http://mail.python.org/mailman/listinfo/python-mode) be a better place > for Emacs-related stuff? Or is my Vim bias influencing me too much? I think the Vim bias has is influencing you too much. It has been known to do that. ;-) (Speaking as a recovering/apostate vi user). I consider this more pdb-related than emacs-specific - I'd expect that integration with, say, Eclipse would benefit from this enhancement as well, so I've been hoping to get some feedback from the list... BTW, I've submitted the patch, it is 809887: http://sourceforge.net/tracker/index.php?func=detail&aid=809887&group_id=5470&atid=305470 kb From martin at v.loewis.de Sun Sep 21 03:38:03 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sun Sep 21 03:38:16 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: References: Message-ID: "Tim Peters" writes: > So what if MAL ammened his suggestion to > > reject signed 2-byte wchar_t value as not-usable > +++++++ > > ? That would be a very sensible suggestion. Regards, Martin From anthony at interlink.com.au Sun Sep 21 03:45:15 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Sun Sep 21 03:46:48 2003 Subject: [Python-Dev] Py2.3.1 In-Reply-To: Message-ID: <200309210745.h8L7jGrU002249@localhost.localdomain> >>> "Tim Peters" wrote > Since CVS doesn't store metadata about branches, whenever mentioning a > branch it's always helpful to name the exact branch tag you're talking > about. Are you talking about the release23-maint branch here? Yep, that's the branch I was referring to.. > If so, please note that I just made a few checkins to repair Windows > breakage in the test suite, on the HEAD, but I don't expect to have time to > backport that stuff (it's not hard, I've just got too much else to do now > and don't even have a release23-maint checkout yet). I think Raymond's already done the backporting. -- Anthony Baxter It's never too late to have a happy childhood. From python at rcn.com Sun Sep 21 04:37:01 2003 From: python at rcn.com (Raymond Hettinger) Date: Sun Sep 21 04:37:19 2003 Subject: [Python-Dev] Py2.3.1 References: <200309210745.h8L7jGrU002249@localhost.localdomain> Message-ID: <005901c3801b$877d1620$e841fea9@oemcomputer> > > If so, please note that I just made a few checkins to repair Windows > > breakage in the test suite, on the HEAD, but I don't expect to have time to > > backport that stuff (it's not hard, I've just got too much else to do now > > and don't even have a release23-maint checkout yet). > > I think Raymond's already done the backporting. Yes. There was only one that applied to Py2.3.1. All tests are passing on Windows except for test_bsddb3 which fails for both 2.3.1 and 2.4. I've sent Greg a note with the test output to see what he can make of it. Raymond ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From jim at zope.com Sun Sep 21 06:06:45 2003 From: jim at zope.com (Jim Fulton) Date: Sun Sep 21 06:07:29 2003 Subject: [Python-Dev] Fun with 2.3 shutdown In-Reply-To: <20030919120100.GB23317@vicky.ecs.soton.ac.uk> References: <20030919120100.GB23317@vicky.ecs.soton.ac.uk> Message-ID: <3F6D7835.5080407@zope.com> Armin Rigo wrote: ... > Another question, what was the rationale for setting module globals to None > instead of simply deleting them (and getting the expected AttributeErrors both > at C and at Python level) ? Or, given cyclic GC, why isn't it enough to clear sys.modules? In theory, shouldn't that make all modules (and thier globals) collectable unless there are threads hanging on to them? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (703) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From mal at lemburg.com Sun Sep 21 07:30:27 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Sep 21 07:30:22 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: References: Message-ID: <3F6D8BD3.7090600@lemburg.com> Tim Peters wrote: > [M.-A. Lemburg] > >>>Since wchar_t is the only case where a signed type can pop >>>up, why not extend the autoconf test to check for signedness >>>and then reject signed wchar_t value as not-usable (ie. >>>undefine HAVE_USABLE_WCHAR_T). > > [martin@v.loewis.de] > >>Because that would exclude a number of relevant systems where wchar_t >>would be usable. > > So what if MAL ammened his suggestion to > > reject signed 2-byte wchar_t value as not-usable > +++++++ > ? That would not solve the problem. Note that we have proper conversion routines that allow converting between wchar_t and Py_UNICODE. These routines must be used for conversions anyway (even if Py_UNICODE and wchar_t happen to be the same type), so from a programmer perspective changing Py_UNICODE to be unsigned won't be noticed and we don't lose anything much. Again, I don't see the point in using a signed type for data that doesn't have any concept of signed values. It's just bad design and we shouldn't try to go down the same route if we don't have to. The Unicode implementation has always defined Py_UNICODE to be an unsigned type; see the Unicode PEP 100: """ Internal Format The internal format for Unicode objects should use a Python specific fixed format implemented as 'unsigned short' (or another unsigned numeric type having 16 bits). Byte order is platform dependent. ... The configure script should provide aid in deciding whether Python can use the native wchar_t type or not (it has to be a 16-bit unsigned type). """ Python can also deal with UCS4 now, but the concept remains the same. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Sep 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From tim.one at comcast.net Sun Sep 21 10:48:21 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Sep 21 10:48:29 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: <3F6D8BD3.7090600@lemburg.com> Message-ID: [Tim] >> So what if MAL ammened his suggestion to >> >> reject signed 2-byte wchar_t value as not-usable >> +++++++ ? [M.-A. Lemburg] > That would not solve the problem. Then what is the problem, specifically? I thought you agreed with Martin that a signed 32-bit type doesn't hurt, since the sign bit remains clear then in all cases of Unicode data. > Note that we have proper conversion routines that allow > converting between wchar_t and Py_UNICODE. These routines must > be used for conversions anyway (even if Py_UNICODE and wchar_t > happen to be the same type), so from a programmer perspective > changing Py_UNICODE to be unsigned won't be noticed and we > don't lose anything much. > > Again, I don't see the point in using a signed type for data > that doesn't have any concept of signed values. It's just > bad design and we shouldn't try to go down the same route > if we don't have to. I don't know why Martin favors wchar_t when possible. The answer to that isn't clear. The answer to why there's an intractable problem if wchar_t happens to be a signed type > 2 bytes also isn't clear. > The Unicode implementation has always defined Py_UNICODE to > be an unsigned type; see the Unicode PEP 100: > > """ > Internal Format > > The internal format for Unicode objects should use a Python > specific fixed format implemented as 'unsigned > short' (or another unsigned numeric type having 16 bits). Byte > order is platform dependent. > > ... > > The configure script should provide aid in deciding whether > Python can use the native wchar_t type or not (it has to be a > 16-bit unsigned type). > """ > > Python can also deal with UCS4 now, but the concept remains the > same. Well, it doesn't have to be a 16-bit type either, even in a UCS2 build, and we had a long argument about that one before, because a particular Cray system didn't have any 16-bit type and the Unicode code wasn't working there. That got repaired when I rewrote the few bits of code that assumed "exactly 16 bits" to live with the weaker "at least 16 bits". In this iteration, Martin agreed that a signed 16-bit wchar_t can be rejected. The question remaining is what actual problem exists when there's a signed wchar_t exceeding 16 bits. Since Jeremy is running on exactly such a system, and the tests pass for him, there's no *obvious* problem with it (the segfault he experienced was due to reading uninitialized memory, and that was a bug, and that's been fixed). From jeremy at zope.com Sun Sep 21 11:08:43 2003 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Sep 21 11:10:25 2003 Subject: [Python-Dev] Improving the CGI module In-Reply-To: <11097042-EAEB-11D7-A692-000393C2D67E@colorstudy.com> References: <11097042-EAEB-11D7-A692-000393C2D67E@colorstudy.com> Message-ID: <1064156922.2074.37.camel@localhost.localdomain> Do we have a SIG yet? Nothing wrong with some discussion on python-dev, but we ought to be capturing this stuff in a SIG archive and making it part of the Plan. Jeremy From jeremy at alum.mit.edu Sun Sep 21 11:13:26 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Sun Sep 21 11:13:44 2003 Subject: [Python-Dev] Fun with 2.3 shutdown In-Reply-To: <200309190537.h8J5bnIG015051@localhost.localdomain> References: <200309190537.h8J5bnIG015051@localhost.localdomain> Message-ID: <1064157206.2074.39.camel@localhost.localdomain> On Fri, 2003-09-19 at 01:37, Anthony Baxter wrote: > Hm. Is this serious enough a problem to hold off 2.3.1 until it's fixed? > My feeling is "no", because it only affects Z3 developers, and they're > hardened to random pain by now, and it's also just a shutdown glitch. I don't think it's worth waiting, because we don't have a plan for fixing it. Even if we did, it's arguable whether it belongs in a bug fix release -- at least not without extensive developer testing with the CVS head. Jeremy From tim.one at comcast.net Sun Sep 21 11:34:29 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Sep 21 11:34:34 2003 Subject: [Python-Dev] Py2.3.1 In-Reply-To: <005901c3801b$877d1620$e841fea9@oemcomputer> Message-ID: [Raymond Hettinger] > All tests are passing on Windows except for test_bsddb3 > which fails for both 2.3.1 and 2.4. Did they also fail for you in 2.3? They failed for me in 2.3, but only under Win98SE, and IIRC you're running on the similar WinME. The bsddb3 tests passed for me on Win2K in 2.3. > I've sent Greg a note with the test output to see what he can make of > it. I'll attach what I see on Win98SE under current CVS Python (HEAD). Looks a lot like the bsddb3 failures I saw on Win98SE under 2.3, several DBAgainError: (11, 'Resource temporarily unavailable -- unable to join the environment') complaints. -------------- next part -------------- Capture file written by TCAP 3.1, TCAP Copyright(c) 1999 Tim Kannel C:\Code\python\PCbuild>python ../lib/test/regrtest.py -vv -u bsddb test_bsddb3 test_bsddb3 test01_associateWithDB (bsddb.test.test_associate.AssociateHashTestCase) ... ok test02_associateAfterDB (bsddb.test.test_associate.AssociateHashTestCase) ... ok test01_associateWithDB (bsddb.test.test_associate.AssociateBTreeTestCase) ... ok test02_associateAfterDB (bsddb.test.test_associate.AssociateBTreeTestCase) ... ok test01_associateWithDB (bsddb.test.test_associate.AssociateRecnoTestCase) ... ok test02_associateAfterDB (bsddb.test.test_associate.AssociateRecnoTestCase) ... ok test01_associateWithDB (bsddb.test.test_associate.ShelveAssociateHashTestCase) ... ok test02_associateAfterDB (bsddb.test.test_associate.ShelveAssociateHashTestCase) ... ok test01_associateWithDB (bsddb.test.test_associate.ShelveAssociateBTreeTestCase) ... ok test02_associateAfterDB (bsddb.test.test_associate.ShelveAssociateBTreeTestCase) ... ok test01_associateWithDB (bsddb.test.test_associate.ShelveAssociateRecnoTestCase) ... ok test02_associateAfterDB (bsddb.test.test_associate.ShelveAssociateRecnoTestCase) ... ok test01_associateWithDB (bsddb.test.test_associate.ThreadedAssociateHashTestCase) ... ok test02_associateAfterDB (bsddb.test.test_associate.ThreadedAssociateHashTestCase) ... ok test01_associateWithDB (bsddb.test.test_associate.ThreadedAssociateBTreeTestCase) ... ok test02_associateAfterDB (bsddb.test.test_associate.ThreadedAssociateBTreeTestCase) ... ok test01_associateWithDB (bsddb.test.test_associate.ThreadedAssociateRecnoTestCase) ... ok test02_associateAfterDB (bsddb.test.test_associate.ThreadedAssociateRecnoTestCase) ... ok test00_version (bsddb.test.test_basics.VersionTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.BasicBTreeTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.BasicBTreeTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.BasicBTreeTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.BasicBTreeTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.BasicBTreeTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.BasicBTreeTestCase) ... ok test05_GetSize (bsddb.test.test_basics.BasicBTreeTestCase) ... ok test06_Truncate (bsddb.test.test_basics.BasicBTreeTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.BasicHashTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.BasicHashTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.BasicHashTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.BasicHashTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.BasicHashTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.BasicHashTestCase) ... ok test05_GetSize (bsddb.test.test_basics.BasicHashTestCase) ... ok test06_Truncate (bsddb.test.test_basics.BasicHashTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.BasicBTreeWithThreadFlagTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.BasicBTreeWithThreadFlagTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.BasicBTreeWithThreadFlagTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.BasicBTreeWithThreadFlagTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.BasicBTreeWithThreadFlagTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.BasicBTreeWithThreadFlagTestCase) ... ok test05_GetSize (bsddb.test.test_basics.BasicBTreeWithThreadFlagTestCase) ... ok test06_Truncate (bsddb.test.test_basics.BasicBTreeWithThreadFlagTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.BasicHashWithThreadFlagTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.BasicHashWithThreadFlagTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.BasicHashWithThreadFlagTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.BasicHashWithThreadFlagTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.BasicHashWithThreadFlagTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.BasicHashWithThreadFlagTestCase) ... ok test05_GetSize (bsddb.test.test_basics.BasicHashWithThreadFlagTestCase) ... ok test06_Truncate (bsddb.test.test_basics.BasicHashWithThreadFlagTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.BasicBTreeWithEnvTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.BasicBTreeWithEnvTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.BasicBTreeWithEnvTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.BasicBTreeWithEnvTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.BasicBTreeWithEnvTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.BasicBTreeWithEnvTestCase) ... ok test05_GetSize (bsddb.test.test_basics.BasicBTreeWithEnvTestCase) ... ok test06_Truncate (bsddb.test.test_basics.BasicBTreeWithEnvTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.BasicHashWithEnvTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.BasicHashWithEnvTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.BasicHashWithEnvTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.BasicHashWithEnvTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.BasicHashWithEnvTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.BasicHashWithEnvTestCase) ... ok test05_GetSize (bsddb.test.test_basics.BasicHashWithEnvTestCase) ... ok test06_Truncate (bsddb.test.test_basics.BasicHashWithEnvTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.BTreeTransactionTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.BTreeTransactionTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.BTreeTransactionTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.BTreeTransactionTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.BTreeTransactionTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.BTreeTransactionTestCase) ... ok test05_GetSize (bsddb.test.test_basics.BTreeTransactionTestCase) ... ok test06_Transactions (bsddb.test.test_basics.BTreeTransactionTestCase) ... ok test06_Truncate (bsddb.test.test_basics.BTreeTransactionTestCase) ... ok test07_TxnTruncate (bsddb.test.test_basics.BTreeTransactionTestCase) ... ok test08_TxnLateUse (bsddb.test.test_basics.BTreeTransactionTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.HashTransactionTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.HashTransactionTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.HashTransactionTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.HashTransactionTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.HashTransactionTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.HashTransactionTestCase) ... ok test05_GetSize (bsddb.test.test_basics.HashTransactionTestCase) ... ok test06_Transactions (bsddb.test.test_basics.HashTransactionTestCase) ... ok test06_Truncate (bsddb.test.test_basics.HashTransactionTestCase) ... ok test07_TxnTruncate (bsddb.test.test_basics.HashTransactionTestCase) ... ok test08_TxnLateUse (bsddb.test.test_basics.HashTransactionTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.BTreeRecnoTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.BTreeRecnoTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.BTreeRecnoTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.BTreeRecnoTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.BTreeRecnoTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.BTreeRecnoTestCase) ... ok test05_GetSize (bsddb.test.test_basics.BTreeRecnoTestCase) ... ok test06_Truncate (bsddb.test.test_basics.BTreeRecnoTestCase) ... ok test07_RecnoInBTree (bsddb.test.test_basics.BTreeRecnoTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.BTreeRecnoWithThreadFlagTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.BTreeRecnoWithThreadFlagTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.BTreeRecnoWithThreadFlagTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.BTreeRecnoWithThreadFlagTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.BTreeRecnoWithThreadFlagTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.BTreeRecnoWithThreadFlagTestCase) ... ok test05_GetSize (bsddb.test.test_basics.BTreeRecnoWithThreadFlagTestCase) ... ok test06_Truncate (bsddb.test.test_basics.BTreeRecnoWithThreadFlagTestCase) ... ok test07_RecnoInBTree (bsddb.test.test_basics.BTreeRecnoWithThreadFlagTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.BTreeDUPTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.BTreeDUPTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.BTreeDUPTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.BTreeDUPTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.BTreeDUPTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.BTreeDUPTestCase) ... ok test05_GetSize (bsddb.test.test_basics.BTreeDUPTestCase) ... ok test06_Truncate (bsddb.test.test_basics.BTreeDUPTestCase) ... ok test08_DuplicateKeys (bsddb.test.test_basics.BTreeDUPTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.HashDUPTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.HashDUPTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.HashDUPTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.HashDUPTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.HashDUPTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.HashDUPTestCase) ... ok test05_GetSize (bsddb.test.test_basics.HashDUPTestCase) ... ok test06_Truncate (bsddb.test.test_basics.HashDUPTestCase) ... ok test08_DuplicateKeys (bsddb.test.test_basics.HashDUPTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.BTreeDUPWithThreadTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.BTreeDUPWithThreadTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.BTreeDUPWithThreadTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.BTreeDUPWithThreadTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.BTreeDUPWithThreadTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.BTreeDUPWithThreadTestCase) ... ok test05_GetSize (bsddb.test.test_basics.BTreeDUPWithThreadTestCase) ... ok test06_Truncate (bsddb.test.test_basics.BTreeDUPWithThreadTestCase) ... ok test08_DuplicateKeys (bsddb.test.test_basics.BTreeDUPWithThreadTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.HashDUPWithThreadTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.HashDUPWithThreadTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.HashDUPWithThreadTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.HashDUPWithThreadTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.HashDUPWithThreadTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.HashDUPWithThreadTestCase) ... ok test05_GetSize (bsddb.test.test_basics.HashDUPWithThreadTestCase) ... ok test06_Truncate (bsddb.test.test_basics.HashDUPWithThreadTestCase) ... ok test08_DuplicateKeys (bsddb.test.test_basics.HashDUPWithThreadTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.BTreeMultiDBTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.BTreeMultiDBTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.BTreeMultiDBTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.BTreeMultiDBTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.BTreeMultiDBTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.BTreeMultiDBTestCase) ... ok test05_GetSize (bsddb.test.test_basics.BTreeMultiDBTestCase) ... ok test06_Truncate (bsddb.test.test_basics.BTreeMultiDBTestCase) ... ok test09_MultiDB (bsddb.test.test_basics.BTreeMultiDBTestCase) ... ok test01_GetsAndPuts (bsddb.test.test_basics.HashMultiDBTestCase) ... ok test02_DictionaryMethods (bsddb.test.test_basics.HashMultiDBTestCase) ... ok test03_SimpleCursorStuff (bsddb.test.test_basics.HashMultiDBTestCase) ... ok test03b_SimpleCursorWithoutGetReturnsNone0 (bsddb.test.test_basics.HashMultiDBTestCase) ... ok test03c_SimpleCursorGetReturnsNone2 (bsddb.test.test_basics.HashMultiDBTestCase) ... ok test04_PartialGetAndPut (bsddb.test.test_basics.HashMultiDBTestCase) ... ok test05_GetSize (bsddb.test.test_basics.HashMultiDBTestCase) ... ok test06_Truncate (bsddb.test.test_basics.HashMultiDBTestCase) ... ok test09_MultiDB (bsddb.test.test_basics.HashMultiDBTestCase) ... ok test01_btopen (bsddb.test.test_compat.CompatibilityTestCase) ... ok test02_hashopen (bsddb.test.test_compat.CompatibilityTestCase) ... ok test03_rnopen (bsddb.test.test_compat.CompatibilityTestCase) ... ok test04_n_flag (bsddb.test.test_compat.CompatibilityTestCase) ... ok test01_both (bsddb.test.test_dbobj.dbobjTestCase) ... ok test02_dbobj_dict_interface (bsddb.test.test_dbobj.dbobjTestCase) ... ok test01_basics (bsddb.test.test_dbshelve.DBShelveTestCase) ... ok test02_cursors (bsddb.test.test_dbshelve.DBShelveTestCase) ... ok test01_basics (bsddb.test.test_dbshelve.BTreeShelveTestCase) ... ok test02_cursors (bsddb.test.test_dbshelve.BTreeShelveTestCase) ... ok test01_basics (bsddb.test.test_dbshelve.HashShelveTestCase) ... ok test02_cursors (bsddb.test.test_dbshelve.HashShelveTestCase) ... ok test01_basics (bsddb.test.test_dbshelve.ThreadBTreeShelveTestCase) ... ok test02_cursors (bsddb.test.test_dbshelve.ThreadBTreeShelveTestCase) ... ok test01_basics (bsddb.test.test_dbshelve.ThreadHashShelveTestCase) ... ok test02_cursors (bsddb.test.test_dbshelve.ThreadHashShelveTestCase) ... ok test01_basics (bsddb.test.test_dbshelve.EnvBTreeShelveTestCase) ... ERROR test02_cursors (bsddb.test.test_dbshelve.EnvBTreeShelveTestCase) ... ok test01_basics (bsddb.test.test_dbshelve.EnvHashShelveTestCase) ... ERROR test02_cursors (bsddb.test.test_dbshelve.EnvHashShelveTestCase) ... ok test01_basics (bsddb.test.test_dbshelve.EnvThreadBTreeShelveTestCase) ... ERROR test02_cursors (bsddb.test.test_dbshelve.EnvThreadBTreeShelveTestCase) ... ok test01_basics (bsddb.test.test_dbshelve.EnvThreadHashShelveTestCase) ... ERROR test02_cursors (bsddb.test.test_dbshelve.EnvThreadHashShelveTestCase) ... ok test01 (bsddb.test.test_dbtables.TableDBTestCase) ... ok test02 (bsddb.test.test_dbtables.TableDBTestCase) ... ok test03 (bsddb.test.test_dbtables.TableDBTestCase) ... ok test04_MultiCondSelect (bsddb.test.test_dbtables.TableDBTestCase) ... ok test_CondObjs (bsddb.test.test_dbtables.TableDBTestCase) ... ok test_CreateOrExtend (bsddb.test.test_dbtables.TableDBTestCase) ... ok test_Delete (bsddb.test.test_dbtables.TableDBTestCase) ... ok test_Modify (bsddb.test.test_dbtables.TableDBTestCase) ... ok test01_close_dbenv_before_db (bsddb.test.test_env_close.DBEnvClosedEarlyCrash) ... ok test02_close_dbenv_delete_db_success (bsddb.test.test_env_close.DBEnvClosedEarlyCrash) ... ok test01_get_returns_none (bsddb.test.test_get_none.GetReturnsNoneTestCase) ... ok test02_get_raises_exception (bsddb.test.test_get_none.GetReturnsNoneTestCase) ... ok test01_join (bsddb.test.test_join.JoinTestCase) ... ok test01_simple (bsddb.test.test_lock.LockingTestCase) ... ok test02_threaded (bsddb.test.test_lock.LockingTestCase) ... ok test03_set_timeout (bsddb.test.test_lock.LockingTestCase) ... ok test01_badpointer (bsddb.test.test_misc.MiscTestCase) ... ok test02_db_home (bsddb.test.test_misc.MiscTestCase) ... ok test01_basic (bsddb.test.test_queue.SimpleQueueTestCase) ... ok test02_basicPost32 (bsddb.test.test_queue.SimpleQueueTestCase) ... ok test01_basic (bsddb.test.test_recno.SimpleRecnoTestCase) ... ok test02_WithSource (bsddb.test.test_recno.SimpleRecnoTestCase) ... ok test03_FixedLength (bsddb.test.test_recno.SimpleRecnoTestCase) ... ok test01_1WriterMultiReaders (bsddb.test.test_thread.BTreeConcurrentDataStore) ... ok test01_1WriterMultiReaders (bsddb.test.test_thread.HashConcurrentDataStore) ... ok test02_SimpleLocks (bsddb.test.test_thread.BTreeSimpleThreaded) ... ok test02_SimpleLocks (bsddb.test.test_thread.HashSimpleThreaded) ... ok test03_ThreadedTransactions (bsddb.test.test_thread.BTreeThreadedTransactions) ... ok test03_ThreadedTransactions (bsddb.test.test_thread.HashThreadedTransactions) ... ok test03_ThreadedTransactions (bsddb.test.test_thread.BTreeThreadedNoWaitTransactions) ... ok test03_ThreadedTransactions (bsddb.test.test_thread.HashThreadedNoWaitTransactions) ... ok ====================================================================== ERROR: test01_basics (bsddb.test.test_dbshelve.EnvBTreeShelveTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\CODE\PYTHON\lib\bsddb\test\test_dbshelve.py", line 75, in test01_basics self.do_open() File "C:\CODE\PYTHON\lib\bsddb\test\test_dbshelve.py", line 238, in do_open self.env.open(homeDir, self.envflags | db.DB_INIT_MPOOL | db.DB_CREATE) DBAgainError: (11, 'Resource temporarily unavailable -- unable to join the environment') ====================================================================== ERROR: test01_basics (bsddb.test.test_dbshelve.EnvHashShelveTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\CODE\PYTHON\lib\bsddb\test\test_dbshelve.py", line 75, in test01_basics self.do_open() File "C:\CODE\PYTHON\lib\bsddb\test\test_dbshelve.py", line 238, in do_open self.env.open(homeDir, self.envflags | db.DB_INIT_MPOOL | db.DB_CREATE) DBAgainError: (11, 'Resource temporarily unavailable -- unable to join the environment') ====================================================================== ERROR: test01_basics (bsddb.test.test_dbshelve.EnvThreadBTreeShelveTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\CODE\PYTHON\lib\bsddb\test\test_dbshelve.py", line 75, in test01_basics self.do_open() File "C:\CODE\PYTHON\lib\bsddb\test\test_dbshelve.py", line 238, in do_open self.env.open(homeDir, self.envflags | db.DB_INIT_MPOOL | db.DB_CREATE) DBAgainError: (11, 'Resource temporarily unavailable -- unable to join the environment') ====================================================================== ERROR: test01_basics (bsddb.test.test_dbshelve.EnvThreadHashShelveTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\CODE\PYTHON\lib\bsddb\test\test_dbshelve.py", line 75, in test01_basics self.do_open() File "C:\CODE\PYTHON\lib\bsddb\test\test_dbshelve.py", line 238, in do_open self.env.open(homeDir, self.envflags | db.DB_INIT_MPOOL | db.DB_CREATE) DBAgainError: (11, 'Resource temporarily unavailable -- unable to join the environment') ---------------------------------------------------------------------- Ran 216 tests in 219.100s FAILED (errors=4) test test_bsddb3 failed -- errors occurred; run in verbose mode for details 1 test failed: test_bsddb3 C:\Code\python\PCbuild>tcap/u From martin at v.loewis.de Sun Sep 21 13:54:18 2003 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Sep 21 13:54:31 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: References: Message-ID: <3F6DE5CA.7080108@v.loewis.de> Tim Peters wrote: > I don't know why Martin favors wchar_t when possible. The answer to that > isn't clear. If the wchar_t is "usable", some routines (notably PU_AsWideChar) are slightly more efficient, so I'd like to make wchar_t "usable" as much as possible. Regards, Martin From aleaxit at yahoo.com Sun Sep 21 14:42:45 2003 From: aleaxit at yahoo.com (Alex Martelli) Date: Sun Sep 21 14:42:51 2003 Subject: [Python-Dev] Discussion on adding rsplit() for strings and unicode objects. In-Reply-To: <200309190612.15965.fincher.8@osu.edu> References: <20030919083346.GA11541@tummy.com> <200309190612.15965.fincher.8@osu.edu> Message-ID: <200309212042.45767.aleaxit@yahoo.com> On Friday 19 September 2003 12:12 pm, Jeremy Fincher wrote: > On Friday 19 September 2003 04:33 am, Sean Reifschneider wrote: > > A few weeks ago I submitted a patch to add an rsplit() method to > > string and unicode objects: > > >Adding rsplit() to string and unicode objects. (2003-09-06) > > > http://python.org/sf/801847 > > I'm all for it. I've had to implement rsplit on my own as well, and always Seconded (or thirded, whatever). Actually had to do it just 3 days ago -- having a need for rsplit I in fact *wrote* my code as if it existed (didn't remember it wasn't there!!!), and it took a while to run tests and get the AttributeError, slap my forehead, and add a "def rsplit" function. I think it should be there by "principle of least surprise" -- experienced Python coders *expect* it to be there (and I offer myself as an example;-). > laden with. As long as they're named appropriately, I can't see the harm > in adding useful methods, especially one such as rsplit that completes a > set of front-to-back/back-to-front methods. Absolutely, but limited to those methods that "should" be there, for completeness, and because they're expected. rsplit is the outstanding one that I can think of right now (because I just happened to need it so recently, of course, and the coincidence of it being discussed now). Alex From barry at python.org Sun Sep 21 15:24:02 2003 From: barry at python.org (Barry Warsaw) Date: Sun Sep 21 15:24:08 2003 Subject: [Python-Dev] Enhancement to pdb in gud.el In-Reply-To: <16233.62352.331254.344092@nick.uklinux.net> References: <16233.62352.331254.344092@nick.uklinux.net> Message-ID: <1064172242.4618.7.camel@anthem> On Thu, 2003-09-18 at 14:04, Nick Roberts wrote: > I've written a mode (gdb-ui.el) which is in the Emacs CVS repository > (http://savannah.gnu.org/projects/emacs) that displays breakpoint icons in the > display margin of Emacs when debugging with GDB. I can almost do the same > thing for pdb (the Emacs interface to the python debugger in gud.el). > Anyone interested? Yes, but you might want to post to python-mode@python.org which is now a full-fledged mailing list and is where all the python-moders tend to hang out. -Barry From tim.one at comcast.net Sun Sep 21 19:02:24 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Sep 21 19:02:29 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: <3F6DE5CA.7080108@v.loewis.de> Message-ID: [Tim] >> I don't know why Martin favors wchar_t when possible. The answer to >> that isn't clear. [Martin v. L?wis] > If the wchar_t is "usable", some routines (notably PU_AsWideChar) are > slightly more efficient, so I'd like to make wchar_t "usable" as much > as possible. OK. So is there an end to this thread <0.9 wink>? At the moment, it appears there's no identified reason to care about signedness of a greater-than 16-bit type, good reason to insist that a 16-bit type is unsigned, and that it's desirable for HAVE_USABLE_WCHAR_T to get defined when possible. What more does it take to bury this? If it's Unixish config chagnes, they won't be coming from me (the Windows build uses an unsigned 16-bit wchar_t). From tim at zope.com Sun Sep 21 19:50:42 2003 From: tim at zope.com (Tim Peters) Date: Sun Sep 21 19:51:19 2003 Subject: [Python-Dev] Fun with 2.3 shutdown In-Reply-To: <20030919120100.GB23317@vicky.ecs.soton.ac.uk> Message-ID: [Armin Rigo] > The behavior of Python regarding whether threads should continue to > run after the main one exited has never been too clear. Wouldn't it > make sense to try to control this aspect more precisely ? threading.py already tried to. Whether or not a native platform thread outlives the main thread varies across platforms. A Thread created via threading.py prevents the main thread from exiting on all platforms so long as the Thread is running, unless the user calls its setDaemon(True) method before starting the thread. Whether a daemon Thread can outlive the main thread remains platform-dependent. These platform dependencies exist because there's no cross-platform way for Python to forcibly end a thread's life from outside the thread, or to force a thread to continue running after the main thread exits. Python *can* wait for a Thread to end on its own in a cross-platform way, and threading.py keeps the main thread alive until all non-daemon Threads have stopped running via that portable method. So it does what it reasonably *could* do. > I have also experienced this kind of shutdown glitches. You mean the uselessly empty "Exception in thread" messages at shutdown? I've only seen those when running Zope3 tests so far (I count *non*-empty "Exception in thread" shutdown msgs as a different glitch, though). I expect every long-time Python programmer has seen the ever-popular "object 'None' has no attribute so-and-so" kinds of shutdown glitches, and we fix more of those every release by purging __del__ methods, and methods invoked by __del__ methods, of references to module globals. The segfaults at shutdown I saw later while trying to debug the Zope3 symptom were new to me. > It definitely needs more thinking, but what about this one: Python > would only shutdown after all threads are finished; In the Zope3 case, there are about a dozen Threads still running, and they'll never finish. Nine are in an Event.wait() that will never trigger; the rest are in an unbounded polling loop with a sleep(3) each time around. They're all daemon Threads, though, so asked Python explicitly not to wait for them to finish. > but exiting the main thread would by default send a SystemExit > exception to all current threads, if that can be done cleanly (oh > well, this problem again). I'm afraid that would be a major change in shutdown semantics for non-daemon Threads, yes? Right now the main thread will happily wait as long as it takes for Threads to decide to stop on their own, and I've seen programs rely on that (fire up a bunch of producer and consumer Threads, then let the main thread fall off the end of the script). It might help the Zope3 problem, though, if SystemExit got raised in daemon Threads when the main thread wanted to exit. But stuffing a pending exception on a Thread's todo list is asynchronous, and in the case of the Zope3 Threads sitting in their sleep(3) polling loops, the SystemExit won't be noticed until the sleep expires. Since 3 might actually be bigger than 3 in some apps , there's really no limit on how long the main thread might have to wait to be sure all the daemon Threads noticed their SystemExit. > Another question, what was the rationale for setting module globals > to None instead of simply deleting them (and getting the expected > AttributeErrors both at C and at Python level) ? I suspect it's because there's no "efficient" way to iterate over a dict and simultaneously mutate it in a size-changing way; the C PyDict_Next() protocol allows the caller to change the value associated with existing keys (like setting them to None), but is unpredictable if the caller adds or deletes keys during PyDict_Next() iteration. Well, due to quirks of the current implementation, I suspect it actually could delete all the keys without harm (the relevant quirk is that the current implementation never resizes a dict in response to a key deletion; it can resize only when a new key gets added; resizing can shuffle the keys around in a new order, which is what makes PyDict_Next unpredictable if the dict does get resized during iteration). From jeremy at alum.mit.edu Sun Sep 21 22:45:20 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Sun Sep 21 22:45:40 2003 Subject: [Python-Dev] latest bsddb3 test problems Message-ID: <1064198720.2074.54.camel@localhost.localdomain> With current cvs on the release23-maint branch, I get a bunch of tracebacks printed to the console without having the test fail. I haven't looked at the test at all, so I don't know what any of these messages mean. It seems, though, that the test should fail if a thread fails with an exception. Jeremy test_bsddb3 Exception in thread reader 4: Traceback (most recent call last): File "/home/jeremy/src/python23/Lib/threading.py", line 436, in __bootstrap self.run() File "/home/jeremy/src/python23/Lib/threading.py", line 416, in run self.__target(*self.__args, **self.__kwargs) File "/home/jeremy/src/python23/Lib/bsddb/test/test_thread.py", line 275, in readerThread rec = dbutils.DeadlockWrap(c.next, max_retries=10) File "/home/jeremy/src/python23/Lib/bsddb/dbutils.py", line 67, in DeadlockWrap return function(*_args, **_kwargs) DBLockDeadlockError: (-30996, 'DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock') Exception in thread reader 3: Traceback (most recent call last): File "/home/jeremy/src/python23/Lib/threading.py", line 436, in __bootstrap self.run() File "/home/jeremy/src/python23/Lib/threading.py", line 416, in run self.__target(*self.__args, **self.__kwargs) File "/home/jeremy/src/python23/Lib/bsddb/test/test_thread.py", line 275, in readerThread rec = dbutils.DeadlockWrap(c.next, max_retries=10) File "/home/jeremy/src/python23/Lib/bsddb/dbutils.py", line 67, in DeadlockWrap return function(*_args, **_kwargs) DBLockDeadlockError: (-30996, 'DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock') Exception in thread reader 1: Traceback (most recent call last): File "/home/jeremy/src/python23/Lib/threading.py", line 436, in __bootstrap self.run() File "/home/jeremy/src/python23/Lib/threading.py", line 416, in run self.__target(*self.__args, **self.__kwargs) File "/home/jeremy/src/python23/Lib/bsddb/test/test_thread.py", line 275, in readerThread rec = dbutils.DeadlockWrap(c.next, max_retries=10) File "/home/jeremy/src/python23/Lib/bsddb/dbutils.py", line 67, in DeadlockWrap return function(*_args, **_kwargs) DBLockDeadlockError: (-30996, 'DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock') Exception in thread writer 1: Traceback (most recent call last): File "/home/jeremy/src/python23/Lib/threading.py", line 436, in __bootstrap self.run() File "/home/jeremy/src/python23/Lib/threading.py", line 416, in run self.__target(*self.__args, **self.__kwargs) File "/home/jeremy/src/python23/Lib/bsddb/test/test_thread.py", line 239, in writerThread self.assertEqual(data, self.makeData(key)) File "/home/jeremy/src/python23/Lib/unittest.py", line 302, in failUnlessEqual raise self.failureException, \ AssertionError: None != '1063-1063-1063-1063-1063' Exception in thread writer 2: Traceback (most recent call last): File "/home/jeremy/src/python23/Lib/threading.py", line 436, in __bootstrap self.run() File "/home/jeremy/src/python23/Lib/threading.py", line 416, in run self.__target(*self.__args, **self.__kwargs) File "/home/jeremy/src/python23/Lib/bsddb/test/test_thread.py", line 254, in writerThread self.assertEqual(data, self.makeData(key)) File "/home/jeremy/src/python23/Lib/unittest.py", line 302, in failUnlessEqual raise self.failureException, \ AssertionError: None != '2000-2000-2000-2000-2000' Exception in thread writer 0: Traceback (most recent call last): File "/home/jeremy/src/python23/Lib/threading.py", line 436, in __bootstrap self.run() File "/home/jeremy/src/python23/Lib/threading.py", line 416, in run self.__target(*self.__args, **self.__kwargs) File "/home/jeremy/src/python23/Lib/bsddb/test/test_thread.py", line 254, in writerThread self.assertEqual(data, self.makeData(key)) File "/home/jeremy/src/python23/Lib/unittest.py", line 302, in failUnlessEqual raise self.failureException, \ AssertionError: None != '0002-0002-0002-0002-0002' From tim.one at comcast.net Sun Sep 21 23:21:04 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Sep 21 23:21:10 2003 Subject: [Python-Dev] latest bsddb3 test problems In-Reply-To: <1064198720.2074.54.camel@localhost.localdomain> Message-ID: [python-dev-bounces@python.org] > With current cvs on the release23-maint branch, I get a bunch of > tracebacks printed to the console without having the test fail. > I haven't looked at the test at all, so I don't know what any of these > messages mean. It seems, though, that the test should fail if a > thread fails with an exception. I've been told that: DBLockDeadlockError: (-30996, 'DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock') isn't an error, it's just something that's going to happen from time to time. You'll find DeadlockWrap in the traceback then, and the tests typically do something like dbutils.DeadlockWrap(d.put, key, self.makeData(key), max_retries=12) But the DeadlockWrap implementation suppresses DBLockDeadlockError until max_retries attempts have been made, so I don't think the tests *expect* ever to see this get raised. It feels like the moral equivalent of our ConcurrentUpdates ZEO tests that way. WRT the rest, we've seen errors like > AssertionError: None != '1063-1063-1063-1063-1063' > AssertionError: None != '2000-2000-2000-2000-2000' > AssertionError: None != '0002-0002-0002-0002-0002' coming out of Lib/bsddb/test/test_thread.py before, on Windows and on Linux, but rarely. IIRC, no progress was made in tracking these down, and Barry and I were dissuaded from pursuing it last time around. These tests have multiple threads writing to a database, and the exceptions are due to not finding the data in the database the threads thought they wrote. In that respect it's also reminiscent of earlier ConcurrentUpdates ZEO tests. However, these are clearly real test failures, and it's not good that unittest thinks these tests pass despite that the threads barf. From anthony at interlink.com.au Sun Sep 21 23:40:51 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Sun Sep 21 23:42:42 2003 Subject: [Python-Dev] latest bsddb3 test problems In-Reply-To: <1064198720.2074.54.camel@localhost.localdomain> Message-ID: <200309220340.h8M3eqju022753@localhost.localdomain> >>> Jeremy Hylton wrote > With current cvs on the release23-maint branch, I get a bunch of > tracebacks printed to the console without having the test fail. > I haven't looked at the test at all, so I don't know what any of these > messages mean. It seems, though, that the test should fail if a thread > fails with an exception. Argh. What platform/compiler/bsddb version? -- Anthony Baxter It's never too late to have a happy childhood. From jeremy at alum.mit.edu Mon Sep 22 00:05:17 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Mon Sep 22 00:05:38 2003 Subject: [Python-Dev] latest bsddb3 test problems In-Reply-To: <200309220340.h8M3eqju022753@localhost.localdomain> References: <200309220340.h8M3eqju022753@localhost.localdomain> Message-ID: <1064203516.2074.102.camel@localhost.localdomain> On Sun, 2003-09-21 at 23:40, Anthony Baxter wrote: > >>> Jeremy Hylton wrote > > With current cvs on the release23-maint branch, I get a bunch of > > tracebacks printed to the console without having the test fail. > > I haven't looked at the test at all, so I don't know what any of these > > messages mean. It seems, though, that the test should fail if a thread > > fails with an exception. > > Argh. What platform/compiler/bsddb version? Python 2.3+ (#3, Sep 21 2003, 13:23:43) [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import bsddb >>> bsddb._bsddb.version() (4, 1, 25) RH 9. Jeremy From 2tiapuzxx at msn.com Mon Sep 22 10:18:56 2003 From: 2tiapuzxx at msn.com (Phil Travis) Date: Mon Sep 22 00:17:30 2003 Subject: [Python-Dev] Pheromone sex attractant f qnzncjejaj nji Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20030922/7c413099/attachment-0001.html From python at rcn.com Mon Sep 22 00:52:42 2003 From: python at rcn.com (Raymond Hettinger) Date: Mon Sep 22 00:53:01 2003 Subject: [Python-Dev] latest bsddb3 test problems References: <1064198720.2074.54.camel@localhost.localdomain> Message-ID: <00d201c380c5$5b6a1f60$0bb9958d@oemcomputer> [Jeremy] > With current cvs on the release23-maint branch, I get a bunch of > tracebacks printed to the console without having the test fail. > I haven't looked at the test at all, so I don't know what any of these > messages mean. It seems, though, that the test should fail if a thread > fails with an exception. I do get a test failure with the following diagnostic: ====================================================================== ERROR: test01_basics (bsddb.test.test_dbshelve.EnvThreadHashShelveTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\PY23\lib\bsddb\test\test_dbshelve.py", line 75, in test01_basics self.do_open() File "C:\PY23\lib\bsddb\test\test_dbshelve.py", line 238, in do_open self.env.open(homeDir, self.envflags | db.DB_INIT_MPOOL | db.DB_CREATE) DBAgainError: (11, 'Resource temporarily unavailable -- unable to join the environment') Raymond Hettinger ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From anthony at interlink.com.au Mon Sep 22 03:04:23 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Sep 22 03:06:06 2003 Subject: [Python-Dev] latest bsddb3 test problems In-Reply-To: <1064198720.2074.54.camel@localhost.localdomain> Message-ID: <200309220704.h8M74NNk026354@localhost.localdomain> I'm seeing this fail as well, on Redhat 9. Backing out Greg's changes over the weekend doesn't help... the tests still mark as "passed", even though there's the fairly horrible error/warnings. I'm guessing something that raises an error in a non-main thread doesn't get picked up by regrtest.py test_bsddb3 is failing on pretty much every box I've tried it on here - Solaris and Redhat linux, most running the latest-n-greatest release of DB4.1 (we use it pretty heavily in a variety of internal applications). Solaris 8 (DB 4.1.25. gcc 2.96): DBAgainError: (11, 'Resource temporarily unavailable -- mmap: Resource temporarily unavailable') Solaris 7 (DB 4.1.25, gcc 2.95.2) Crashes with a SIGBUS. The version of _bsddb.c from a week ago has a lot of errors, but doesn't crash. Redhat 10beta/Rawhide (DB 4.1.25) DBNoMemoryError: (12, 'Cannot allocate memory -- Lock table is out of available locks') alternating with SIGBUS Redhat 7.1,7.2,7.3 (DB 3.2.9) Crashes with a SIGBUS. I'm going to check with Spambayes as an actual application and see whether the tests that are failing are at fault, or if we've got something that's just busted. Can anyone report the bsddb3 tests as passing on their system? Note that they're not run by default. I'm at a loss as to whether to delay the release or not... Anthony -- Anthony Baxter It's never too late to have a happy childhood. From greg at electricrain.com Mon Sep 22 03:50:32 2003 From: greg at electricrain.com (Gregory P. Smith) Date: Mon Sep 22 03:50:36 2003 Subject: [Python-Dev] latest bsddb3 test problems In-Reply-To: <200309220704.h8M74NNk026354@localhost.localdomain> References: <1064198720.2074.54.camel@localhost.localdomain> <200309220704.h8M74NNk026354@localhost.localdomain> Message-ID: <20030922075032.GC16574@zot.electricrain.com> > I'm seeing this fail as well, on Redhat 9. Backing out Greg's changes over > the weekend doesn't help... the tests still mark as "passed", even though > there's the fairly horrible error/warnings. I'm guessing something that > raises an error in a non-main thread doesn't get picked up by regrtest.py > > test_bsddb3 is failing on pretty much every box I've tried it on here - > Solaris and Redhat linux, most running the latest-n-greatest release of > DB4.1 (we use it pretty heavily in a variety of internal applications). > > Solaris 8 (DB 4.1.25. gcc 2.96): > DBAgainError: (11, 'Resource temporarily unavailable -- mmap: Resource temporarily unavailable') > > Solaris 7 (DB 4.1.25, gcc 2.95.2) > Crashes with a SIGBUS. The version of _bsddb.c from a week ago has a lot of > errors, but doesn't crash. That's sounds like inconsistent behaviour. Look at the diffs. All of the changes since the python 2.3 release to _bsddb.c have been trivial. The _bsddb.c file version is not the culprit. > Redhat 10beta/Rawhide (DB 4.1.25) > DBNoMemoryError: (12, 'Cannot allocate memory -- Lock table is out of available locks') > alternating with SIGBUS > > Redhat 7.1,7.2,7.3 (DB 3.2.9) > Crashes with a SIGBUS. > > > I'm going to check with Spambayes as an actual application and see whether > the tests that are failing are at fault, or if we've got something that's > just busted. Can anyone report the bsddb3 tests as passing on their system? > Note that they're not run by default. It is passing on the systems that i've tested it on in the past few days: (Using BerkeleyDB 4.0, 4.1 and a prerelease of 4.2). Linux/i686 Gentoo 1.4 Linux/alpha Debian "testing" I do see the occasional DBDeadLockError which can be ignored. (Yes, I agree, the test suite needs work to get rid of things like that. if its not going to consider it an error it should at least catch with a comment stating why to avoid the confusion) NOTE: A weakness of the bsddb tests that I have witnessed causing problems: It often doesn't wipe its temporary directory before the tests (in theory it should clean up on shutdown, but thats only if all goes well in the tearDown methods). If a previous run was aborted abnormally and left a database environment & db around it can cause problems during future test runs if the testsuite trys to use that environment and db. > I'm at a loss as to whether to delay the release or not... > > Anthony Do the same tests fail similarly when run on python 2.3? I'd imagine so. IMHO, I wouldn't hold python 2.3.1 up due to bsddb test suite issues. -greg From anthony at interlink.com.au Mon Sep 22 04:11:31 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Sep 22 04:13:29 2003 Subject: [Python-Dev] latest bsddb3 test problems In-Reply-To: <20030922075032.GC16574@zot.electricrain.com> Message-ID: <200309220811.h8M8BVEN027031@localhost.localdomain> >>> "Gregory P. Smith" wrote > That's sounds like inconsistent behaviour. Look at the diffs. All of > the changes since the python 2.3 release to _bsddb.c have been trivial. > The _bsddb.c file version is not the culprit. Yeah. Trying it repeatedly shows that it crashes, sometimes, on the old or new code. > It is passing on the systems that i've tested it on in the past few days: > (Using BerkeleyDB 4.0, 4.1 and a prerelease of 4.2). > > Linux/i686 Gentoo 1.4 > Linux/alpha Debian "testing" The only debian box I can get to easily is the sourceforge compile farm one, and some charming individual appears to have fork-bombed it into uselessness. > I do see the occasional DBDeadLockError which can be ignored. (Yes, > I agree, the test suite needs work to get rid of things like that. > if its not going to consider it an error it should at least catch with > a comment stating why to avoid the confusion) That would be good. Note that I'm not just seeing DBDeadlockErrors, but a whole swag of different errors on the different machines. > NOTE: A weakness of the bsddb tests that I have witnessed causing problems: > > It often doesn't wipe its temporary directory before the tests (in theory > it should clean up on shutdown, but thats only if all goes well in the > tearDown methods). If a previous run was aborted abnormally and left > a database environment & db around it can cause problems during future > test runs if the testsuite trys to use that environment and db. That shouldn't be a problem here, as I'm running the build and test processes from different directories for each system. > Do the same tests fail similarly when run on python 2.3? I'd imagine so. > IMHO, I wouldn't hold python 2.3.1 up due to bsddb test suite issues. Going to try that next. -- Anthony Baxter It's never too late to have a happy childhood. From mal at lemburg.com Mon Sep 22 05:21:48 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Sep 22 05:21:52 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: References: Message-ID: <3F6EBF2C.3040108@lemburg.com> Tim Peters wrote: > [Tim] > >>>I don't know why Martin favors wchar_t when possible. The answer to >>>that isn't clear. > > [Martin v. L?wis] > >>If the wchar_t is "usable", some routines (notably PU_AsWideChar) are >>slightly more efficient, so I'd like to make wchar_t "usable" as much >>as possible. > > OK. So is there an end to this thread <0.9 wink>? At the moment, it > appears there's no identified reason to care about signedness of a > greater-than 16-bit type, Sure there is: first of all, having a single type that can be signed on some platforms and unsigned on others is a bad thing per se and second the 32-bit signed wchar_t value was what triggered this thread in the first place. > good reason to insist that a 16-bit type is > unsigned, and that it's desirable for HAVE_USABLE_WCHAR_T to get defined > when possible. What more does it take to bury this? If it's Unixish config > chagnes, they won't be coming from me (the Windows build uses an unsigned > 16-bit wchar_t). That's what it takes, right. I'll work on it. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Sep 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mwh at python.net Mon Sep 22 06:18:21 2003 From: mwh at python.net (Michael Hudson) Date: Mon Sep 22 06:17:45 2003 Subject: [Python-Dev] Py2.3.1 In-Reply-To: <200309201616.h8KGGaf5021370@localhost.localdomain> (Anthony Baxter's message of "Sun, 21 Sep 2003 02:16:35 +1000") References: <200309201616.h8KGGaf5021370@localhost.localdomain> Message-ID: <2misnl6s9e.fsf@starship.python.net> Anthony Baxter writes: > I'm planning on starting the dance of the seven release tarballs Tuesday > morning AU time, which is more or less 0:00 Tuesday UTC. Assuming that > no-one objects to this timeframe, could people be _extremely_ conservative > with checkins on the branch for the 24 hours before this? I'm going to > be starting to cut test release tarballs probably some time tomorrow and > building it on as many platforms as I can ssh to easily - I can make this > available to other people if they want to help test, or if they've got > some sort of strange version of Unix (Irix and HP/UX are always good > ones to check for platform oddities). Um. Has someone done the 'reading through cvs logs' ritual for 2.3.1? It's tedious, but valuable. In particular, I'm not *certain* all the results of the refleak fixing spree got backported, and some of them should probably acquire NEWS entries. I'm not sure I want to promise being able to do this in the next twelve hours... (If anyone wants to review the _hotshot.c patch attached to bug #808756, that'd be grand, too). Cheers, mwh -- On the other hand, the following areas are subject to boycott in reaction to the rampant impurity of design or execution, as determined after a period of study, in no particular order: ... http://www.naggum.no/profile.html From mal at lemburg.com Mon Sep 22 07:18:32 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Sep 22 07:18:38 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: <3F6EBF2C.3040108@lemburg.com> References: <3F6EBF2C.3040108@lemburg.com> Message-ID: <3F6EDA88.7050407@lemburg.com> M.-A. Lemburg wrote: >> good reason to insist that a 16-bit type is >> unsigned, and that it's desirable for HAVE_USABLE_WCHAR_T to get defined >> when possible. What more does it take to bury this? If it's Unixish >> config chagnes, they won't be coming from me (the Windows build uses an >> unsigned 16-bit wchar_t). > > That's what it takes, right. I'll work on it. While working on the config changes, I noticed that Python now defaults to UCS4 when it find a Tcl/Tk version that supports UCS4... I can't say that I particularly like this, since the config script now makes an implicit choice based on the third-party software configuration with consequences that are not made obvious for the user. E.g. on platforms that happen to have Tcl/tk installed with UCS4 configuration, Python will compile using UCS4 (regardeless of whether the user wants to use Tcl/Tk or not), on system that don't have such Tcl/tk installation, Python compiles using UCS2. I'd suggest to make the UCS4 choice explicit again. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Sep 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mcherm at mcherm.com Mon Sep 22 07:46:52 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Mon Sep 22 07:46:34 2003 Subject: [Python-Dev] Re: Fsrit Propttoye: SCLABMRE Message-ID: <1064231212.3f6ee12c8b2a8@mcherm.com> Tihs drecvoisy souhld be supespserd imelidemtay... thnik waht it wlil do to SymaeapBs! -- Mcaeihl Chdemisre From mwh at python.net Mon Sep 22 07:50:38 2003 From: mwh at python.net (Michael Hudson) Date: Mon Sep 22 07:50:07 2003 Subject: [Python-Dev] pep 310 (reliable acquisition/release pairs) In-Reply-To: <20030919174240.C31102@prim.han.de> (Holger Krekel's message of "Fri, 19 Sep 2003 17:42:40 +0200") References: <20030919174240.C31102@prim.han.de> Message-ID: <2msmmp59f5.fsf@starship.python.net> Holger Krekel writes: > hello, > > admittedly i only followed Brett's very nice summaries (thanks!) > in the last month so i may have missed some details. But here are a few > comments regarding "PEP 310 Reliable Acquisition/Release Pairs". > > The PEP actually is about interacting with the execution > of a code block. It allows to define (one-shot) interception points > for entering and leaving a code block. Now there are at least > two interesting cases which the PEP does (quite explicitely) not cover: > > - what to do with exceptions > > - what to do with yield > > IMHO introducing a new block statement at this stage in language > development warrants an effort to tackle these cases (and maybe more > like e.g. allowing the handler to trigger looping). > > This is probably best done with trying to directly design a protocol between > the "interpreter-loop" and the - what i'd call - the "execution handler". Well, in writing PEP 310 (as I suspect you know) I was aiming for a simple, almost entirely syntactic way of shortening a common pattern. You seem to be gunning for something far deeper here. [...] > So while i am not strictly against the proposal i'd humbly ask for > not hurrying into accepting the PEP as is. Python 2.4 is not closeby > so i hope there is still some time to discuss this. While I agree and am in no hurry to rush, I'm not sure I see that accepting PEP 310 as is necessarily hinders your more subtle aims... Cheers, mwh -- SCSI is not magic. There are fundamental technical reasons why it is necessary to sacrifice a young goat to your SCSI chain now and then. -- John Woods From jepler at unpythonic.net Mon Sep 22 08:04:38 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Mon Sep 22 08:05:00 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: <3F6EDA88.7050407@lemburg.com> References: <3F6EBF2C.3040108@lemburg.com> <3F6EDA88.7050407@lemburg.com> Message-ID: <20030922120433.GB28986@unpythonic.net> On Mon, Sep 22, 2003 at 01:18:32PM +0200, M.-A. Lemburg wrote: > While working on the config changes, I noticed that Python now > defaults to UCS4 when it find a Tcl/Tk version that supports UCS4... > > I can't say that I particularly like this, since the config script > now makes an implicit choice based on the third-party software > configuration with consequences that are not made obvious for the > user. > > E.g. on platforms that happen to have Tcl/tk installed > with UCS4 configuration, Python will compile using UCS4 (regardeless > of whether the user wants to use Tcl/Tk or not), on system > that don't have such Tcl/tk installation, Python compiles using > UCS2. > > I'd suggest to make the UCS4 choice explicit again. See the (complete lack of) discussion on the bug at http://python.org/sf/798202 I can only say that this is a convenient patch for me, and probably for any redhat9 user who has not built his own tcl/tk. On the other hand, this patch is unlikely to actually affect anybody on a different platform. I don't think it's such a bad idea to have Python detect the configuration it must use to make a very important, bundled extension work properly. Would you feel any different if a different patch was added, which made --enable-unicode=tcl enable the behavior of this patch? Jeff From mal at lemburg.com Mon Sep 22 09:22:17 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Sep 22 09:22:23 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: <20030922120433.GB28986@unpythonic.net> References: <3F6EBF2C.3040108@lemburg.com> <3F6EDA88.7050407@lemburg.com> <20030922120433.GB28986@unpythonic.net> Message-ID: <3F6EF789.1070808@lemburg.com> Jeff Epler wrote: > On Mon, Sep 22, 2003 at 01:18:32PM +0200, M.-A. Lemburg wrote: > >>While working on the config changes, I noticed that Python now >>defaults to UCS4 when it find a Tcl/Tk version that supports UCS4... >> >>I can't say that I particularly like this, since the config script >>now makes an implicit choice based on the third-party software >>configuration with consequences that are not made obvious for the >>user. >> >>E.g. on platforms that happen to have Tcl/tk installed >>with UCS4 configuration, Python will compile using UCS4 (regardeless >>of whether the user wants to use Tcl/Tk or not), on system >>that don't have such Tcl/tk installation, Python compiles using >>UCS2. >> >>I'd suggest to make the UCS4 choice explicit again. > > See the (complete lack of) discussion on the bug at > http://python.org/sf/798202 > > I can only say that this is a convenient patch for me, and probably for > any redhat9 user who has not built his own tcl/tk. On the other hand, > this patch is unlikely to actually affect anybody on a different > platform. > > I don't think it's such a bad idea to have Python detect the > configuration it must use to make a very important, bundled extension > work properly. Would you feel any different if a different patch was > added, which made --enable-unicode=tcl enable the behavior of this > patch? Yes :-) ... "explicit is better than implicit" -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Sep 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From barry at python.org Mon Sep 22 09:50:12 2003 From: barry at python.org (Barry Warsaw) Date: Mon Sep 22 09:50:45 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: <20030922120433.GB28986@unpythonic.net> References: <3F6EBF2C.3040108@lemburg.com> <3F6EDA88.7050407@lemburg.com> <20030922120433.GB28986@unpythonic.net> Message-ID: <1064238611.1986.0.camel@anthem> On Mon, 2003-09-22 at 08:04, Jeff Epler wrote: > I don't think it's such a bad idea to have Python detect the > configuration it must use to make a very important, bundled extension > work properly. Would you feel any different if a different patch was > added, which made --enable-unicode=tcl enable the behavior of this > patch? Seems just as easy to document that on RH9 with the default tcl/tk, you need to include --enable-unicode=ucs4. -Barry From anthony at interlink.com.au Mon Sep 22 10:15:42 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Sep 22 10:17:24 2003 Subject: [Python-Dev] Py2.3.1 In-Reply-To: <2misnl6s9e.fsf@starship.python.net> Message-ID: <200309221415.h8MEFgEv031731@localhost.localdomain> >>> Michael Hudson wrote > Um. Has someone done the 'reading through cvs logs' ritual for 2.3.1? > It's tedious, but valuable. In particular, I'm not *certain* all the > results of the refleak fixing spree got backported, and some of them > should probably acquire NEWS entries. I'm not sure I want to promise > being able to do this in the next twelve hours... I did a bit of a scan, and I think most of them are in. I'll have another look tomorrow morning. At this point I'd like to say "please hold off on changes to the release23-maint branch, unless you're involved in the release process." -- Anthony Baxter It's never too late to have a happy childhood. From theller at python.net Mon Sep 22 10:43:30 2003 From: theller at python.net (Thomas Heller) Date: Mon Sep 22 10:43:38 2003 Subject: [Python-Dev] Py2.3.1 In-Reply-To: <200309221415.h8MEFgEv031731@localhost.localdomain> (Anthony Baxter's message of "Tue, 23 Sep 2003 00:15:42 +1000") References: <200309221415.h8MEFgEv031731@localhost.localdomain> Message-ID: <1xu8vq7h.fsf@python.net> Anthony Baxter writes: >>>> Michael Hudson wrote >> Um. Has someone done the 'reading through cvs logs' ritual for 2.3.1? >> It's tedious, but valuable. In particular, I'm not *certain* all the >> results of the refleak fixing spree got backported, and some of them >> should probably acquire NEWS entries. I'm not sure I want to promise >> being able to do this in the next twelve hours... > > I did a bit of a scan, and I think most of them are in. I'll have another > look tomorrow morning. > > At this point I'd like to say "please hold off on changes to the > release23-maint branch, unless you're involved in the release process." I built a windows installer and started testing it under WinXP and win98SE a few hours ago, there haven't been any checkins to the release-23maint branch in the meantime except one to gopherlib. On winXP everything worked (although test_largefile took ages to complete on my notebook), on Win98SE (not the desktop) the testsuite fails in test_winsound (do I need a soundcard for this?) and in test_bsddb3. Thomas From skip at pobox.com Mon Sep 22 12:19:32 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Sep 22 12:19:47 2003 Subject: [Python-Dev] 2.3.1 windows binaries In-Reply-To: References: Message-ID: <16239.8468.552863.416218@montanaro.dyndns.org> Thomas> Is it important to use the same openssl version that Python Thomas> 2.3.0 used, or is the readme file correct in saying that the Thomas> latest version is the one to use? And does this have to be Thomas> mentioned somewhere (Misc/NEWS, the readme file for windows)? I would use the latest version. I believe there were security-related fixes applied OpenSSL in the past month or so (so many security fixes have come out recently, that I'm not sure if it's OpenSSL or OpenSSH -- or both -- which got patched). It's probably worth a mention somewhere which versions you used for all external packages. Skip From barry at barrys-emacs.org Mon Sep 22 12:56:17 2003 From: barry at barrys-emacs.org (Barry Scott) Date: Mon Sep 22 12:57:18 2003 Subject: [Python-Dev] Discussion on adding rsplit() for strings and unicode objects. In-Reply-To: <200309190612.15965.fincher.8@osu.edu> References: <20030919083346.GA11541@tummy.com> <200309190612.15965.fincher.8@osu.edu> Message-ID: <6.0.0.22.0.20030922175257.020ecdd0@torment.chelsea.private> >I'm all for it. I've had to implement rsplit on my own as well, and always >lamented (but never did anything, unfortunately) about its lack in the >standard library. That strings have rfind, rindex, rstrip, and rjust, but >not rsplit always seemed somewhat less than complete to me. These "r" are not a single family... some mean reverse some mean right. BArry From tim.one at comcast.net Mon Sep 22 13:25:33 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Sep 22 13:26:08 2003 Subject: [Python-Dev] Py2.3.1 In-Reply-To: <1xu8vq7h.fsf@python.net> Message-ID: [Thomas Heller] > I built a windows installer and started testing it under WinXP and > win98SE a few hours ago, there haven't been any checkins to the > release-23maint branch in the meantime except one to gopherlib. > > On winXP everything worked (although test_largefile took ages to > complete on my notebook), on Win98SE (not the desktop) the testsuite > fails in test_winsound (do I need a soundcard for this?) and in > test_bsddb3. See my msg yesterday: test_bsddb3 always fails on Win98SE, and that's nothing new (it failed the same ways under 2.3). test_winsound recently started failing on my Win98SE box too. Don't know why, and ran out of time to dig into it yesterday. I seem to recall it failing the same way for Guido a long time ago. Maybe some MS service pack or update changed endcase behavior -- don't know. From hpk at trillke.net Mon Sep 22 13:41:37 2003 From: hpk at trillke.net (Holger Krekel) Date: Mon Sep 22 13:41:45 2003 Subject: [Python-Dev] pep 310 (reliable acquisition/release pairs) In-Reply-To: <2msmmp59f5.fsf@starship.python.net>; from mwh@python.net on Mon, Sep 22, 2003 at 12:50:38PM +0100 References: <20030919174240.C31102@prim.han.de> <2msmmp59f5.fsf@starship.python.net> Message-ID: <20030922194137.C24546@prim.han.de> Michael Hudson wrote: > Holger Krekel writes: > > The PEP actually is about interacting with the execution > > of a code block. It allows to define (one-shot) interception points > > for entering and leaving a code block. Now there are at least > > two interesting cases which the PEP does (quite explicitely) not cover: > > > > - what to do with exceptions > > > > - what to do with yield > > > > IMHO introducing a new block statement at this stage in language > > development warrants an effort to tackle these cases (and maybe more > > like e.g. allowing the handler to trigger looping). > > > > This is probably best done with trying to directly design a protocol between > > the "interpreter-loop" and the - what i'd call - the "execution handler". > > Well, in writing PEP 310 (as I suspect you know) I was aiming for a > simple, almost entirely syntactic way of shortening a common pattern. > You seem to be gunning for something far deeper here. Hmmm, introducing a new block statement just for "shortening a common pattern" doesn't sound like a sufficient reason to me. OTOH i'd try to argue that in order to add reliable resource synchronization an effort to get it working in the context of exceptions and generators would add more than just a macro :-) Also thinking more along the lines of Armin's alternative idea (incorporate it in into the for-loop protocol) is interesting as it avoids adding a new keyword which seems of great value to me. on a side note, it would be great if we could soon (after integration of the parser/compiler) branch off PyPy to do quick experiments with different approaches. IIRC Guido mentioned at EuroPython that this might be an interesting use of PyPy (to easily test new language features). This way it's also easier for PyPy to keep up to date :-) cheers, holger From pedronis at bluewin.ch Mon Sep 22 14:19:16 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Mon Sep 22 14:17:34 2003 Subject: [Python-Dev] pep 310 (reliable acquisition/release pairs) In-Reply-To: <20030922194137.C24546@prim.han.de> References: <2msmmp59f5.fsf@starship.python.net> <20030919174240.C31102@prim.han.de> <2msmmp59f5.fsf@starship.python.net> Message-ID: <5.2.1.1.0.20030922200953.027ef7f8@pop.bluewin.ch> At 19:41 22.09.2003 +0200, Holger Krekel wrote: >Michael Hudson wrote: > > Holger Krekel writes: > > > The PEP actually is about interacting with the execution > > > of a code block. It allows to define (one-shot) interception points > > > for entering and leaving a code block. Now there are at least > > > two interesting cases which the PEP does (quite explicitely) not cover: > > > > > > - what to do with exceptions > > > > > > - what to do with yield > > > > > > IMHO introducing a new block statement at this stage in language > > > development warrants an effort to tackle these cases (and maybe more > > > like e.g. allowing the handler to trigger looping). > > > > > > This is probably best done with trying to directly design a protocol > between > > > the "interpreter-loop" and the - what i'd call - the "execution > handler". > > > > Well, in writing PEP 310 (as I suspect you know) I was aiming for a > > simple, almost entirely syntactic way of shortening a common pattern. > > You seem to be gunning for something far deeper here. > >Hmmm, introducing a new block statement just for "shortening a common >pattern" doesn't sound like a sufficient reason to me. OTOH i'd try to argue >that in order to add reliable resource synchronization an effort to get >it working in the context of exceptions and generators would add more >than just a macro :-) > >Also thinking more along the lines of Armin's alternative idea >(incorporate it in into the for-loop protocol) is interesting as >it avoids adding a new keyword which seems of great value to me. can we avoid to have now the discussion that we don't wanted now. >on a side note, it would be great if we could soon (after integration of the >parser/compiler) branch off PyPy to do quick experiments with different >approaches. IIRC Guido mentioned at EuroPython that this might >be an interesting use of PyPy (to easily test new language features). >This way it's also easier for PyPy to keep up to date :-) I won't say anything, see above, BUT ... From barry at python.org Mon Sep 22 14:18:00 2003 From: barry at python.org (Barry Warsaw) Date: Mon Sep 22 14:18:07 2003 Subject: [Python-Dev] Re: latest bsddb3 test problems In-Reply-To: <200309220704.h8M74NNk026354@localhost.localdomain> References: <1064198720.2074.54.camel@localhost.localdomain> <200309220704.h8M74NNk026354@localhost.localdomain> Message-ID: <3F6F3CD8.6010408@python.org> Anthony Baxter wrote: > > test_bsddb3 is failing on pretty much every box I've tried it on here - > Solaris and Redhat linux, most running the latest-n-greatest release of > DB4.1 (we use it pretty heavily in a variety of internal applications). > > Solaris 8 (DB 4.1.25. gcc 2.96): > DBAgainError: (11, 'Resource temporarily unavailable -- mmap: Resource temporarily unavailable') > > Solaris 7 (DB 4.1.25, gcc 2.95.2) > Crashes with a SIGBUS. The version of _bsddb.c from a week ago has a lot of > errors, but doesn't crash. > > Redhat 10beta/Rawhide (DB 4.1.25) > DBNoMemoryError: (12, 'Cannot allocate memory -- Lock table is out of available locks') > alternating with SIGBUS > > Redhat 7.1,7.2,7.3 (DB 3.2.9) > Crashes with a SIGBUS. FWIW, I just tested it on RH9, DB 4.1.25, release23-maint branch. Worked goodly for me. -Barry % ./python Lib/test/test_bsddb3.py -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Sleepycat Software: Berkeley DB 4.1.25: (December 19, 2002) bsddb.db.version(): (4, 1, 25) bsddb.db.__version__: 4.2.0 bsddb.db.cvsid: $Id: _bsddb.c,v 1.17.6.2 2003/09/21 23:10:23 greg Exp $ python version: 2.3+ (#1, Sep 22 2003, 13:35:52) [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= ........................................................................................................................................................................................................................ ---------------------------------------------------------------------- Ran 216 tests in 131.194s OK From tim.one at comcast.net Mon Sep 22 14:25:26 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Sep 22 14:25:41 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: <3F6EBF2C.3040108@lemburg.com> Message-ID: [Tim] >> At the moment, it appears there's no identified reason to care about >> signedness of a greater-than 16-bit type, [M.-A. Lemburg] > Sure there is: first of all, having a single type that can > be signed on some platforms and unsigned on others is a bad > thing per se We inherit that from C, though -- it's fine by C if wchar_t is signed or unsigned, just as it refused to define the signedness of char. > and second the 32-bit signed wchar_t value was what triggered this > thread in the first place. What triggered the thread originally was a segfault due to the code making a branch based on the content of uninitialized memory. The code clearly didn't *think* it was reading up random heap bits, so that was a bug regardless of wchar_t's signedness. That wchar_t happened to be a signed 32-bit type on Jeremy's box is what uncovered the read-uninitialized-memory bug. If there's no other code vulnerable to bad behavior if wchar_t is a signed 32-bit type (nobody has identified another case), objections to it being signed anyway seem technically groundless. Martin did give a technical reason (efficiency) for wanting to continue to use wchar_t on Jeremy's system. From barry at python.org Mon Sep 22 14:18:00 2003 From: barry at python.org (Barry Warsaw) Date: Mon Sep 22 14:50:54 2003 Subject: [Python-Dev] Re: latest bsddb3 test problems In-Reply-To: <200309220704.h8M74NNk026354@localhost.localdomain> References: <1064198720.2074.54.camel@localhost.localdomain> <200309220704.h8M74NNk026354@localhost.localdomain> Message-ID: <3F6F3CD8.6010408@python.org> Anthony Baxter wrote: > > test_bsddb3 is failing on pretty much every box I've tried it on here - > Solaris and Redhat linux, most running the latest-n-greatest release of > DB4.1 (we use it pretty heavily in a variety of internal applications). > > Solaris 8 (DB 4.1.25. gcc 2.96): > DBAgainError: (11, 'Resource temporarily unavailable -- mmap: Resource temporarily unavailable') > > Solaris 7 (DB 4.1.25, gcc 2.95.2) > Crashes with a SIGBUS. The version of _bsddb.c from a week ago has a lot of > errors, but doesn't crash. > > Redhat 10beta/Rawhide (DB 4.1.25) > DBNoMemoryError: (12, 'Cannot allocate memory -- Lock table is out of available locks') > alternating with SIGBUS > > Redhat 7.1,7.2,7.3 (DB 3.2.9) > Crashes with a SIGBUS. FWIW, I just tested it on RH9, DB 4.1.25, release23-maint branch. Worked goodly for me. -Barry % ./python Lib/test/test_bsddb3.py -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Sleepycat Software: Berkeley DB 4.1.25: (December 19, 2002) bsddb.db.version(): (4, 1, 25) bsddb.db.__version__: 4.2.0 bsddb.db.cvsid: $Id: _bsddb.c,v 1.17.6.2 2003/09/21 23:10:23 greg Exp $ python version: 2.3+ (#1, Sep 22 2003, 13:35:52) [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= ........................................................................................................................................................................................................................ ---------------------------------------------------------------------- Ran 216 tests in 131.194s OK From mal at lemburg.com Mon Sep 22 15:13:26 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Sep 22 15:13:31 2003 Subject: [Python-Dev] Re: [Python-checkins]python/dist/src/Objects unicodeobject.c, 2.197, 2.198 In-Reply-To: References: Message-ID: <3F6F49D6.9050203@lemburg.com> Tim Peters wrote: > [Tim] > >>>At the moment, it appears there's no identified reason to care about >>>signedness of a greater-than 16-bit type, > > > [M.-A. Lemburg] > >>Sure there is: first of all, having a single type that can >>be signed on some platforms and unsigned on others is a bad >>thing per se > > > We inherit that from C, though -- it's fine by C if wchar_t is signed or > unsigned, just as it refused to define the signedness of char. It maybe fine for C... it is not for the Unicode implementation since that has always assumed Py_UNICODE to be unsigned. This is fixed now. >>and second the 32-bit signed wchar_t value was what triggered this >>thread in the first place. > > What triggered the thread originally was a segfault due to the code making a > branch based on the content of uninitialized memory. The code clearly > didn't *think* it was reading up random heap bits, so that was a bug > regardless of wchar_t's signedness. True, but the test (unicode->str[0] < 256) is what revealed a second bug and that's what we've been discussing all along. > That wchar_t happened to be a signed > 32-bit type on Jeremy's box is what uncovered the read-uninitialized-memory > bug. > > If there's no other code vulnerable to bad behavior if wchar_t is a signed > 32-bit type (nobody has identified another case), objections to it being > signed anyway seem technically groundless. There are more comparisons of the above type in the code and even worse: it is documented that Py_UNICODE is unsigned, so it's very likely that code external to the Python distribution such as codec packages or applications talking to libraries use that assumption as well. > Martin did give a technical > reason (efficiency) for wanting to continue to use wchar_t on Jeremy's > system. Python won't be using wchar_t on those systems anymore, so the problem is solved and the original intent restored. If efficiency matters programmers are always free to cast Py_UNICODE to wchar_t on these systems for fast read-only access. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Sep 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mfb at lotusland.dyndns.org Mon Sep 22 20:36:36 2003 From: mfb at lotusland.dyndns.org (Matthew F. Barnes) Date: Mon Sep 22 20:51:19 2003 Subject: [Python-Dev] Consistent logging in the standard library In-Reply-To: <3122.192.168.1.1.1063886697.squirrel@server.lotusland.dyndns.org> References: <005101c37d64$d37270c0$652b6992@alpha><1063835810.14316.20.camel@geddy ><16232.57433.388130.573539@montanaro.dyndns.org> <00ad01c37db8$d8bc9d40$652b6992@alpha> <3122.192.168.1.1.1063886697.squirrel@server.lotusland.dyndns.org> Message-ID: <2449.192.168.1.106.1064277396.squirrel@server.lotusland.dyndns.org> To recap recent discussion on this thread, there seems to be consensus among those I've spoken to that there should be a namespace in the logging hierarchy named "python" that is reserved for the standard library, and that standard library modules and packages should use loggers within this namespace that are named after themselves (i.e. "python." + __name__). For example, the asyncore module should use a logger named "python.asyncore". The modules may also define child loggers as needed, such as "python.asyncore.socket". This week's topic: Configuration I tend to think that, at least for Python 2.4, each logger should be configured to mimic the behavior of the Python 2.3 version of its corresponding module in terms of message format and destination (e.g. log-file, sys.stdout, sys.stderr, etc.). I think it would be nice for each "standard library logger" to default to such a configuration automatically, while still allowing a configuration file to customize them. The issue is where this default configuration should reside and when it should be applied. My reference implementation currently has each module configure its own logger at the end of the module, so that those statements are executed when the module is imported. But I'm not confident that this is the best approach, or even a correct approach. For example, consider what would happen if a configuration file that defines a section for the python.asyncore logger is processed *before* the asyncore module is imported. I'm looking for some helpful suggestions from the community. - What's the best approach for configuring the "standard library loggers", or should we at all? - Should the standard library define its own logging configuration file, and what would be the semantics around that? - Perhaps the logging API should provide for a way to test whether a particular logger is defined, so that the modules could check that before applying their default settings? Matthew Barnes From fdrake01 at comcast.net Tue Sep 23 01:39:53 2003 From: fdrake01 at comcast.net (fdrake01@comcast.net) Date: Tue Sep 23 01:40:02 2003 Subject: [Python-Dev] Doc freeze for Python 2.3.1 Message-ID: Python's Doc/ tree should be considered frozen on the release23-maint branch. I've made the version number updates, but the machine I normally build the documentation on is having some connectivity problems (which hopefully can be resolved tomorrow morning). Using what I have available to me now, I should be able to build everything but the iSilo version of the docs; I don't think a delay for those is needed; they can be added when available. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From aleaxit at yahoo.com Tue Sep 23 03:46:18 2003 From: aleaxit at yahoo.com (Alex Martelli) Date: Tue Sep 23 03:46:24 2003 Subject: [Python-Dev] Discussion on adding rsplit() for strings and unicode objects. In-Reply-To: <6.0.0.22.0.20030922175257.020ecdd0@torment.chelsea.private> References: <20030919083346.GA11541@tummy.com> <200309190612.15965.fincher.8@osu.edu> <6.0.0.22.0.20030922175257.020ecdd0@torment.chelsea.private> Message-ID: <200309230946.18385.aleaxit@yahoo.com> On Monday 22 September 2003 06:56 pm, Barry Scott wrote: > >I'm all for it. I've had to implement rsplit on my own as well, and > > always lamented (but never did anything, unfortunately) about its lack in > > the standard library. That strings have rfind, rindex, rstrip, and > > rjust, but not rsplit always seemed somewhat less than complete to me. > > These "r" are not a single family... some mean reverse some mean right. ??? They are: >>> [x for x in dir('') if x[:1]=='r'] ['replace', 'rfind', 'rindex', 'rjust', 'rstrip'] 'replace' is clearly distinct from the other (it's not an 'eplace' from the right:-). All others can be read as "from the right" (reading them as "reverse" is very obviously strained). Find and Index the rightmost occurrence, right-justify, strip characters from the right of the string. Alex From python-dev-list at pythonconsulting.com Tue Sep 23 04:55:39 2003 From: python-dev-list at pythonconsulting.com (Steve Purcell) Date: Tue Sep 23 04:55:44 2003 Subject: [Python-Dev] Merge unittest.py fixes to 2.3 maintenance branch? Message-ID: <200309230955.39974.python-dev-list@pythonconsulting.com> Hi all, I recently checked into HEAD fixes for a couple of niggling issues with the unittest module, and I think it might be worth merging them into the 2.3 maintenance branch. Any thumbs up or down on that? I see that a tag is set for 2.3.1, but it's not clear to me if that's final. Most significant changes are: - Fixed loading of tests by name when name refers to unbound method (PyUnit issue 563882, thanks to Alexandre Fayolle) - Ignore non-callable attributes of classes when searching for test method names (PyUnit issue 769338, thanks to Seth Falcon) - New assertTrue and assertFalse aliases for comfort of JUnit users - Automatically discover 'runTest()' test methods (PyUnit issue 469444, thanks to Roeland Rengelink) - Dropped Python 1.5.2 compatibility, merged appropriate shortcuts from Python CVS; should work with Python >= 2.1. - Removed all references to string module by using string methods instead Best wishes, -Steve -- Steve Purcell, Pythangelist http://www.pythonconsulting.com/ From anthony at interlink.com.au Tue Sep 23 05:23:33 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Sep 23 05:25:32 2003 Subject: [Python-Dev] latest bsddb3 test problems In-Reply-To: <200309220811.h8M8BVEN027031@localhost.localdomain> Message-ID: <200309230923.h8N9NZHf032580@localhost.localdomain> On a friend's debian testing machine, the bsddb test suite crashes with a deadlock on _some_ occasions. On a Mac OS X machine, the test suite hangs :-( On the Mac OS X machine, the bsddb3 stuff doesn't get built under 2.3. On the other hand, code using bsddb seems to work fine, so it's almost certainly just that the test suite needs to be fixed to not be broken. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Tue Sep 23 05:27:03 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Sep 23 05:28:50 2003 Subject: [Python-Dev] Merge unittest.py fixes to 2.3 maintenance branch? In-Reply-To: <200309230955.39974.python-dev-list@pythonconsulting.com> Message-ID: <200309230927.h8N9R3I4032660@localhost.localdomain> >>> Steve Purcell wrote > Hi all, > > I recently checked into HEAD fixes for a couple of niggling issues with the > unittest module, and I think it might be worth merging them into the 2.3 > maintenance branch. Any thumbs up or down on that? I see that a tag is set > for 2.3.1, but it's not clear to me if that's final. Pretty much. Thomas and I are doing a bunch of testing on the release candidate tarball and exe, but unless we see something seriously busted, I don't think cutting a new release is going to happen. On the plus side, if the discipline to keep features out of the maintenance releases continues, there's no reason to avoid a 2.3.2 somewhere down the track. -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Tue Sep 23 05:43:12 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Sep 23 05:44:53 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go Message-ID: <200309230943.h8N9hCd1000446@localhost.localdomain> At www.python.org/2.3.1 there's links to the release candidates for 2.3.1 - Thomas and I are still running all sorts of tests against it, but if someone else wants to give it a go, that'd be good. We're on IRC #python-dev on irc.freenode.net. From anthony at interlink.com.au Tue Sep 23 05:46:15 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Sep 23 05:55:15 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <200309230943.h8N9hCd1000446@localhost.localdomain> Message-ID: <200309230946.h8N9kFrP000507@localhost.localdomain> >>> Anthony Baxter wrote > > At www.python.org/2.3.1 there's links to the release candidates for > 2.3.1 - Thomas and I are still running all sorts of tests against it, > but if someone else wants to give it a go, that'd be good. We're on > IRC #python-dev on irc.freenode.net. It's probably worth pointing out that this page isn't linked into the site yet, nor is it in the sitemap. From aahz at pythoncraft.com Tue Sep 23 10:07:22 2003 From: aahz at pythoncraft.com (Aahz) Date: Tue Sep 23 10:07:36 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <200309230946.h8N9kFrP000507@localhost.localdomain> References: <200309230943.h8N9hCd1000446@localhost.localdomain> <200309230946.h8N9kFrP000507@localhost.localdomain> Message-ID: <20030923140722.GB3988@panix.com> On Tue, Sep 23, 2003, Anthony Baxter wrote: > > It's probably worth pointing out that this page isn't linked into the > site yet, nor is it in the sitemap. Don't worry, one of the webmasters (probably me, AMK, or Fred) will make sure to tidy up. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "It is easier to optimize correct code than to correct optimized code." --Bill Harlan From aahz at pythoncraft.com Tue Sep 23 10:09:31 2003 From: aahz at pythoncraft.com (Aahz) Date: Tue Sep 23 10:09:43 2003 Subject: [Python-Dev] Merge unittest.py fixes to 2.3 maintenance branch? In-Reply-To: <200309230955.39974.python-dev-list@pythonconsulting.com> References: <200309230955.39974.python-dev-list@pythonconsulting.com> Message-ID: <20030923140931.GC3988@panix.com> On Tue, Sep 23, 2003, Steve Purcell wrote: > > Most significant changes are: > > - Fixed loading of tests by name when name refers to unbound > method (PyUnit issue 563882, thanks to Alexandre Fayolle) > - Ignore non-callable attributes of classes when searching for test > method names (PyUnit issue 769338, thanks to Seth Falcon) These two are probably okay to port back (though too late for 2.3.1). > - New assertTrue and assertFalse aliases for comfort of JUnit users > - Automatically discover 'runTest()' test methods (PyUnit issue 469444, > thanks to Roeland Rengelink) > - Dropped Python 1.5.2 compatibility, merged appropriate shortcuts from > Python CVS; should work with Python >= 2.1. > - Removed all references to string module by using string methods instead These should wait on 2.4. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "It is easier to optimize correct code than to correct optimized code." --Bill Harlan From jeremy at alum.mit.edu Tue Sep 23 11:21:45 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Tue Sep 23 11:22:14 2003 Subject: [Python-Dev] latest bsddb3 test problems In-Reply-To: <200309230923.h8N9NZHf032580@localhost.localdomain> References: <200309230923.h8N9NZHf032580@localhost.localdomain> Message-ID: <1064330505.2072.11.camel@localhost.localdomain> On Tue, 2003-09-23 at 05:23, Anthony Baxter wrote: > On a friend's debian testing machine, the bsddb test suite crashes with > a deadlock on _some_ occasions. On a Mac OS X machine, the test suite > hangs :-( > > On the Mac OS X machine, the bsddb3 stuff doesn't get built under 2.3. > > On the other hand, code using bsddb seems to work fine, so it's almost > certainly just that the test suite needs to be fixed to not be broken. I think the question is whether the tests are doing something completely artificial that fails, or if they are simply provoking failures that would be particularly difficult to provoke in real life applications. In Zope's BDBStorage, we've definitely seen some weird behavior. For example, we had a bug in our code where we tried to delete an entry from a BTree that did not exist. That operation failed with a DBNotFoundError. We caught that error can called txn.abort(); the abort failed with a DBRunRecoveryError. We didn't have any idea what the abort would fail in that case. It may be that we misunderstood how to use the Berkeley storage, but it leaves me with the uneasy feeling that something isn't right. The failing tests make me feel the same way. Jeremy From arigo at tunes.org Tue Sep 23 11:27:56 2003 From: arigo at tunes.org (Armin Rigo) Date: Tue Sep 23 11:30:29 2003 Subject: [Python-Dev] Fun with 2.3 shutdown In-Reply-To: <3F6D7835.5080407@zope.com> References: <20030919120100.GB23317@vicky.ecs.soton.ac.uk> <3F6D7835.5080407@zope.com> Message-ID: <20030923152756.GA20907@vicky.ecs.soton.ac.uk> Hello Jim, On Sun, Sep 21, 2003 at 06:06:45AM -0400, Jim Fulton wrote: > Or, given cyclic GC, why isn't it enough to clear sys.modules? Because the modules dictionary is stored in PyThreadState_Get()->interp->modules, where the GC cannot see it, which prevent the dictionary from being freed; in turn, it prevents any module from being freed. Still, it means (at a first sight) that there is actually no need to clear each module's globals at all. Just removing them all from interp->modules is fine. In practice there are probably a few issues that I'm not fully aware of, like what occurs if we change module_dealloc() so that it no longer clears the module's global dictionary -- a tweak that seems to come from a pre-GC history. Not speaking about threads in particular, and compatibility problems apart, wouldn't it make sense to change sys.modules and interp->modules into a WeakValueDictionary (or some equivalent refcount-based hack like the one for interned strings)? This would give memory management at the module level, and would allow a cleaner shutdown procedure as well. It would however change Python's semantics of "a module is only loaded once unless you ask or you mess with sys.modules explicitely". A bientot, Armin. From python-dev-list at pythonconsulting.com Tue Sep 23 11:42:45 2003 From: python-dev-list at pythonconsulting.com (Steve Purcell) Date: Tue Sep 23 11:42:54 2003 Subject: [Python-Dev] Merge unittest.py fixes to 2.3 maintenance branch? In-Reply-To: <20030923140931.GC3988@panix.com> References: <200309230955.39974.python-dev-list@pythonconsulting.com> <20030923140931.GC3988@panix.com> Message-ID: <200309231642.45495.python-dev-list@pythonconsulting.com> On Tuesday 23 September 2003 15:09, Aahz wrote: > On Tue, Sep 23, 2003, Steve Purcell wrote: > > - Fixed loading of tests by name when name refers to unbound > > method (PyUnit issue 563882, thanks to Alexandre Fayolle) > > - Ignore non-callable attributes of classes when searching for test > > method names (PyUnit issue 769338, thanks to Seth Falcon) > > These two are probably okay to port back (though too late for 2.3.1). Great. > > - New assertTrue and assertFalse aliases for comfort of JUnit users > > - Automatically discover 'runTest()' test methods (PyUnit issue 469444, > > thanks to Roeland Rengelink) > > - Dropped Python 1.5.2 compatibility, merged appropriate shortcuts from > > Python CVS; should work with Python >= 2.1. > > - Removed all references to string module by using string methods instead > > These should wait on 2.4. Yup, fair enough, though I might except the fix for issue 469444 from that, since it leads to unexpected behaviour when following examples in the std library docs. (i.e. TestCase classes that define only the method 'runTest()' will not be automatically run by unittest.main()). Best wishes, -Steve -- Steve Purcell, Pythangelist http://www.pythonconsulting.com/ From pje at telecommunity.com Tue Sep 23 12:09:18 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Sep 23 12:09:23 2003 Subject: [Python-Dev] Fun with 2.3 shutdown In-Reply-To: <20030923152756.GA20907@vicky.ecs.soton.ac.uk> References: <3F6D7835.5080407@zope.com> <20030919120100.GB23317@vicky.ecs.soton.ac.uk> <3F6D7835.5080407@zope.com> Message-ID: <5.1.0.14.0.20030923120249.025a0720@mail.telecommunity.com> At 04:27 PM 9/23/03 +0100, Armin Rigo wrote: >Not speaking about threads in particular, and compatibility problems apart, >wouldn't it make sense to change sys.modules and interp->modules into a >WeakValueDictionary (or some equivalent refcount-based hack like the one for >interned strings)? This would give memory management at the module level, and >would allow a cleaner shutdown procedure as well. It would however change >Python's semantics of "a module is only loaded once unless you ask or you mess >with sys.modules explicitely". If I understand your suggestion correctly, the following code would break under such an arrangement: Foo.py: ====== x = 1 def getX(): return X Bar.py: ======= from Foo import getX # returns getX() bound to one set of globals import Foo # imports a *new* Foo module, with new dictionary! Foo.x = 2 # change the new Foo.x to 2 print getX() # call the getX() with the old globals, and print 1 So, unless a module's dictionary were to reference the module (or functions were to reference the module rather than (or in addition to) the module dictionary), it seems the proposed semantics would lead to unexpected results. From arigo at tunes.org Tue Sep 23 12:49:09 2003 From: arigo at tunes.org (Armin Rigo) Date: Tue Sep 23 12:51:43 2003 Subject: [Python-Dev] Fun with 2.3 shutdown In-Reply-To: <5.1.0.14.0.20030923120249.025a0720@mail.telecommunity.com> References: <3F6D7835.5080407@zope.com> <20030919120100.GB23317@vicky.ecs.soton.ac.uk> <3F6D7835.5080407@zope.com> <5.1.0.14.0.20030923120249.025a0720@mail.telecommunity.com> Message-ID: <20030923164909.GC8415@vicky.ecs.soton.ac.uk> Hello Phillip, On Tue, Sep 23, 2003 at 12:09:18PM -0400, Phillip J. Eby wrote: > So, unless a module's dictionary were to reference the module (or functions > were to reference the module rather than (or in addition to) the module > dictionary), it seems the proposed semantics would lead to unexpected > results. Right. I'm not sure I understand the reasons behind the current module/globals relationship. As modules zap their globals with None when they are deallocated, we observe the following behavior: (foo.py) def g(): return 5 def f(): return g() >>> from foo import f >>> import sys; del sys.modules['test4'] >>> f() Traceback (most recent call last): File "", line 1, in ? File "foo.py", line 4, in f return g() TypeError: 'NoneType' object is not callable Possibly far-fetched, but I wouldn't be surprized to find large applications that mess with sys.modules in some way. For example, in one case, to ensure that a whole collection of interdependent modules will be reloaded on demand after I detect a change in one of them, I'm simply removing them all from sys.modules. Armin From pje at telecommunity.com Tue Sep 23 13:36:10 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Sep 23 13:36:15 2003 Subject: [Python-Dev] Fun with 2.3 shutdown In-Reply-To: <20030923164909.GC8415@vicky.ecs.soton.ac.uk> References: <5.1.0.14.0.20030923120249.025a0720@mail.telecommunity.com> <3F6D7835.5080407@zope.com> <20030919120100.GB23317@vicky.ecs.soton.ac.uk> <3F6D7835.5080407@zope.com> <5.1.0.14.0.20030923120249.025a0720@mail.telecommunity.com> Message-ID: <5.1.0.14.0.20030923132744.03e06560@mail.telecommunity.com> At 05:49 PM 9/23/03 +0100, Armin Rigo wrote: >Hello Phillip, > >On Tue, Sep 23, 2003 at 12:09:18PM -0400, Phillip J. Eby wrote: > > So, unless a module's dictionary were to reference the module (or > functions > > were to reference the module rather than (or in addition to) the module > > dictionary), it seems the proposed semantics would lead to unexpected > > results. > >Right. > >I'm not sure I understand the reasons behind the current module/globals >relationship.As modules zap their globals with None when they are Well, a function can have globals that aren't a module dictionary, so that's probably why functions don't refer to their modules directly. As for the None thing, it's to break circular references. >deallocated, we observe the following behavior: > >(foo.py) >def g(): > return 5 >def f(): > return g() > > >>> from foo import f > >>> import sys; del sys.modules['test4'] > >>> f() >Traceback (most recent call last): > File "", line 1, in ? > File "foo.py", line 4, in f > return g() >TypeError: 'NoneType' object is not callable Yeah, I've run into this (and weirder!) things before, when tracing down refcount bugs in C extensions. But, I don't think anybody in their right mind would *rely* on this behavior, so your suggestion of e.g. clearing sys.modules and letting GC do the rest seems worthy of investigation. >Possibly far-fetched, but I wouldn't be surprized to find large applications >that mess with sys.modules in some way. For example, in one case, to ensure >that a whole collection of interdependent modules will be reloaded on demand >after I detect a change in one of them, I'm simply removing them all from >sys.modules. Of course, you have to also delete all modules and objects that reference the modules you want to delete. And that would still be the case under your "clear sys.modules at shutdown" proposal. OTOH, it's not clear what to do with uncollectable objects. E.g., if you have a module global that is or contains an object with a __del__ method, and is part of a cycle back to that module. I think you'd *still* be stuck going back and clearing the modules or setting their contents to None. From nick at nick.uklinux.net Tue Sep 23 14:25:33 2003 From: nick at nick.uklinux.net (Nick Roberts) Date: Tue Sep 23 14:31:58 2003 Subject: [Python-Dev] Integration with Eclipse In-Reply-To: <3F6D1AB7.1050608@sabaydi.com> References: <3F6B11BA.6000008@sabaydi.com> <16235.18128.422738.929390@nick.uklinux.net> <3F6B533E.5000403@sabaydi.com> <16236.28754.105629.565153@nick.uklinux.net> <3F6C7B09.5060608@sabaydi.com> <16236.51140.284372.526221@nick.uklinux.net> <3F6CF076.30409@ocf.berkeley.edu> <3F6D1AB7.1050608@sabaydi.com> Message-ID: <16240.36893.520437.974584@nick.uklinux.net> Kevin J. Butler writes: > ... > > I consider this more pdb-related than emacs-specific - > I'd expect that integration with, say, Eclipse would benefit from this > enhancement as well, so I've been hoping to get some feedback from the > list... > > BTW, I've submitted the patch, it is 809887: > http://sourceforge.net/tracker/index.php?func=detail&aid=809887&group_id=5470&atid=305470 > > kb The syntax of responses from pdb commands suggests to me that there is a fair amount of GDB in the underlying code. The GDB developers have written an interface called GDB/MI (MI = Machine Interface) for integration with an IDE. Since the Eclipse project is using this interface to integrate with GDB there may be some overlap for doing the same thing with with pdb. Nick From jeremy at zope.com Tue Sep 23 16:48:07 2003 From: jeremy at zope.com (Jeremy Hylton) Date: Tue Sep 23 16:48:52 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <200309230943.h8N9hCd1000446@localhost.localdomain> References: <200309230943.h8N9hCd1000446@localhost.localdomain> Message-ID: <1064350087.2072.66.camel@localhost.localdomain> On Tue, 2003-09-23 at 05:43, Anthony Baxter wrote: > At www.python.org/2.3.1 there's links to the release candidates for > 2.3.1 - Thomas and I are still running all sorts of tests against it, > but if someone else wants to give it a go, that'd be good. We're on > IRC #python-dev on irc.freenode.net. You most post a note to python-dev when you think you've got the final release files. There are lots of things that can go wrong the first time around, and we can do some last minute sanity checks. I'd be happy to test on RH9 and Win2k. Jeremy From theller at python.net Tue Sep 23 17:02:42 2003 From: theller at python.net (Thomas Heller) Date: Tue Sep 23 17:02:49 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <1064350087.2072.66.camel@localhost.localdomain> (Jeremy Hylton's message of "Tue, 23 Sep 2003 16:48:07 -0400") References: <200309230943.h8N9hCd1000446@localhost.localdomain> <1064350087.2072.66.camel@localhost.localdomain> Message-ID: Jeremy Hylton writes: > On Tue, 2003-09-23 at 05:43, Anthony Baxter wrote: >> At www.python.org/2.3.1 there's links to the release candidates for >> 2.3.1 - Thomas and I are still running all sorts of tests against it, >> but if someone else wants to give it a go, that'd be good. We're on >> IRC #python-dev on irc.freenode.net. > > You most post a note to python-dev when you think you've got the final > release files. There are lots of things that can go wrong the first > time around, and we can do some last minute sanity checks. I'd be happy > to test on RH9 and Win2k. They are not yet ready. Tim Peters found that some Windows version numbers have not been incremented, and I'm currently changing this. It seems at least the source tarball and the windows installer have to been rebuilt. But since these are changes which should not affect the functionality too much, testing can go on. Thomas From jeremy at zope.com Tue Sep 23 17:07:57 2003 From: jeremy at zope.com (Jeremy Hylton) Date: Tue Sep 23 17:09:23 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: References: <200309230943.h8N9hCd1000446@localhost.localdomain> <1064350087.2072.66.camel@localhost.localdomain> Message-ID: <1064351277.2072.68.camel@localhost.localdomain> On Tue, 2003-09-23 at 17:02, Thomas Heller wrote: > > You most post a note to python-dev when you think you've got the final (BTW this was supposed to say "You might post" but I got a little ost happy.) > > release files. There are lots of things that can go wrong the first > > time around, and we can do some last minute sanity checks. I'd be happy > > to test on RH9 and Win2k. > > They are not yet ready. Tim Peters found that some Windows version > numbers have not been incremented, and I'm currently changing this. > It seems at least the source tarball and the windows installer have to > been rebuilt. > > But since these are changes which should not affect the functionality > too much, testing can go on. It can't go on unless you tell me where the files are . Jeremy From theller at python.net Tue Sep 23 17:17:54 2003 From: theller at python.net (Thomas Heller) Date: Tue Sep 23 17:18:00 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <1064351277.2072.68.camel@localhost.localdomain> (Jeremy Hylton's message of "Tue, 23 Sep 2003 17:07:57 -0400") References: <200309230943.h8N9hCd1000446@localhost.localdomain> <1064350087.2072.66.camel@localhost.localdomain> <1064351277.2072.68.camel@localhost.localdomain> Message-ID: Jeremy Hylton writes: > It can't go on unless you tell me where the files are . I thought you had seen this: >>> Anthony Baxter wrote > > At www.python.org/2.3.1 there's links to the release candidates for > 2.3.1 - Thomas and I are still running all sorts of tests against it, > but if someone else wants to give it a go, that'd be good. We're on > IRC #python-dev on irc.freenode.net. Thomas From doko at cs.tu-berlin.de Tue Sep 23 17:32:27 2003 From: doko at cs.tu-berlin.de (Matthias Klose) Date: Tue Sep 23 17:36:53 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <200309230943.h8N9hCd1000446@localhost.localdomain> References: <200309230943.h8N9hCd1000446@localhost.localdomain> Message-ID: <16240.48107.934445.199555@gargle.gargle.HOWL> Anthony Baxter writes: > > At www.python.org/2.3.1 there's links to the release candidates for > 2.3.1 - Thomas and I are still running all sorts of tests against it, > but if someone else wants to give it a go, that'd be good. We're on > IRC #python-dev on irc.freenode.net. please could you consider - either to hand-edit the generated configure script - or rebuild configure with autoconf from current CVS to enable a correct configuration on HPUX? Please see http://mail.python.org/pipermail/python-dev/2003-August/037648.html I think the first solution should be more conservative at this point, maybe using a nested ifdef instead of using the defined keyword. If neither solution is appropriate, please add a note to the target specific install section. Thanks, Matthias From zeddicus at satokar.com Tue Sep 23 18:14:33 2003 From: zeddicus at satokar.com (Michael Bartl) Date: Tue Sep 23 18:14:37 2003 Subject: [Python-Dev] Help offered Message-ID: <20030923221433.GA1764@satokar> Hi all! I've sent my first patch to python sf.net and noticed that there are still patches from ~2001! Is there a general patch review cycle or something? I'd like to offer my help if there is interest. (writing/reviewing patches, sorting out bugreports, ...) My "python" background: - Working as a software engineer in air traffic control. - I suggested and embedded the python interpreter into our layout interpreter, which was kind of tricky due to the PowerPC platform. (python doesn't really like to be cross-compiled) This leads to the fact that some air traffic control centers around the world are now Python powered :) Have fun, Bartl Michael From aahz at pythoncraft.com Tue Sep 23 20:19:09 2003 From: aahz at pythoncraft.com (Aahz) Date: Tue Sep 23 20:19:15 2003 Subject: [Python-Dev] Help offered In-Reply-To: <20030923221433.GA1764@satokar> References: <20030923221433.GA1764@satokar> Message-ID: <20030924001908.GB28288@panix.com> On Wed, Sep 24, 2003, Michael Bartl wrote: > > I've sent my first patch to python sf.net and noticed that there > are still patches from ~2001! Is there a general patch review cycle > or something? I'd like to offer my help if there is interest. > (writing/reviewing patches, sorting out bugreports, ...) Excellent! Currently, it's rather ad-hoc. Just pick a patch that interests you and make a comment about its suitability. Make sure to check whether the patch comes with appropriate documentation. [*] If you don't see activity on the patch within a week or two, feel free to ask here. Right now isn't the best time to poke people because we're gearing up for the 2.3.1 release. Read the Developer's Guide if you haven't yet (under "Python Project" on the home page). > This leads to the fact that some air traffic control centers around > the world are now Python powered :) Cool! -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "It is easier to optimize correct code than to correct optimized code." --Bill Harlan From anthony at interlink.com.au Tue Sep 23 20:38:02 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Sep 23 20:39:53 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <16240.48107.934445.199555@gargle.gargle.HOWL> Message-ID: <200309240038.h8O0c2VX012396@localhost.localdomain> >>> Matthias Klose wrote > please could you consider > > - either to hand-edit the generated configure script > > - or rebuild configure with autoconf from current CVS > > to enable a correct configuration on HPUX? Michael Hudson and I spent several painful hours on this last night - at the moment, there's an entry in the bugs.html page detailing the workaround (edit pyconfig.h after running configure). At this point in the cycle I'm extremely uncomfortable trying to either hand-edit configure or running a new autoconf. I'm recutting the tarball now, I'll add a note to the README as well. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Tue Sep 23 21:29:51 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Sep 23 21:32:00 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: Message-ID: <200309240129.h8O1Tqft013440@localhost.localdomain> There's a new release candidate tarball/exe on www.python.org/2.3.1 Feedback welcome - either here or irc.freenode.net #python-dev. Changes since yesterday: Windows build numbers updated HP-UX README note added Misc/RPM notes updated From jepler at unpythonic.net Tue Sep 23 22:50:25 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Tue Sep 23 22:50:44 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <200309240129.h8O1Tqft013440@localhost.localdomain> References: <200309240129.h8O1Tqft013440@localhost.localdomain> Message-ID: <20030924025019.GA1590@unpythonic.net> As reported on IRC, I had good test results on the following machines with the RC2 tarball: FreeBSD zuul.dsndata.com 4.7-RELEASE FreeBSD 4.7-RELEASE #0: Wed Oct 9 15:08:34 GMT 2002 Linux parrot 2.4.20-9 #1 Wed Apr 2 13:42:50 EST 2003 i686 i686 i386 GNU/Linux Red Hat Linux release 9 (Shrike) Linux bald 2.4.7-10 #1 Thu Sep 6 17:21:28 EDT 2001 i586 unknown Red Hat Linux release 7.2 (Enigma) Tkinter works OOTB on the redhat machines without any configure option. I don't think tcl/tk are on the bsd machine, so I didn't try. (I just ran "make test", didn't enable any resources) Jeff From python at rcn.com Tue Sep 23 23:46:52 2003 From: python at rcn.com (Raymond Hettinger) Date: Tue Sep 23 23:47:12 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <200309240129.h8O1Tqft013440@localhost.localdomain> Message-ID: <000001c3824e$7edbcd60$04af958d@oemcomputer> Tested without exception on WinME. * ran regrtest -u all -r * checked chm files * ran unittests on own applications * exercised IDLE Raymond Hettinger -----Original Message----- From: python-dev-bounces@python.org [mailto:python-dev-bounces@python.org] On Behalf Of Anthony Baxter Sent: Tuesday, September 23, 2003 9:30 PM To: Thomas Heller Cc: jeremy@zope.com; python-dev@python.org Subject: Re: [Python-Dev] 2.3.1 is (almost) a go There's a new release candidate tarball/exe on www.python.org/2.3.1 Feedback welcome - either here or irc.freenode.net #python-dev. Changes since yesterday: Windows build numbers updated HP-UX README note added Misc/RPM notes updated ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From bac at OCF.Berkeley.EDU Wed Sep 24 02:25:34 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Wed Sep 24 02:25:54 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <200309240129.h8O1Tqft013440@localhost.localdomain> References: <200309240129.h8O1Tqft013440@localhost.localdomain> Message-ID: <3F7138DE.7080504@ocf.berkeley.edu> Anthony Baxter wrote: > > There's a new release candidate tarball/exe on www.python.org/2.3.1 > > Feedback welcome - either here or irc.freenode.net #python-dev. > Ran ``./configure; make; make test`` and no failures on OS X 10.2.8 . -Brett From anthony at interlink.com.au Wed Sep 24 03:15:37 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Sep 24 03:17:22 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go Message-ID: <200309240715.h8O7Fbj4024972@localhost.localdomain> Last chance for feedback on python2.3.1 ... :) Draft of announce message at www.python.org/2.3.1/announce.txt -- Anthony Baxter It's never too late to have a happy childhood. From martin at v.loewis.de Wed Sep 24 03:26:12 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Wed Sep 24 03:26:50 2003 Subject: [Python-Dev] Help offered In-Reply-To: <20030924001908.GB28288@panix.com> References: <20030923221433.GA1764@satokar> <20030924001908.GB28288@panix.com> Message-ID: Aahz writes: > Excellent! Currently, it's rather ad-hoc. Just pick a patch that > interests you and make a comment about its suitability. Make sure to > check whether the patch comes with appropriate documentation. [*] If > you don't see activity on the patch within a week or two, feel free to > ask here. Right now isn't the best time to poke people because we're > gearing up for the 2.3.1 release. Also, try to classify the patch somehow, indicating what most likely the problem is for the patch not being reviewed/accepted: - the patch might introduce a questionable feature, and nobody has dared to reject it yet, in order to not offend the submitter. In that case, voice an opinion (and record it in the patches tracker) on whether you are in favour of or opposed to the proposed feature (ideally giving a rationale on why you are). If there are enough voices showing opposition, the submitter might withdraw the patch. If there are enough voices in favour, a core developer might accept it. - the patch might be incomplete. Ping the submitter. If the submitter is incomplete, either complete it yourself, or suggest rejection of the patch. - the patch might be so involved that a quick review does not reveal whether it is correct. I personally find patches involving memory management bugs to be in that category. Review the patch excessively: perform tests, study all possible code paths to uncover cases that have been ignored. When done, record in the patch what kind of review you have done, and either indicate the problems you have found, or recommend acceptance. - the patch might be in an area where the "core" developers have little expertise; you often find that trivial patches are ignored just because everybody thought not to be an expert in the subject. Perform a quick review of the patch to find out whether it meets the formal criteria (completeness), and whether it is perhaps obviously correct or obviously incorrect. If not, come up with a strategy to obtain a decision on the patch: * become an expert yourself in the subject area, and recommend acceptance or rejection afterwards. I have done that many times, and find it both time consuming and rewarding. * find an expert and have it review the patch * ask the submitter to clarify all questionable aspects of the patch, i.e. have him explain the patch to you. If the patch is for a little-used feature (e.g. for an obscure platform), it might be acceptable to incorporate incorrect patches, since nobody will notice, anyway - it might be enough that the submitter believes the patch is correct. I personally find processing of patches more important: it takes often less time per item, brings the project forward, and gives the contributor the satisfaction of having contributed. For bug reports, I would recommend the following classification: - determine whether the report is complete: does the submitter indicate observed behaviour, expected behaviour? If relevant, OS version, Python version? Is a test case included (not mandatory, but desirable)? - determine whether the report is reproducable: try reproducing it with the included test case, or come up with a test case yourself (and list it in the bug report). It might be: * unreproducable: you have the seemingly same setup as the the submitter, but you fail to reproduce it. Ask the submitter for clarification. * fixed: you can reproduce it for the version the submitter has used (or you are reasonably sure that the bug existed in that version), yet the problem disappears with the current version. Indicate that in the report, and recommend to close the report as fixed. * hard to reproduce: you cannot reproduce the bug because you lack some environment. Try to confirm presence of the bug by reviewing the code, or get yourself access to the missing environment. - determine whether the report is invalid: is it really a bug? E.g. is the observed behaviour documented? Is the documentation silent on this aspect, and the observed behaviour surprising to experts? If the documented behaviour deviates from the observed behaviour (and perhaps both from the expected behaviour): which should be the correct one? Once you have classified the report, and found it reports a proper bug, consider asking the submitter for a patch (do so kindly, not demanding). If it is clear that the submitter is not going to provide a patch, try developing one yourself. Regards, Martin From anthony at interlink.com.au Wed Sep 24 04:02:47 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Sep 24 04:04:44 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 Message-ID: <200309240802.h8O82mqF026019@localhost.localdomain> On behalf of the Python development team and the Python community, I'm happy to announce the release of Python 2.3.1 (final). Python 2.3.1 is a pure bug fix release of Python 2.3, released in late July. A number of obscure crash-causing bugs have been fixed, various memory leaks have been squished, but no new features have been added to the language or to the library. For more information on Python 2.3.1, including download links for various platforms, release notes, and known issues, please see: http://www.python.org/2.3.1 Highlights of this new release include: - Bugfixes - The Windows installer now ships with documentation in searchable htmlhelp format, rather than individual HTML files. Highlights of the previous major Python release (2.3) are available from the Python 2.3 page, at http://www.python.org/2.3/highlights.html This release was the work of a number of dedicated python-dev folks who took the time to apply bugfixes back to the 2.3 code. The community owes them many thanks for this work. Enjoy the new release, Anthony Anthony Baxter anthony@python.org Python 2.3.1 Release Manager (on behalf of the entire python-dev team) From anthony at interlink.com.au Wed Sep 24 04:06:14 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Sep 24 04:07:55 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: <200309240802.h8O82mqF026019@localhost.localdomain> Message-ID: <200309240806.h8O86Eti026111@localhost.localdomain> >>> Anthony Baxter wrote > On behalf of the Python development team and the Python community, I'm > happy to announce the release of Python 2.3.1 (final). Can someone who's an SF admin submit a News item on SF? It appears only project admins can do this. From anthony at interlink.com.au Wed Sep 24 04:15:14 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Sep 24 04:17:03 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: <200309240802.h8O82mqF026019@localhost.localdomain> Message-ID: <200309240815.h8O8FIfo026231@localhost.localdomain> Oh yeah - please hold off on the release23-maint branch for a bit, in case Jack wants to cut a 2.3.1 release for the Mac... Jack? Anthony -- Anthony Baxter It's never too late to have a happy childhood. From zeddicus at satokar.com Wed Sep 24 04:22:38 2003 From: zeddicus at satokar.com (Michael Bartl) Date: Wed Sep 24 04:22:42 2003 Subject: [Python-Dev] Help offered In-Reply-To: References: <20030923221433.GA1764@satokar> <20030924001908.GB28288@panix.com> Message-ID: <20030924082238.GA11926@satokar> On Wed, Sep 24, 2003 at 09:26:12AM +0200, Martin v. L?wis wrote: [snip - probably the biggest list of recommendations ever seen :)] First of all thanks for the fast and exhaustive answer. A last question before I'm working on bugs/patches. I sent a patch to fix a mkstemp() bug. The documentation and the implementation had different opinions on how to return the path. What has precedence? Nothing I guess. It's rather a bug in the doc or in the code. Should I contact the original author(s) of tempfile.py, which seem to be Guido and Tim Peters? Best regards, Michael From martin at v.loewis.de Wed Sep 24 05:02:46 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Wed Sep 24 05:03:37 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: <200309240806.h8O86Eti026111@localhost.localdomain> References: <200309240806.h8O86Eti026111@localhost.localdomain> Message-ID: Anthony Baxter writes: > Can someone who's an SF admin submit a News item on SF? It appears only > project admins can do this. Done! Martin From martin at v.loewis.de Wed Sep 24 05:07:53 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Wed Sep 24 05:08:51 2003 Subject: [Python-Dev] Help offered In-Reply-To: <20030924082238.GA11926@satokar> References: <20030923221433.GA1764@satokar> <20030924001908.GB28288@panix.com> <20030924082238.GA11926@satokar> Message-ID: Michael Bartl writes: > What has precedence? Nothing I guess. It's rather a bug in the doc or in > the code. Should I contact the original author(s) of tempfile.py, which > seem to be Guido and Tim Peters? Not necessarily. If your common sense dictates that one is better than the other, just explain it so. If you change the code, consider that existing code may break. Don't worry about the potential breakage, instead, document it - both in Misc/NEWS, and, if breakage is likely, in Doc/whatsnew/whatsnew24.tex. Document the change itself in the documentation, using \versionchanged. [You might notice that I have abstained from actually thinking about the change itself :-] Regards, Martin From theller at python.net Wed Sep 24 06:14:27 2003 From: theller at python.net (Thomas Heller) Date: Wed Sep 24 06:14:34 2003 Subject: [Python-Dev] Editing peps with XEmacs Message-ID: Usually the M-q 'fill-paragraph or region' works nicely to format the current paragraph. It does not work when the paragraph has this form: ___ When the year changes, copyright legends need to be updated in many places, including the README and LICENSE files. because of the '___' marker. Is there any trick I do not know? A REST mode or something like that? Thanks, Thomas From barry at python.org Wed Sep 24 07:53:26 2003 From: barry at python.org (Barry Warsaw) Date: Wed Sep 24 07:53:31 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <200309240715.h8O7Fbj4024972@localhost.localdomain> References: <200309240715.h8O7Fbj4024972@localhost.localdomain> Message-ID: <1064404406.1958.71.camel@anthem> On Wed, 2003-09-24 at 03:15, Anthony Baxter wrote: > Last chance for feedback on python2.3.1 ... :) > > Draft of announce message at www.python.org/2.3.1/announce.txt I pulled down rc2 and ran the tests randomly on RH9, with "-u all -r". I see that test_mimetypes failed unexpectedly: test test_mimetypes failed -- Traceback (most recent call last): File "/tmp/Python-2.3.1/Lib/test/test_mimetypes.py", line 52, in test_guess_all_types eq(all, ['.bat', '.c', '.h', '.ksh', '.pl', '.txt']) File "/tmp/Python-2.3.1/Lib/unittest.py", line 302, in failUnlessEqual raise self.failureException, \ AssertionError: ['.asc', '.bat', '.c', '.h', '.ksh', '.pl', '.txt'] != ['.bat', '.c', '.h', '.ksh', '.pl', '.txt'] test_bsddb3 produced two locker killed messages, which didn't happen last time I tested from the HEAD. Then again I wasn't running the tests with -r last time either, but I also understand that these tracebacks can happen and don't cause the tests to fail. In the first run, test_queue also failed, with this output: test_queue test test_queue failed -- blocking function '>' appeared not to block Of these, it seems like the mimetypes one should be fixed before the release. I doubt I'll have time today to debug it, but I could probably arrange to run another test on RH9 if need be. -Barry From barry at python.org Wed Sep 24 07:55:13 2003 From: barry at python.org (Barry Warsaw) Date: Wed Sep 24 07:55:16 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <1064404406.1958.71.camel@anthem> References: <200309240715.h8O7Fbj4024972@localhost.localdomain> <1064404406.1958.71.camel@anthem> Message-ID: <1064404512.1958.73.camel@anthem> On Wed, 2003-09-24 at 07:53, Barry Warsaw wrote: > Of these, it seems like the mimetypes one should be fixed before the > release. I doubt I'll have time today to debug it, but I could probably > arrange to run another test on RH9 if need be. Then again, I guess there's always 2.3.2... stinkin'-timezones-ly y'rs, -Barry From barry at python.org Wed Sep 24 07:59:43 2003 From: barry at python.org (Barry Warsaw) Date: Wed Sep 24 07:59:46 2003 Subject: [Python-Dev] Editing peps with XEmacs In-Reply-To: References: Message-ID: <1064404783.1958.80.camel@anthem> On Wed, 2003-09-24 at 06:14, Thomas Heller wrote: > Usually the M-q 'fill-paragraph or region' works nicely to format the > current paragraph. It does not work when the paragraph has this form: > > ___ When the year changes, copyright legends need to be updated in > many places, including the README and LICENSE files. > > because of the '___' marker. > Is there any trick I do not know? A REST mode or something like that? filladapt! -Barry From jepler at unpythonic.net Wed Sep 24 08:10:51 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Wed Sep 24 08:11:01 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <1064404406.1958.71.camel@anthem> References: <200309240715.h8O7Fbj4024972@localhost.localdomain> <1064404406.1958.71.camel@anthem> Message-ID: <20030924121045.GB1590@unpythonic.net> On Wed, Sep 24, 2003 at 07:53:26AM -0400, Barry Warsaw wrote: > I pulled down rc2 and ran the tests randomly on RH9, with "-u all -r". > I see that test_mimetypes failed unexpectedly: > > test test_mimetypes failed -- Traceback (most recent call last): > File "/tmp/Python-2.3.1/Lib/test/test_mimetypes.py", line 52, in > test_guess_all_types > eq(all, ['.bat', '.c', '.h', '.ksh', '.pl', '.txt']) > File "/tmp/Python-2.3.1/Lib/unittest.py", line 302, in failUnlessEqual > raise self.failureException, \ > AssertionError: ['.asc', '.bat', '.c', '.h', '.ksh', '.pl', '.txt'] != > ['.bat', '.c', '.h', '.ksh', '.pl', '.txt'] This order fails: [jepler@parrot Lib]$ ../python test/regrtest.py test_urllib2 test_mimetypes This order doesn't: [jepler@parrot Lib]$ ../python test/regrtest.py test_mimetypes test_urllib2 urllib2 calls mimetypes.guess_type without zeroing out mimetypes.knownfiles. I think this is the cause. Adding this patch makes both orders pass. Perhaps a better way to fix it would be to test for a superset in test_mimetypes. assert set(all) >= set('.bat', ...) --- test_urllib2.py.orig 2003-09-24 07:08:28.000000000 -0500 +++ test_urllib2.py 2003-09-24 07:07:31.000000000 -0500 @@ -11,6 +11,10 @@ else: verify(0) +import mimetypes +mimetypes.knownfiles = [] +mimetypes.inited = False + # XXX Name hacking to get this to work on Windows. fname = os.path.abspath(urllib2.__file__).replace('\\', '/') if fname[1:2] == ":": From jepler at unpythonic.net Wed Sep 24 08:19:20 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Wed Sep 24 08:19:30 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <20030924121045.GB1590@unpythonic.net> References: <200309240715.h8O7Fbj4024972@localhost.localdomain> <1064404406.1958.71.camel@anthem> <20030924121045.GB1590@unpythonic.net> Message-ID: <20030924121920.GC1590@unpythonic.net> On Wed, Sep 24, 2003 at 07:10:51AM -0500, Jeff Epler wrote: > Perhaps a better way to fix it would be to test for a superset in > test_mimetypes. assert set(all) >= set('.bat', ...) The general idea is implemented in this patch: --- test_mimetypes.py.orig 2003-09-24 07:11:12.000000000 -0500 +++ test_mimetypes.py 2003-09-24 07:18:58.000000000 -0500 @@ -1,18 +1,21 @@ import mimetypes import StringIO import unittest +import sets from test import test_support -# Tell it we don't know about external files: -mimetypes.knownfiles = [] -mimetypes.inited = False - - class MimeTypesTestCase(unittest.TestCase): def setUp(self): self.db = mimetypes.MimeTypes() + def assertSuperset(self, first, second, msg=None): + first = sets.Set(first) + second = sets.Set(second) + if not first >= second: + raise self.failureException, \ + (msg or '%r >= %r' % (first, second)) + def test_default_data(self): eq = self.assertEqual eq(self.db.guess_type("foo.html"), ("text/html", None)) @@ -45,15 +48,16 @@ eq(self.db.guess_extension('image/jpg', strict=False), '.jpg') def test_guess_all_types(self): + ss = self.assertSuperset eq = self.assertEqual # First try strict all = self.db.guess_all_extensions('text/plain', strict=True) all.sort() - eq(all, ['.bat', '.c', '.h', '.ksh', '.pl', '.txt']) + ss(all, ['.bat', '.c', '.h', '.ksh', '.pl', '.txt']) # And now non-strict all = self.db.guess_all_extensions('image/jpg', strict=False) all.sort() - eq(all, ['.jpg']) + ss(all, ['.jpg']) # And now for no hits all = self.db.guess_all_extensions('image/jpg', strict=True) eq(all, []) From anthony at interlink.com.au Wed Sep 24 08:33:38 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Sep 24 08:35:24 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <1064404406.1958.71.camel@anthem> Message-ID: <200309241233.h8OCXcbg028668@localhost.localdomain> >>> Barry Warsaw wrote > Of these, it seems like the mimetypes one should be fixed before the > release. I doubt I'll have time today to debug it, but I could probably > arrange to run another test on RH9 if need be. D'oh! Since it's only a failure of the test code (not the module itself) I'll add a note to the bugs page in the test section. Jeff Epler's fix seems appropriate. There's always 2.3.2 - since 2.4 is still a fair way off, it seems likely that a 2.3.2 sometime around January would be appropriate. Of course, if the bsddb3 weirdness does actually turn out to be something more serious, it might be sooner. who-uses-bsddb-anyway-just-those-zope-guys-, Anthony -- Anthony Baxter It's never too late to have a happy childhood. From Paul.Moore at atosorigin.com Wed Sep 24 08:41:42 2003 From: Paul.Moore at atosorigin.com (Moore, Paul) Date: Wed Sep 24 08:42:42 2003 Subject: [Python-Dev] 2.3.1 - minor nit in HTML Help docs Message-ID: <16E1010E4581B049ABC51D4975CEDB8802C09792@UKDCX001.uk.int.atosorigin.com> The HTML Help documentation in the Windows installer has "Release 2.4a0" in the footer for each page. I'm guessing that this is a fairly minor tweak that might need adding to the list of tasks for preparing a release... Paul From theller at python.net Wed Sep 24 09:02:59 2003 From: theller at python.net (Thomas Heller) Date: Wed Sep 24 09:03:04 2003 Subject: [Python-Dev] 2.3.1 - minor nit in HTML Help docs In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8802C09792@UKDCX001.uk.int.atosorigin.com> (Paul Moore's message of "Wed, 24 Sep 2003 13:41:42 +0100") References: <16E1010E4581B049ABC51D4975CEDB8802C09792@UKDCX001.uk.int.atosorigin.com> Message-ID: <8yoeibjw.fsf@python.net> "Moore, Paul" writes: > The HTML Help documentation in the Windows installer has > "Release 2.4a0" in the footer for each page. The same is true for all the html docs online at , which isn't surprising. Thomas From anthony at interlink.com.au Wed Sep 24 09:11:54 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Sep 24 09:13:53 2003 Subject: [Python-Dev] 2.3.1 - minor nit in HTML Help docs In-Reply-To: <8yoeibjw.fsf@python.net> Message-ID: <200309241311.h8ODBsS2029657@localhost.localdomain> >>> Thomas Heller wrote > "Moore, Paul" writes: > > > The HTML Help documentation in the Windows installer has > > "Release 2.4a0" in the footer for each page. > > The same is true for all the html docs online at > , which isn't surprising. This would be part of the "Fred does Doc magic" section of the release PEPs ;) Anthony -- Anthony Baxter It's never too late to have a happy childhood. From barry at python.org Wed Sep 24 09:20:27 2003 From: barry at python.org (Barry Warsaw) Date: Wed Sep 24 09:20:33 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <200309241233.h8OCXcbg028668@localhost.localdomain> References: <200309241233.h8OCXcbg028668@localhost.localdomain> Message-ID: <1064409627.1958.90.camel@anthem> On Wed, 2003-09-24 at 08:33, Anthony Baxter wrote: > D'oh! Since it's only a failure of the test code (not the module itself) > I'll add a note to the bugs page in the test section. +1 > Jeff Epler's fix seems appropriate. There's always 2.3.2 - since 2.4 is still > a fair way off, it seems likely that a 2.3.2 sometime around January would be > appropriate. Of course, if the bsddb3 weirdness does actually turn out to be > something more serious, it might be sooner. +1 > who-uses-bsddb-anyway-just-those-zope-guys-, Just a bunch of anti-spam weirdos. -Barry From jeremy at zope.com Wed Sep 24 09:39:48 2003 From: jeremy at zope.com (Jeremy Hylton) Date: Wed Sep 24 09:40:44 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <1064404406.1958.71.camel@anthem> References: <200309240715.h8O7Fbj4024972@localhost.localdomain> <1064404406.1958.71.camel@anthem> Message-ID: <1064410788.2072.100.camel@localhost.localdomain> On Wed, 2003-09-24 at 07:53, Barry Warsaw wrote: > On Wed, 2003-09-24 at 03:15, Anthony Baxter wrote: > > Last chance for feedback on python2.3.1 ... :) > test test_mimetypes failed -- Traceback (most recent call last): > File "/tmp/Python-2.3.1/Lib/test/test_mimetypes.py", line 52, in > test_guess_all_types > eq(all, ['.bat', '.c', '.h', '.ksh', '.pl', '.txt']) > File "/tmp/Python-2.3.1/Lib/unittest.py", line 302, in failUnlessEqual > raise self.failureException, \ > AssertionError: ['.asc', '.bat', '.c', '.h', '.ksh', '.pl', '.txt'] != > ['.bat', '.c', '.h', '.ksh', '.pl', '.txt'] > I've seen test_mimetypes fail frequently, depending on what order the tests are run. We worked on this a little for 2.3c1, but I didn't have the patience to stick with it. The problem is that the mimetypes module initializes some global variables when it is first used. test_mimetypes avoids initializing the module, because it can't guess what extensions will be known on a particular platform. But some of the other tests *use* mimetypes, so they cause it to be initialized. > In the first run, test_queue also failed, with this output: > > test_queue > test test_queue failed -- blocking function ' >' appeared not to block I see this from time-to-time, too. The tests are timed based and sometimes fail when there's not enough CPU time during the test interval. Jeremy From skip at pobox.com Wed Sep 24 10:54:42 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Sep 24 10:55:02 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <1064409627.1958.90.camel@anthem> References: <200309241233.h8OCXcbg028668@localhost.localdomain> <1064409627.1958.90.camel@anthem> Message-ID: <16241.45106.296743.387428@montanaro.dyndns.org> >> who-uses-bsddb-anyway-just-those-zope-guys-, Barry> Just a bunch of anti-spam weirdos. Many are deserting what at times appears to be a sinking bsddb ship in favor of pickles... For long-running SpamBayes apps like pop3proxy and the Outlook plugin, pickles make sense if shared access to the scoring database is not needed. Skip From python-kbutler at sabaydi.com Wed Sep 24 11:12:38 2003 From: python-kbutler at sabaydi.com (Kevin J. Butler) Date: Wed Sep 24 11:12:52 2003 Subject: [Python-Dev] Integration with Eclipse In-Reply-To: <16240.36893.520437.974584@nick.uklinux.net> References: <3F6B11BA.6000008@sabaydi.com> <16235.18128.422738.929390@nick.uklinux.net> <3F6B533E.5000403@sabaydi.com> <16236.28754.105629.565153@nick.uklinux.net> <3F6C7B09.5060608@sabaydi.com> <16236.51140.284372.526221@nick.uklinux.net> <3F6CF076.30409@ocf.berkeley.edu> <3F6D1AB7.1050608@sabaydi.com> <16240.36893.520437.974584@nick.uklinux.net> Message-ID: <3F71B466.4040502@sabaydi.com> Nick Roberts wrote: >Kevin J. Butler writes: > > ... > > > > I consider this more pdb-related than emacs-specific - > > I'd expect that integration with, say, Eclipse would benefit from this > > enhancement as well, so I've been hoping to get some feedback from the > > list... > > > > BTW, I've submitted the patch, it is 809887: > > http://sourceforge.net/tracker/index.php?func=detail&aid=809887&group_id=5470&atid=305470 > > > > kb > >The syntax of responses from pdb commands suggests to me that there is a fair >amount of GDB in the underlying code. > Nope - similarity of interface only. pdb is pure python (and thus works in the Java implementation 'jython' as well), and I haven't heard that GDB has been incorporating python (yet ;-) ). kb From barry at python.org Wed Sep 24 11:32:57 2003 From: barry at python.org (Barry Warsaw) Date: Wed Sep 24 11:33:07 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <16241.45106.296743.387428@montanaro.dyndns.org> References: <200309241233.h8OCXcbg028668@localhost.localdomain> <1064409627.1958.90.camel@anthem> <16241.45106.296743.387428@montanaro.dyndns.org> Message-ID: <1064417577.1082.0.camel@geddy> On Wed, 2003-09-24 at 10:54, Skip Montanaro wrote: > what at times appears to be a sinking bsddb ship Dang. -Barry From anthony at interlink.com.au Wed Sep 24 11:50:45 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Sep 24 11:53:27 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <16241.45106.296743.387428@montanaro.dyndns.org> Message-ID: <200309241550.h8OFojlx031706@localhost.localdomain> >>> Skip Montanaro wrote > Many are deserting what at times appears to be a sinking bsddb ship in favor > of pickles... For long-running SpamBayes apps like pop3proxy and the > Outlook plugin, pickles make sense if shared access to the scoring database > is not needed. :-( I haven't been following the spambayes lists too closely. Are there concrete problems with bsddb that are cropping up, or just a general wariness of it? If there _is_ a problem with bsddb, it needs to be addressed. Too many things depend on it. -- Anthony Baxter It's never too late to have a happy childhood. From tim.one at comcast.net Wed Sep 24 11:54:05 2003 From: tim.one at comcast.net (Tim Peters) Date: Wed Sep 24 11:54:14 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <1064404406.1958.71.camel@anthem> Message-ID: [Barry Warsaw] > ... > In the first run, test_queue also failed, with this output: > > test_queue > test test_queue failed -- blocking function ' of >' appeared not to block If there isn't already a bug report open on this, could you file one? It's a timing thing, and the test needs to be rewritten to avoid it (that doesn't look easy, alas), or at least the message has to be changed to say up front that sometimes it's just going to happen. From anthony at interlink.com.au Wed Sep 24 11:57:53 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Sep 24 11:59:39 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: Message-ID: <200309241557.h8OFvrNL031813@localhost.localdomain> >>> "Tim Peters" wrote > If there isn't already a bug report open on this, could you file one? It's > a timing thing, and the test needs to be rewritten to avoid it (that doesn't > look easy, alas), or at least the message has to be changed to say up front > that sometimes it's just going to happen. If no-one does it soon, I plan to file a bunch of bug reports against the various entries on 2.3.1's bugs.html page tomorrow. -- Anthony Baxter It's never too late to have a happy childhood. From tim at zope.com Wed Sep 24 12:01:04 2003 From: tim at zope.com (Tim Peters) Date: Wed Sep 24 12:09:02 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <200309241550.h8OFojlx031706@localhost.localdomain> Message-ID: Adding spambayes-dev in case anyone there has more reliable info. [Anthony Baxter] > I haven't been following the spambayes lists too closely. Are there > concrete problems with bsddb that are cropping up, or just a general > wariness of it? > > If there _is_ a problem with bsddb, it needs to be addressed. Too > many things depend on it. Reports of database corruption are common in spambayes. At least one knowledgable tester reported his problems went away after moving to a recent Sleepycat release (4.1.25, IIRC). I'm not sure any reports have come from people using the Outlook client (I was happy using a dict, but switched all 3 of my Outlook classifiers to use Berkeley instead, *hoping* to provoke a problem -- but no luck yet). They seem to come from non-Outlook people using Berkeley for the message info database. Richie got a whittled down threaded test that fails on Windows and Linux, and there's already a (Python) bug report open on that; it's not thought to be relevant to how spambayes uses Berkeley, though. From skip at pobox.com Wed Sep 24 12:41:28 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Sep 24 12:45:29 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <200309241550.h8OFojlx031706@localhost.localdomain> References: <16241.45106.296743.387428@montanaro.dyndns.org> <200309241550.h8OFojlx031706@localhost.localdomain> Message-ID: <16241.51512.389962.16881@montanaro.dyndns.org> >> Many are deserting what at times appears to be a sinking bsddb ship >> in favor of pickles... For long-running SpamBayes apps like >> pop3proxy and the Outlook plugin, pickles make sense if shared access >> to the scoring database is not needed. Anthony> :-( Anthony> I haven't been following the spambayes lists too closely. Are Anthony> there concrete problems with bsddb that are cropping up, or Anthony> just a general wariness of it? It's a combination of things. First and foremost, using the pickle storage is much faster assuming your database is not too huge and you have a long-running spambayes app. Second, people still use pre-2.3 Windows versions of Python to run their spambayes apps. By default, these people get dumbdbm by default. That's a disaster waiting to happen because of storage size and speed concerns. There were also bugs in dumbdbm (which have since been fixed). Anthony> If there _is_ a problem with bsddb, it needs to be Anthony> addressed. Too many things depend on it. I think it's more that bsddb is an innocent bystander in a drive-by shooting, though some folks have been seeing DB_RUNRECOVERY (sp?) errors. The anydbm-style access to Berkeley databases doesn't provide the necessary locking for multiple process (or multiple thread?) access. I agree if there are problems they should be fixed. One step in the right direction would probably be to create a default DBEnv for use by the bsddb.*open() compatibility calls. Also, the docs should mention threading issues for people using the more featureful API so they don't unwittingly shoot themselves in the foot. (Maybe DBEnv=None should not be allowed in open calls if threads are enabled?) Skip From skip at pobox.com Wed Sep 24 13:15:14 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Sep 24 13:32:29 2003 Subject: [Python-Dev] Python Glossary Message-ID: <16241.53538.40615.455582@montanaro.dyndns.org> Thanks to people who submitted content for the Python Glossary: http://manatee.mojam.com/python-glossary I just completed a first pass at folding it into the Python tutorial. It's now a separate appendix to that document. It will take awhile for the changes to show up on the website and in distributions, but if you have named CVS access to the Python distribution, you can get to it now. People with only anonymous CVS access or no CVS access at all will have to wait awhile for SourceForge to update the server. After a day or so it should be available here: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Doc/tut/ I would appreciate it if people could proofread it (if you're anxious to get to work, mail me and I'll send you a copy of the glossary.tex file). I made some changes to the definitions, but was focused more on getting the content into the tutorial than on wordsmithing. As a side-effect, the tutorial now also has an index. All but one or two of the entries it contains are from the glossary. The entire tutorial needs to be "indexified". If you would like to help with that, please feel free. In any case, any changes should be submitted as a documentation bug or patch (if you have some proposed context diffs) and assigned to me (SF username: montanaro). Please don't feel shy about adding new entries to the existing Wiki-based glossary either. I will notice updates and migrate new or corrected terms over as I have time. I don't plan to dismantle it, at least not for the forseeable future. At some point I may migrate it to the Python.org Wiki though. Skip From DavidA at ActiveState.com Wed Sep 24 13:32:36 2003 From: DavidA at ActiveState.com (David Ascher) Date: Wed Sep 24 13:54:15 2003 Subject: [Python-Dev] Announcement: ActiveState acquired by Sophos Message-ID: <3F71D534.3010101@ActiveState.com> To the Python community -- As some of you have heard by now, ActiveState was acquired yesterday by Sophos Plc, a UK-based global anti-virus company, as described in this press release: http://www.activestate.com/Corporate/Communications/Releases/Press1064364490.html and in this open letter to ActiveState customers: http://www.activestate.com/corporate/letter/. I figure that many of you who know about ActiveState's background in the open source community probably have questions regarding what this means for ActiveState's future, and the future of Python at ActiveState. In short, there is no downside, only great opportunities. What will not change: - Our line of tools, language distributions and enterprise support for open source programming languages will continue to be developed and evolve with customer needs. - Our involvement in the open source language communities will continue as before. - All of the ActiveState staff is retained in the acquisition. - The Vancouver, BC office is poised to expand, not shrink. - Email addresses, phone numbers, etc. will continue to function as before. What will change: - ActiveState will now have a broader global impact and access to the resources of a larger company with a great deal of experience helping businesses secure their networks. - Our PureMessage anti-spam product will be integrated with the Sophos MailMonitor product and be sold as Sophos PureMessage, while the rest of our product line will still be sold under the ActiveState banner. - ActiveState is now a wholly-owned subsidiary of Sophos, with a focused mission to providing tools and services for users of open source programming languages. I want to point out to this audience in particular that the acquisition of ActiveState is a success story for open source programming technologies. It is in part thanks to our use of programming languages like Perl, Python and Tcl _and_ to our involvement in the programming communities around these languages that we've been able to quickly build and sell enterprise-grade software such as PureMessage, as well as very successful programming tools like Komodo, the Perl Dev Kit, etc. I'm glad to say that Sophos appreciates this and is looking to expand the use of these technologies in the larger company, not shrink it. Score one for the good guys! While I have to get back to work (on the usual things, like adding Python features to Komodo, help plan the next release of ActivePython, help plan for PyCon 2004, iron out legal issues for the PSF and a few other things like that), I'll be monitoring these lists, and welcome email questions as well -- send them to me at DavidA@ActiveState.com. Cheers, and see you all at PyCon 2004! -- David Ascher ActiveState, a division of Sophos From Jack.Jansen at oratrix.com Wed Sep 24 08:43:09 2003 From: Jack.Jansen at oratrix.com (Jack Jansen) Date: Wed Sep 24 15:32:13 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: <200309240815.h8O8FIfo026231@localhost.localdomain> Message-ID: On woensdag, sep 24, 2003, at 10:15 Europe/Amsterdam, Anthony Baxter wrote: > > Oh yeah - please hold off on the release23-maint branch for a bit, > in case Jack wants to cut a 2.3.1 release for the Mac... Jack? I don't know whether I'll find the time, give me until the end of the week to decide, please. And off on a tangent: I think we need a python-dev-announce mailing list or something of the sort. I've been busy with Real Work the last month, and I only skim python-dev when I have time, so today is the first time I hear about 2.3.1 (and, being on a train without internet-access right now I can't even test it:-). I'm not happy with this. For one thing, I haven't seen any indication that anyone tried a framework build on MacOSX, and how such a build interacts with the pre-installed 2.3 on Panther. Even though I wouldn't have had time to do any fixes for MacOSX that are on my plate I would at least have tested that 2.3.1 doesn't break anything... I wouldn't mind a checklist of things that need to be tested (plus preferred people responsible for them) before a release goes out. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From barry at python.org Wed Sep 24 15:47:38 2003 From: barry at python.org (Barry Warsaw) Date: Wed Sep 24 15:47:40 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: References: Message-ID: <1064432857.1082.20.camel@geddy> On Wed, 2003-09-24 at 08:43, Jack Jansen wrote: > I wouldn't mind a checklist of things that need to be tested (plus > preferred > people responsible for them) before a release goes out. IMO, this needs to be added to PEP 102 and there needs to be a list of people who will give a go, no-go on a new release. There needs to be timeouts though because we're all way too busy and it may just be that a release needs to go out before a person on the critical path has a chance to sign off on it. -Barry From zeddicus at satokar.com Wed Sep 24 16:42:13 2003 From: zeddicus at satokar.com (Michael Bartl) Date: Wed Sep 24 16:42:18 2003 Subject: [Python-Dev] Python in ATC (Air Traffic control) Message-ID: <20030924204212.GA12794@satokar> Hi all together! Just to stop repeating myself I thought I'd post this to the python-dev list. In a foregoing post to the python-dev list I mentioned the use of python in the field of Air Traffic control. It seems that many people are interested in a description of where exactly python is used in this high-demanding field, may it be for marketing or personal reasons. I'm going to write a "python success story" about it, if my boss has no problem with that (which I assume). I can't give you any other information except to wait until the beginning of next week, when I will probably have finished the content :o) regards, Michael Bartl From fperez at colorado.edu Wed Sep 24 16:47:50 2003 From: fperez at colorado.edu (Fernando Perez) Date: Wed Sep 24 16:47:54 2003 Subject: [Python-Dev] Hi, and a question about filename completion via readline in Python 2.3 Message-ID: <3F7202F6.9050508@colorado.edu> Hi all, since this is my first post here, I figured I'd briefly introduce myself. I'm the lead developer of ipython (http://ipython.scipy.org), a shell for interactive python work. I've been a fan (and big advocate) of python over the last 2 years, mainly for scientific computing. For those interested, you can find a brief overview of ipython at: http://ipython.scipy.org/misc/ipython_scipy03.pdf Besides thanking the python team for this fantastic language, I have a question concerning some changes apparently made to the C part of readline support for python 2.3. If I understood correctly this thread: http://mail.python.org/pipermail/python-list/2002-February/085063.html as of python 2.3, readline will NOT default anymore to providing filename completions when nothing matches in the available namespaces. There is mention in that thread of making it a configurable parameter, but I failed to find any specifics on how to do that. While perhaps I'm missing something, so far (after getting this reported as an ipython bug from 2.3 users) I've failed to be able to recover filename completion under 2.3. While I'm sure you found good reasons for this change, under some environments it turns out to be really quite nasty. One of the strengths of ipython is its tight integration with the underlying system environment, and its users rely constantly on filename/directory completion for navigating their filesystem as they work. I personally (and many others) use ipython as my main python shell, and I fell it provides a superior working environment to anything else I've seen out there (especially if you need to combine python work with manipulating underlying files). So ultimately my concerns are: - is there currently a way to re-enable readline filename completion, as suggested in that thread? - if not, can such a change be made? I hope if some of you test ipython, you'll find out that actually that kind of integration with the filesystem can make a very strong argument for python-based shells as fantastic tools for many data-driven environments. I have already knowledge of several projects which are either using or considering ipython as the underlying shell for scientific data analysis environments, and this change will be a major setback for their usability. Again, many thanks for all your hard work on python. Best regards, Fernando. From zeddicus at satokar.com Wed Sep 24 16:47:35 2003 From: zeddicus at satokar.com (Michael Bartl) Date: Wed Sep 24 16:52:20 2003 Subject: [Python-Dev] PEP on patch processing Message-ID: <20030924204734.GB12794@satokar> Hi! Martin v. L?wis has previously posted a guide on how to successfully process patches on this mailinglist. I just wanted to ask if there is interest in a PEP about that topic, because I couldn't find any relevant information on python.org. If you do not agree with me I still would recommend to put it on the developers pages on python.org. ... just an idea regards, Michael Bartl From aahz at pythoncraft.com Wed Sep 24 16:57:55 2003 From: aahz at pythoncraft.com (Aahz) Date: Wed Sep 24 16:57:58 2003 Subject: [Python-Dev] PEP on patch processing In-Reply-To: <20030924204734.GB12794@satokar> References: <20030924204734.GB12794@satokar> Message-ID: <20030924205755.GA26474@panix.com> On Wed, Sep 24, 2003, Michael Bartl wrote: > > Martin v. L?wis has previously posted a guide on how to successfully > process patches on this mailinglist. > > I just wanted to ask if there is interest in a PEP about that topic, > because I couldn't find any relevant information on python.org. Given the existence of PEP 3, this sounds like an excellent idea! -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "It is easier to optimize correct code than to correct optimized code." --Bill Harlan From jason at mastaler.com Wed Sep 24 18:05:40 2003 From: jason at mastaler.com (Jason R. Mastaler) Date: Wed Sep 24 18:05:55 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 References: <200309240802.h8O82mqF026019@localhost.localdomain> Message-ID: Oops. See python.org/sf/812007 Python 2.3.1 (#1, Sep 24 2003, 15:55:24) [GCC 3.2 20020903 (Red Hat Linux 8.0 3.2-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from os import fsync Traceback (most recent call last): File "", line 1, in ? ImportError: cannot import name fsync >>> From anthony at interlink.com.au Wed Sep 24 23:20:10 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Sep 24 23:21:58 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: Message-ID: <200309250320.h8P3KAxs007608@localhost.localdomain> Mea maxima culpa. I'll add a new section to the start of PEP101 and PEP102 about communicating with the appropriate people directly. We had a bunch of testers try out the release candidates on various Mac OS X releases - they reported no breakages. There's also been few/no checkins to the Mac-specific chunks of the tree for the release23-maint branch. I'm happy to store up checkins for the 23 branch for a couple of weeks to give you time to do a release if you so wish (consider it penance for my forgetting to email you). -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Wed Sep 24 23:21:43 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Sep 24 23:23:27 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: <1064432857.1082.20.camel@geddy> Message-ID: <200309250321.h8P3Lh7Z007641@localhost.localdomain> >>> Barry Warsaw wrote > IMO, this needs to be added to PEP 102 and there needs to be a list of > people who will give a go, no-go on a new release. There needs to be > timeouts though because we're all way too busy and it may just be that a > release needs to go out before a person on the critical path has a > chance to sign off on it. I'm re-doing PEP 101 and 102 to change the various names to roles. I'll add a section to the top saying "You need to get a person signed up and agreed for each of the following roles. The people who have done each role in the past are..." Anthony -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Wed Sep 24 23:40:23 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Sep 24 23:42:04 2003 Subject: [Python-Dev] a small data point Message-ID: <200309250340.h8P3eNrI008096@localhost.localdomain> Of the people who bothered to email or otherwise contact me about 2.3.1, the overwhelming response has been "thank you for not adding any new features". just a small data point for the features-vs-stability issue. Anthony From barry at python.org Thu Sep 25 00:35:20 2003 From: barry at python.org (Barry Warsaw) Date: Thu Sep 25 00:35:28 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: <200309250321.h8P3Lh7Z007641@localhost.localdomain> References: <200309250321.h8P3Lh7Z007641@localhost.localdomain> Message-ID: <1064464520.12343.42.camel@anthem> On Wed, 2003-09-24 at 23:21, Anthony Baxter wrote: > I'm re-doing PEP 101 and 102 to change the various names to roles. I'll add > a section to the top saying "You need to get a person signed up and agreed > for each of the following roles. The people who have done each role in the > past are..." +1 -B From fdrake at acm.org Thu Sep 25 01:52:00 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Sep 25 01:52:14 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: <200309250321.h8P3Lh7Z007641@localhost.localdomain> References: <1064432857.1082.20.camel@geddy> <200309250321.h8P3Lh7Z007641@localhost.localdomain> Message-ID: <16242.33409.339583.936076@grendel.zope.com> Anthony Baxter writes: > I'm re-doing PEP 101 and 102 to change the various names to roles. I'll add > a section to the top saying "You need to get a person signed up and agreed > for each of the following roles. The people who have done each role in the > past are..." Make sure you only list the people willing to do it again or at least discuss the role with new volunteers. ;-) -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From fdrake at acm.org Thu Sep 25 02:12:30 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Sep 25 02:12:42 2003 Subject: [Python-Dev] 2.3.1 - minor nit in HTML Help docs In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8802C09792@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB8802C09792@UKDCX001.uk.int.atosorigin.com> Message-ID: <16242.34638.614846.862608@grendel.zope.com> Moore, Paul writes: > The HTML Help documentation in the Windows installer has > "Release 2.4a0" in the footer for each page. I'm guessing > that this is a fairly minor tweak that might need adding > to the list of tasks for preparing a release... I'm pretty sure I know how this happened, and is actually a (fairly minor) accident of how I built the docs this time combined with the slow growth of the formatting tools from being a pile of scripts very specific to building the Python documentation to tools that can format the types of documents used in the Python documentation. What I did this time that was new is this: I used the formatting tools from the head of the trunk instead of from the branch. They picked up all the right content files, but used the wrong "shared" files from the Doc/texinputs/ directory. (The trunk files were used instead of the branch files.) This is a strong indication that the conflation of content and style files in Doc/texinputs/ is a bad thing; style should be provided by the tool, and additional directories of content should be configurable for the specific content. For TeX-based tools, additional directories of TeX source files are specified by the TEXINPUTS environment variable, but the Doc/tools/mkhowto script simply clobbers that currently. There are a bunch of conventions associated with path configuration in TeX (per the kpathsea library), but implementing this for mkhowto shouldn't take long. I expect to be able to handle this over the next few days, after which I'll generate new documentation packages for Python 2.3.1. The fixes will be made on the trunk (since that's where the formatting tools are being maintained); some changes to the Makefile and the organization of the files in Doc/texinputs/ on the release23-maint branch will be needed; not sure when these should be committed (in particular, before/after a MacPython release). Once I've got this done, I'll propose a more managable approach to dealing with maintenance of the formatting tools. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From mwh at python.net Thu Sep 25 05:54:50 2003 From: mwh at python.net (Michael Hudson) Date: Thu Sep 25 05:54:12 2003 Subject: [Python-Dev] Hi, and a question about filename completion via readline in Python 2.3 In-Reply-To: <3F7202F6.9050508@colorado.edu> (Fernando Perez's message of "Wed, 24 Sep 2003 14:47:50 -0600") References: <3F7202F6.9050508@colorado.edu> Message-ID: <2mbrt942hh.fsf@starship.python.net> Fernando Perez writes: > Besides thanking the python team for this fantastic language, I have a > question concerning some changes apparently made to the C part of > readline support for python 2.3. If I understood correctly this > thread: > > http://mail.python.org/pipermail/python-list/2002-February/085063.html > > as of python 2.3, readline will NOT default anymore to providing > filename completions when nothing matches in the available namespaces. > There is mention in that thread of making it a configurable parameter, > but I failed to find any specifics on how to do that. > > While perhaps I'm missing something, so far (after getting this > reported as an ipython bug from 2.3 users) I've failed to be able to > recover filename completion under 2.3. > > While I'm sure you found good reasons for this change, under some > environments it turns out to be really quite nasty. One of the > strengths of ipython is its tight integration with the underlying > system environment, and its users rely constantly on > filename/directory completion for navigating their filesystem as they > work. I personally (and many others) use ipython as my main python > shell, and I fell it provides a superior working environment to > anything else I've seen out there (especially if you need to combine > python work with manipulating underlying files). Bet pyrepl still canes it for multi-line work :-) > So ultimately my concerns are: > > - is there currently a way to re-enable readline filename completion, > as suggested in that thread? Yes. Do it yourself, in your completer function. Is it really that hard? I would have thought os.path + glob would make it a five line or so excercise. > - if not, can such a change be made? Well, I'm not going to . Feel free to submit a patch. Cheers, mwh -- You owe the Oracle a star-spangled dunce cap. -- Internet Oracularity Internet Oracularity #1299-08 From anthony at interlink.com.au Thu Sep 25 02:17:41 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Sep 25 06:04:04 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: <16242.33409.339583.936076@grendel.zope.com> Message-ID: <200309250617.h8P6HgsA010531@localhost.localdomain> >>> "Fred L. Drake, Jr." wrote > Make sure you only list the people willing to do it again or at least > discuss the role with new volunteers. ;-) Bah. That's no fun. I'm-sure-Barry-said-he'd-build-the-windows-installer-, Anthony From mwh at python.net Thu Sep 25 06:20:52 2003 From: mwh at python.net (Michael Hudson) Date: Thu Sep 25 06:20:14 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: <200309250320.h8P3KAxs007608@localhost.localdomain> (Anthony Baxter's message of "Thu, 25 Sep 2003 13:20:10 +1000") References: <200309250320.h8P3KAxs007608@localhost.localdomain> Message-ID: <2m7k3x41a3.fsf@starship.python.net> Anthony Baxter writes: > Mea maxima culpa. I'll add a new section to the start of PEP101 and PEP102 > about communicating with the appropriate people directly. > > We had a bunch of testers try out the release candidates on various Mac OS X > releases - they reported no breakages. There's also been few/no checkins > to the Mac-specific chunks of the tree for the release23-maint branch. > > I'm happy to store up checkins for the 23 branch for a couple of weeks to > give you time to do a release if you so wish (consider it penance for my > forgetting to email you). I think in future we should probably do the "2.3.2c1", wait a week, do "2.3.2" thing. It's more work, but there have been enough nits that I think it would have been worthwhile (readline vs. no threads, fsync and the docs issue spring to mind). Cheers, mwh -- 81. In computing, turning the obvious into the useful is a living definition of the word "frustration". -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html From theller at python.net Thu Sep 25 07:22:41 2003 From: theller at python.net (Thomas Heller) Date: Thu Sep 25 07:22:52 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: <2m7k3x41a3.fsf@starship.python.net> (Michael Hudson's message of "Thu, 25 Sep 2003 11:20:52 +0100") References: <200309250320.h8P3KAxs007608@localhost.localdomain> <2m7k3x41a3.fsf@starship.python.net> Message-ID: Michael Hudson writes: > I think in future we should probably do the "2.3.2c1", wait a week, do > "2.3.2" thing. It's more work, but there have been enough nits that I > think it would have been worthwhile (readline vs. no threads, fsync > and the docs issue spring to mind). +1 Thomas From anthony at interlink.com.au Thu Sep 25 07:43:46 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Sep 25 07:45:31 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: Message-ID: <200309251143.h8PBhlKf019379@localhost.localdomain> >>> Thomas Heller wrote > Michael Hudson writes: > > > I think in future we should probably do the "2.3.2c1", wait a week, do > > "2.3.2" thing. It's more work, but there have been enough nits that I > > think it would have been worthwhile (readline vs. no threads, fsync > > and the docs issue spring to mind). > > +1 +1 from me, also. I'll make sure to put words to the effect in the PEPs. -- Anthony Baxter It's never too late to have a happy childhood. From zeddicus at satokar.com Thu Sep 25 12:24:05 2003 From: zeddicus at satokar.com (Michael Bartl) Date: Thu Sep 25 12:24:09 2003 Subject: [Python-Dev] PEP draft - Guidelines to processing patches Message-ID: <20030925162405.GA3858@satokar> Hi all! I've written a draft for a PEP - Guidlines for processing patches. Please have a look at it and give me some feedback. Private mail is prefered so not to spam the list ;) Please note that credits for the current content go to Martin v. L?wis. regards, Michael Bartl -------------- next part -------------- PEP: --- Title: Guidelines for Processing Patches Version: --- Author: gnosticus@gmx.at (Michael Bartl) Status: Draft Type: Informational Created: 25-Sep-2003 Post-History: Introduction This PEP contains guidelines how to process patches for the Python project on SourceForge[1]. Still to be done is to collect a list of people willing to process patches and their areas of expertise. Guidelines 1. Try to classify the patch somehow, indicating what most likely the problem is for the patch not being reviewed/accepted. 1.1 The patch might introduce a questionable feature, and nobody has dared to reject it yet, in order to not offend the submitter. In that case, voice an opinion (and record it in the patches tracker) on whether you are in favour of or opposed to the proposed feature (ideally giving a rationale on why you are). If there are enough voices showing opposition, the submitter might withdraw the patch. If there are enough voices in favour, a core developer might accept it. 1.2 The patch might be incomplete. Try to contact the submitter for a complete version. If the submitter is unreachable or does not want to finish the work, either complete it yourself, or suggest rejection of the patch. 1.3 The patch might be so complex/involved that a quick review does not reveal whether it is correct. Review the patch excessively: Perform tests, study all possible code paths to uncover cases that have been ignored. When done, record in the patch what kind of review you have done, and either indicate the problems you have found, or recommend acceptance. 2. The patch might be in an area where the "core" developers have little expertise; you often find that trivial patches are ignored just because everybody thought not to be an expert in the subject. Perform a quick review of the patch to find out whether it meets the formal criteria (completeness), and whether it is perhaps obviously correct or obviously incorrect. If not, come up with a strategy to obtain a decision on the patch: 2.1 Become an expert yourself in the subject area, and recommend acceptance or rejection afterwards. This is a both time consuming and rewarding effort. 2.2 Find an expert and have it review the patch (have a look at the people posting to python-dev more than occassionally). 2.3 Ask the submitter to clarify all questionable aspects of the patch, i.e. have him explain the patch to you. If the patch is for a little-used feature (e.g. for an obscure platform), it might be acceptable to incorporate incorrect patches, since nobody will notice, anyway - it might be enough that the submitter believes the patch is correct. References [1] http://sourceforge.net/projects/python From fperez at colorado.edu Thu Sep 25 13:43:33 2003 From: fperez at colorado.edu (Fernando Perez) Date: Thu Sep 25 13:43:37 2003 Subject: [Python-Dev] Hi, and a question about filename completion via readline in Python 2.3 In-Reply-To: <2mbrt942hh.fsf@starship.python.net> References: <3F7202F6.9050508@colorado.edu> <2mbrt942hh.fsf@starship.python.net> Message-ID: <3F732945.8020101@colorado.edu> Michael Hudson wrote: > Fernando Perez writes: >>work. I personally (and many others) use ipython as my main python >>shell, and I fell it provides a superior working environment to >>anything else I've seen out there (especially if you need to combine >>python work with manipulating underlying files). > > > Bet pyrepl still canes it for multi-line work :-) It does for on-site multi-line work (though ipython now has an @edit command which is a reasonable substitute). But ipython has a ton more features that make it a fuller environment. As I said before (long ago), it would be cool if we could integrate the two a bit more. Care to give it a go (off-list)? >>So ultimately my concerns are: >> >>- is there currently a way to re-enable readline filename completion, >> as suggested in that thread? > > > Yes. Do it yourself, in your completer function. Is it really that > hard? I would have thought os.path + glob would make it a five line > or so excercise. Oh well, I'll do it myself then. I'm a bit bummed to see functionality disappear in a release, but I can live with it. I have more and more 2.3 users who aren't too happy about this change, so I'll just go ahead and implement it. Cheers, f. From mwh at python.net Thu Sep 25 13:50:10 2003 From: mwh at python.net (Michael Hudson) Date: Thu Sep 25 13:49:36 2003 Subject: [Python-Dev] Hi, and a question about filename completion via readline in Python 2.3 In-Reply-To: <3F732945.8020101@colorado.edu> (Fernando Perez's message of "Thu, 25 Sep 2003 11:43:33 -0600") References: <3F7202F6.9050508@colorado.edu> <2mbrt942hh.fsf@starship.python.net> <3F732945.8020101@colorado.edu> Message-ID: <2my8wc3gh9.fsf@starship.python.net> Fernando Perez writes: > Michael Hudson wrote: >> Fernando Perez writes: > >>>work. I personally (and many others) use ipython as my main python >>>shell, and I fell it provides a superior working environment to >>>anything else I've seen out there (especially if you need to combine >>>python work with manipulating underlying files). >> Bet pyrepl still canes it for multi-line work :-) > > It does for on-site multi-line work (though ipython now has an @edit > command which is a reasonable substitute). But ipython has a ton more > features that make it a fuller environment. As I said before (long > ago), it would be cool if we could integrate the two a bit more. Care > to give it a go (off-list)? not right now. Too much else to do. But one day, I promise! >>>So ultimately my concerns are: >>> >>>- is there currently a way to re-enable readline filename completion, >>> as suggested in that thread? >> Yes. Do it yourself, in your completer function. Is it really that >> hard? I would have thought os.path + glob would make it a five line >> or so excercise. > > Oh well, I'll do it myself then. I'm a bit bummed to see > functionality disappear in a release, but I can live with it. The problem before was that there was *no way* of preventing the files from appearing in the completion list. I don't think their appearence there was ever intended. I just don't know enough about readline to know if implementing a readline.set_filenames_in_completion() function is feasible. But that won't appease your users who have 2.3, of course. > I have more and more 2.3 users who aren't too happy about this > change, so I'll just go ahead and implement it. Didn't *any* of them try the betas? This change is plenty old. Cheers, mwh -- The only problem with Microsoft is they just have no taste. -- Steve Jobs, (From _Triumph of the Nerds_ PBS special) and quoted by Aahz Maruch on comp.lang.python From jason at mastaler.com Thu Sep 25 14:37:08 2003 From: jason at mastaler.com (Jason R. Mastaler) Date: Thu Sep 25 14:37:14 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 References: <200309250320.h8P3KAxs007608@localhost.localdomain> <2m7k3x41a3.fsf@starship.python.net> Message-ID: Michael Hudson writes: > I think in future we should probably do the "2.3.2c1", wait a week, > do "2.3.2" thing. It's more work, but there have been enough nits > that I think it would have been worthwhile (readline vs. no threads, > fsync and the docs issue spring to mind). Will there be a 2.3.2 release, or some other sort of bugfix release to address the problems in 2.3.1? I think the os.fsync() problem is serious enough to warrant revoking 2.3.1 and making a new release to replace it rather quickly. From Jack.Jansen at cwi.nl Thu Sep 25 16:10:37 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Thu Sep 25 16:10:47 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: <200309250320.h8P3KAxs007608@localhost.localdomain> References: <200309250320.h8P3KAxs007608@localhost.localdomain> Message-ID: <53F0B466-EF94-11D7-A80F-000A27B19B96@cwi.nl> On 25-sep-03, at 5:20, Anthony Baxter wrote: > > Mea maxima culpa. I'll add a new section to the start of PEP101 and > PEP102 > about communicating with the appropriate people directly. Don't worry: I assume there's no harm done, and otherwise I'll just lobby for a quick 2.3.2. > We had a bunch of testers try out the release candidates on various > Mac OS X > releases - they reported no breakages. There's also been few/no > checkins > to the Mac-specific chunks of the tree for the release23-maint branch. There's a couple of version numbers that would have been nice to update, in the various .plist files. In other words, if you build 2.3.1 from source with a framework build then "Show Info" on the applications will show them to be 2.3, not 2.3.1. Same for the Apple Help Book that is built in to the IDE. > I'm happy to store up checkins for the 23 branch for a couple of weeks > to > give you time to do a release if you so wish (consider it penance for > my > forgetting to email you). No, don't worry. I'm going to skip 2.3.1 for MacPython-OS9. I'll catch up again with 2.3.2. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From martin at v.loewis.de Thu Sep 25 18:47:35 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Sep 25 18:48:13 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: <53F0B466-EF94-11D7-A80F-000A27B19B96@cwi.nl> References: <200309250320.h8P3KAxs007608@localhost.localdomain> <53F0B466-EF94-11D7-A80F-000A27B19B96@cwi.nl> Message-ID: Jack Jansen writes: > There's a couple of version numbers that would have been nice to > update, in the various .plist files. In other words, if you build > 2.3.1 from source with a framework build then "Show Info" on the > applications will show them to be 2.3, not 2.3.1. Same for the Apple > Help Book that is built in to the IDE. As a procedural note, I would suggest that suggest changes are done immediately *after* the previous release, instead of shortly before the next one. The release branch should be releasable at any time, without having to ask anybody for permission. What gets done is done, what doesn't get done has to wait for the next release. The less coordination, the better. Regards, Martin From martin at v.loewis.de Thu Sep 25 18:49:34 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Sep 25 18:49:40 2003 Subject: [Python-Dev] PEP on patch processing In-Reply-To: <20030924204734.GB12794@satokar> References: <20030924204734.GB12794@satokar> Message-ID: Michael Bartl writes: > I just wanted to ask if there is interest in a PEP about that topic, > because I couldn't find any relevant information on python.org. A PEP would not be appropriate: it would indicate the need for BDFL pronouncement on the procedures presented, or consensus within some group, when I think neither consensus nor BDFL pronouncement is necessary. Instead, a plain web page on python.org would be sufficient. Regards, Martin From martin at v.loewis.de Thu Sep 25 18:50:47 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Sep 25 18:50:56 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: References: <200309240802.h8O82mqF026019@localhost.localdomain> Message-ID: "Jason R. Mastaler" writes: > Python 2.3.1 (#1, Sep 24 2003, 15:55:24) > [GCC 3.2 20020903 (Red Hat Linux 8.0 3.2-7)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from os import fsync > Traceback (most recent call last): > File "", line 1, in ? > ImportError: cannot import name fsync > >>> Why is that a bug? Your system just does not have a declared fsync function. Regards, Martin From aahz at pythoncraft.com Thu Sep 25 19:46:31 2003 From: aahz at pythoncraft.com (Aahz) Date: Thu Sep 25 19:46:35 2003 Subject: [Python-Dev] PEP on patch processing In-Reply-To: References: <20030924204734.GB12794@satokar> Message-ID: <20030925234631.GC5808@panix.com> On Fri, Sep 26, 2003, Martin v. L?wis wrote: > Michael Bartl writes: >> >> I just wanted to ask if there is interest in a PEP about that topic, >> because I couldn't find any relevant information on python.org. > > A PEP would not be appropriate: it would indicate the need for BDFL > pronouncement on the procedures presented, or consensus within some > group, when I think neither consensus nor BDFL pronouncement is > necessary. Instead, a plain web page on python.org would be > sufficient. Hmmmm.... While I'd agree with you in the absence of anything else, it seems to me that the existence of PEP 3 indicates that this subject is PEP-worthy. Unless you want to argue that PEP 3 should be removed?... -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "It is easier to optimize correct code than to correct optimized code." --Bill Harlan From pycon at python.org Thu Sep 25 20:23:55 2003 From: pycon at python.org (PyCon Chair) Date: Thu Sep 25 20:24:03 2003 Subject: [Python-Dev] PyCon Announcement: Mitch Kapor is opening keynote speaker Message-ID: <20030926002355.GA10224@panix.com> [Apologies to people who see this twice; there was a glitch in the original sending.] Mitch Kapor to Keynote PyCon DC 2004 ------------------------------------ Washington, DC, September 24, 2003 The Python Software Foundation announced today that Mitch Kapor will be the opening keynote speaker for their second US Python community conference next March. Mr. Kapor is well known as a co-founder of Lotus Development Corporation and as a co-founder and the first chairman of the San Francisco-based Electronic Freedom Foundation, and has been an active entrepreneur in information technology for many years. More recently he founded the Open Software Applications Foundation (OSAF - see http://www.osafoundation.org/). The OSAF is using Python to build Chandler, an open source Personal Information Manager. PyCon is a community-oriented conference for developers of Python applications and for those with an interest in the development of the Python programming language. It offers the opportunity to learn about significant advances in Python's uses and development community, to participate in a programming sprint with some of the leading minds in the Open Source community, and to meet fellow developers from around the world. The organizers work to make the conference affordable and accessible to all. Pycon DC 2004 will be held March 24-26, 2004 in Washington, D.C. There will be a four-day development sprint before the conference. Volunteers interested in helping to run PyCon should subscribe to the organizers mailing list at http://mail.python.org/mailman/listinfo/pycon-organizers Don't miss any PyCon announcements! Subscribe at http://mail.python.org/mailman/listinfo/pycon-announce You can discuss PyCon with other interested people by subscribing at http://mail.python.org/mailman/listinfo/pycon-interest The central resource for PyCon DC 2004 is http://www.python.org/pycon/dc2004/ -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "It is easier to optimize correct code than to correct optimized code." --Bill Harlan From fdrake at acm.org Thu Sep 25 22:28:03 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Sep 25 22:28:17 2003 Subject: [Python-Dev] PEP on patch processing In-Reply-To: References: <20030924204734.GB12794@satokar> Message-ID: <16243.42035.595883.668201@grendel.zope.com> Martin v. L?wis writes: > A PEP would not be appropriate: it would indicate the need for BDFL > pronouncement on the procedures presented, or consensus within some > group, when I think neither consensus nor BDFL pronouncement is > necessary. Instead, a plain web page on python.org would be > sufficient. If some4one writes up a reST document describing this, it would be easy enough to add it to the dev/ area. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From bac at OCF.Berkeley.EDU Thu Sep 25 23:23:20 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Thu Sep 25 23:23:25 2003 Subject: [Python-Dev] PEP on patch processing In-Reply-To: <16243.42035.595883.668201@grendel.zope.com> References: <20030924204734.GB12794@satokar> <16243.42035.595883.668201@grendel.zope.com> Message-ID: <3F73B128.5020704@ocf.berkeley.edu> Fred L. Drake, Jr. wrote: > Martin v. L?wis writes: > > A PEP would not be appropriate: it would indicate the need for BDFL > > pronouncement on the procedures presented, or consensus within some > > group, when I think neither consensus nor BDFL pronouncement is > > necessary. Instead, a plain web page on python.org would be > > sufficient. > > If some4one writes up a reST document describing this, it would be > easy enough to add it to the dev/ area. > Just so people don't go too far with this, I am going to be typing up something about the Python development process. It is going to be my entry for a talk at PyCon this year. So something that encompasses all of this will get written up by December. If it isn't a total piece of literary trash I will add it to /dev on the site. -Brett From fdrake at acm.org Thu Sep 25 23:35:05 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Sep 25 23:35:34 2003 Subject: [Python-Dev] PEP on patch processing In-Reply-To: <3F73B128.5020704@ocf.berkeley.edu> References: <20030924204734.GB12794@satokar> <16243.42035.595883.668201@grendel.zope.com> <3F73B128.5020704@ocf.berkeley.edu> Message-ID: <16243.46057.561208.909564@grendel.zope.com> Brett C. writes: > So something that encompasses all of this will get written up by > December. If it isn't a total piece of literary trash I will add it to > /dev on the site. Sounds good to me! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From mwh at python.net Fri Sep 26 04:46:48 2003 From: mwh at python.net (Michael Hudson) Date: Fri Sep 26 04:46:10 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "26 Sep 2003 00:50:47 +0200") References: <200309240802.h8O82mqF026019@localhost.localdomain> Message-ID: <2mk77w2ayv.fsf@starship.python.net> martin@v.loewis.de (Martin v. L?wis) writes: > "Jason R. Mastaler" writes: > >> Python 2.3.1 (#1, Sep 24 2003, 15:55:24) >> [GCC 3.2 20020903 (Red Hat Linux 8.0 3.2-7)] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> from os import fsync >> Traceback (most recent call last): >> File "", line 1, in ? >> ImportError: cannot import name fsync >> >>> > > Why is that a bug? Your system just does not have a declared fsync > function. Err, no. I assume you've read the rest of your mail by now... Cheers, mwh -- The Internet is full. Go away. -- http://www.disobey.com/devilshat/ds011101.htm From mwh at python.net Fri Sep 26 04:47:31 2003 From: mwh at python.net (Michael Hudson) Date: Fri Sep 26 04:46:51 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "26 Sep 2003 00:47:35 +0200") References: <200309250320.h8P3KAxs007608@localhost.localdomain> <53F0B466-EF94-11D7-A80F-000A27B19B96@cwi.nl> Message-ID: <2mfzik2axo.fsf@starship.python.net> martin@v.loewis.de (Martin v. L?wis) writes: > Jack Jansen writes: > >> There's a couple of version numbers that would have been nice to >> update, in the various .plist files. In other words, if you build >> 2.3.1 from source with a framework build then "Show Info" on the >> applications will show them to be 2.3, not 2.3.1. Same for the Apple >> Help Book that is built in to the IDE. > > As a procedural note, I would suggest that suggest changes are done > immediately *after* the previous release, instead of shortly before > the next one. The release branch should be releasable at any time, > without having to ask anybody for permission. What gets done is done, > what doesn't get done has to wait for the next release. The less > coordination, the better. +1 I guess this should be added to PEP 102, too. Cheers, mwh -- BUGS Never use this function. This function modifies its first argument. The identity of the delimiting character is lost. This function cannot be used on constant strings. -- the glibc manpage for strtok(3) From zeddicus at satokar.com Fri Sep 26 05:34:38 2003 From: zeddicus at satokar.com (Michael Bartl) Date: Fri Sep 26 05:34:42 2003 Subject: [Python-Dev] PEP on patch processing In-Reply-To: <16243.46057.561208.909564@grendel.zope.com> References: <20030924204734.GB12794@satokar> <16243.42035.595883.668201@grendel.zope.com> <3F73B128.5020704@ocf.berkeley.edu> <16243.46057.561208.909564@grendel.zope.com> Message-ID: <20030926093438.GA2288@satokar> On Thu, Sep 25, 2003 at 11:35:05PM -0400, Fred L. Drake, Jr. wrote: > > Brett C. writes: > > So something that encompasses all of this will get written up by > > December. If it isn't a total piece of literary trash I will add it to > > /dev on the site. > > Sounds good to me! +1 We could still put the content of the current PEP draft into a reST format and put it up on /dev for now? From anthony at interlink.com.au Fri Sep 26 01:34:17 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Sep 26 06:37:17 2003 Subject: [Python-Dev] RELEASED Python 2.3.1 In-Reply-To: <53F0B466-EF94-11D7-A80F-000A27B19B96@cwi.nl> Message-ID: <200309260534.h8Q5YHAb013833@localhost.localdomain> >>> Jack Jansen wrote > There's a couple of version numbers that would have been nice to update, > in the various .plist files. In other words, if you build 2.3.1 from > source with a framework build then "Show Info" on the applications will show > them to be 2.3, not 2.3.1. Same for the Apple Help Book that is built in to > the IDE. If you could point to the files, I'll add them to the PEPs. -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Fri Sep 26 01:24:46 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Sep 26 06:37:23 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: Message-ID: <200309260524.h8Q5Ok1t013688@localhost.localdomain> >>> "Jason R. Mastaler" wrote > Will there be a 2.3.2 release, or some other sort of bugfix release to > address the problems in 2.3.1? I think the os.fsync() problem is > serious enough to warrant revoking 2.3.1 and making a new release > to replace it rather quickly. There's a workaround on the 2.3.1 bugs page. The only problems I'm aware of with 2.3.1 are the HP/UX build problems (workaround on bugs.html), the documentation mistakenly identifying itself as '2.4.0alpha', and the os.fsync problem. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From amk at amk.ca Fri Sep 26 07:46:39 2003 From: amk at amk.ca (A.M. Kuchling) Date: Fri Sep 26 07:46:47 2003 Subject: [Python-Dev] Re: PEP on patch processing References: <20030924204734.GB12794@satokar> <16243.42035.595883.668201@grendel.zope.com> <3F73B128.5020704@ocf.berkeley.edu> Message-ID: On Thu, 25 Sep 2003 20:23:20 -0700, Brett C. wrote: > So something that encompasses all of this will get written up by > December. If it isn't a total piece of literary trash I will add it to > /dev on the site. Note that http://www.python.org/patches/ has some material on patch processing that also belongs in /dev/. --amk From amk at amk.ca Fri Sep 26 07:50:07 2003 From: amk at amk.ca (A.M. Kuchling) Date: Fri Sep 26 08:00:33 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 References: <200309250320.h8P3KAxs007608@localhost.localdomain> <2m7k3x41a3.fsf@starship.python.net> Message-ID: On Thu, 25 Sep 2003 12:37:08 -0600, Jason R. Mastaler wrote: > Will there be a 2.3.2 release, or some other sort of bugfix release to > address the problems in 2.3.1? Another minor nit: the 2.3.1 .tgz file contains the CVS/ subdirectories. This doesn't actually hurt anything, though, just cluttering up the tree a bit. --amk From barry at python.org Fri Sep 26 08:21:41 2003 From: barry at python.org (Barry Warsaw) Date: Fri Sep 26 08:21:44 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: Message-ID: On Friday, September 26, 2003, at 07:50 AM, A.M. Kuchling wrote: > On Thu, 25 Sep 2003 12:37:08 -0600, Jason R. Mastaler > wrote: >> Will there be a 2.3.2 release, or some other sort of bugfix release to >> address the problems in 2.3.1? > > Another minor nit: the 2.3.1 .tgz file contains the CVS/ > subdirectories. This doesn't actually hurt anything, though, just > cluttering up the tree a bit. Was it not made with cvs export? -Barry From tjreedy at udel.edu Fri Sep 26 09:28:03 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Fri Sep 26 09:28:10 2003 Subject: [Python-Dev] Re: PEP on patch processing References: <20030924204734.GB12794@satokar><16243.42035.595883.668201@grendel.zope.com><3F73B128.5020704@ocf.berkeley.edu><16243.46057.561208.909564@grendel.zope.com> <20030926093438.GA2288@satokar> Message-ID: "Michael Bartl" wrote in message news:20030926093438.GA2288@satokar... > On Thu, Sep 25, 2003 at 11:35:05PM -0400, Fred L. Drake, Jr. wrote: > > > > Brett C. writes: > > > So something that encompasses all of this will get written up by > > > December. If it isn't a total piece of literary trash I will add it to > > > /dev on the site. > > > > Sounds good to me! > > +1 > > We could still put the content of the current PEP draft into a reST format > and put it up on /dev for now? Please do. If there were good step-by-step instructions on how to review a bug or patch submission, and 'official' encouragement to do as much/many as one can, even if not all steps, then I might actually try to do some -- and possibly before December. Terry J. Reedy From tjreedy at udel.edu Fri Sep 26 09:52:15 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Fri Sep 26 09:52:21 2003 Subject: [Python-Dev] Re: PEP on patch processing References: <20030924204734.GB12794@satokar> <16243.42035.595883.668201@grendel.zope.com><3F73B128.5020704@ocf.berkeley.edu> Message-ID: "A.M. Kuchling" wrote in message news:oprv3uz1v6qmnqtt@news.gmane.org... > Note that http://www.python.org/patches/ has some material on patch > processing that also belongs in /dev/. I check all the links on ../dev/ I noticed and found no direct link to /patches/. I think there should be (Explanations list, bottom of page) The /devfaq/ has this entry: 3.2. How to submit a patch? Please read the Patch Submission Guidelines at http://www.python.org/patches A recent copy can be found in the Appendix of this FAQ. Q. Which of these two nearly identical pages is more recent? Guess:Appendix A2 has "It's almost a year since Python 1.5.2 was released". /patches/ is undated (dating pages in one change I would suggest for the site) but the differences suggest to me that it is newer. Please delete the older version. Directing would-be new submitters to both seems senseless and can only discourage, not encourage. Terry J. Reedy cc webmaster@python.org From skip at pobox.com Fri Sep 26 10:24:19 2003 From: skip at pobox.com (Skip Montanaro) Date: Fri Sep 26 10:24:34 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: References: <200309240802.h8O82mqF026019@localhost.localdomain> Message-ID: <16244.19475.777510.550060@montanaro.dyndns.org> Martin> "Jason R. Mastaler" writes: >> Python 2.3.1 (#1, Sep 24 2003, 15:55:24) >> [GCC 3.2 20020903 (Red Hat Linux 8.0 3.2-7)] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> from os import fsync >> Traceback (most recent call last): >> File "", line 1, in ? >> ImportError: cannot import name fsync >> >>> Martin> Why is that a bug? Your system just does not have a declared Martin> fsync function. No, it was a typo in the configure scripts. I checked in a fix for both the head and release23-maint branches yesterday. Red Hat 8.0 does have fsync: FSYNC(2) Linux Programmer's Manual FSYNC(2) NAME fsync, fdatasync - synchronize a file's complete in-core state with that on disk ... That is from an 8.0 system. Skip From guido at python.org Fri Sep 26 11:10:30 2003 From: guido at python.org (Guido van Rossum) Date: Fri Sep 26 11:11:29 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: Your message of "Fri, 26 Sep 2003 08:21:41 EDT." References: Message-ID: <200309261510.h8QFAUb21375@12-236-84-31.client.attbi.com> > > Another minor nit: the 2.3.1 .tgz file contains the CVS/ > > subdirectories. This doesn't actually hurt anything, though, just > > cluttering up the tree a bit. > > Was it not made with cvs export? Ouch. Isn't this in the PEP? The CVS directories *do* hurt, IMO (by confusing people). --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Fri Sep 26 11:35:03 2003 From: barry at python.org (Barry Warsaw) Date: Fri Sep 26 11:36:07 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <200309261510.h8QFAUb21375@12-236-84-31.client.attbi.com> References: <200309261510.h8QFAUb21375@12-236-84-31.client.attbi.com> Message-ID: <1064590503.30783.12.camel@anthem> On Fri, 2003-09-26 at 11:10, Guido van Rossum wrote: > > > Another minor nit: the 2.3.1 .tgz file contains the CVS/ > > > subdirectories. This doesn't actually hurt anything, though, just > > > cluttering up the tree a bit. > > > > Was it not made with cvs export? > > Ouch. Isn't this in the PEP? The CVS directories *do* hurt, IMO (by > confusing people). It's in PEPs 101 and 102. -Barry From jason at mastaler.com Fri Sep 26 11:47:14 2003 From: jason at mastaler.com (Jason R. Mastaler) Date: Fri Sep 26 11:47:27 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> Message-ID: Anthony Baxter writes: >> Will there be a 2.3.2 release, or some other sort of bugfix release >> to address the problems in 2.3.1? I think the os.fsync() problem >> is serious enough to warrant revoking 2.3.1 and making a new >> release to replace it rather quickly. > > There's a workaround on the 2.3.1 bugs page. The only problems I'm > aware of with 2.3.1 are the HP/UX build problems (workaround on > bugs.html), the documentation mistakenly identifying itself as > '2.4.0alpha', and the os.fsync problem. There's a workaround, but the problem is that I think I'll be the one bearing the brunt of this bug for who knows how long. I've gotten about a half dozen problem reports about it already on the TMDA lists. TMDA uses os.fsync() in its mbox and Maildir delivery code, so practically every user will find their TMDA broken as soon as they install or upgrade to 2.3.1. Since Python point releases are usually so solid, they don't consider it might be a bug in the interpreter, and assume it's a TMDA bug. TMDA is just one example. Repeat for the other packages out there which rely on os.fsync(). The longer 2.3.1 is out there, the more it will be installed, incorporated into OS package distributions, etc, and the harder this breakage will be to shake. From barry at python.org Fri Sep 26 11:53:29 2003 From: barry at python.org (Barry Warsaw) Date: Fri Sep 26 11:53:36 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> Message-ID: <1064591608.30783.30.camel@anthem> On Fri, 2003-09-26 at 11:47, Jason R.Mastaler wrote: > The longer 2.3.1 is out there, the more it will be installed, > incorporated into OS package distributions, etc, and the harder this > breakage will be to shake. It certainly seems reasonable to me to release 2.3.2 fairly quickly. -Barry From theller at python.net Fri Sep 26 11:58:35 2003 From: theller at python.net (Thomas Heller) Date: Fri Sep 26 12:00:40 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <1064591608.30783.30.camel@anthem> (Barry Warsaw's message of "Fri, 26 Sep 2003 11:53:29 -0400") References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> Message-ID: Barry Warsaw writes: > On Fri, 2003-09-26 at 11:47, Jason R.Mastaler wrote: > >> The longer 2.3.1 is out there, the more it will be installed, >> incorporated into OS package distributions, etc, and the harder this >> breakage will be to shake. > > It certainly seems reasonable to me to release 2.3.2 fairly quickly. But not earlier than at least one week after 2.3.2rc1 ;-) Thomas From jason at mastaler.com Fri Sep 26 12:07:29 2003 From: jason at mastaler.com (Jason R. Mastaler) Date: Fri Sep 26 12:07:35 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> Message-ID: Thomas Heller writes: >> It certainly seems reasonable to me to release 2.3.2 fairly quickly. > > But not earlier than at least one week after 2.3.2rc1 ;-) Well, if you're going to wait that long, I think you might consider adding some sort of warning to the 2.3.1 download area. My fear is that some large OS vendor (e.g, Redhat) will pick up 2.3.1 for a future release. From python at discworld.dyndns.org Fri Sep 26 12:38:46 2003 From: python at discworld.dyndns.org (Charles Cazabon) Date: Fri Sep 26 12:34:05 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: ; from jason@mastaler.com on Fri, Sep 26, 2003 at 10:07:29AM -0600 References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> Message-ID: <20030926103846.A26782@discworld.dyndns.org> Jason R. Mastaler wrote: > Thomas Heller writes: > > >> It certainly seems reasonable to me to release 2.3.2 fairly quickly. > > > > But not earlier than at least one week after 2.3.2rc1 ;-) > > Well, if you're going to wait that long, I think you might consider > adding some sort of warning to the 2.3.1 download area. My fear is > that some large OS vendor (e.g, Redhat) will pick up 2.3.1 for a > future release. Too late: 2.3.1 is in at least some versions of Debian already -- I'm getting reports on the os.fsync() issue already from my users (it breaks getmail). Could 2.3.1 be revoked or replaced ASAP? "2.3.1b" or similar with only this change would suit me. Charles -- ----------------------------------------------------------------------- Charles Cazabon GPL'ed software available at: http://www.qcc.ca/~charlesc/software/ ----------------------------------------------------------------------- From guido at python.org Fri Sep 26 12:40:12 2003 From: guido at python.org (Guido van Rossum) Date: Fri Sep 26 12:41:03 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: Your message of "Fri, 26 Sep 2003 10:38:46 MDT." <20030926103846.A26782@discworld.dyndns.org> References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> <20030926103846.A26782@discworld.dyndns.org> Message-ID: <200309261640.h8QGeCb21645@12-236-84-31.client.attbi.com> > Too late: 2.3.1 is in at least some versions of Debian already -- > I'm getting reports on the os.fsync() issue already from my users > (it breaks getmail). Could 2.3.1 be revoked or replaced ASAP? > "2.3.1b" or similar with only this change would suit me. That doesn't fit in the numbering scheme, it would have to be 2.3.2. Given the amount of traffic about this I'm tempted to agree that we need to move quickly and issue a fix for this issue (only). --Guido van Rossum (home page: http://www.python.org/~guido/) From anthony at interlink.com.au Fri Sep 26 12:42:59 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Sep 26 12:45:01 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <1064590503.30783.12.camel@anthem> Message-ID: <200309261643.h8QGh05H020344@localhost.localdomain> >>> Barry Warsaw wrote > > > Was it not made with cvs export? > > > > Ouch. Isn't this in the PEP? The CVS directories *do* hurt, IMO (by > > confusing people). > > It's in PEPs 101 and 102. I'm very very confused, since I _did_ use cvs export. Really. Repeatedly. Ugh. Given this, I'm happy to look at a 2.3.2 in a couple of weeks - I'd like very much to finish the PEP rewrite and get it reviewed first. Arrrrrgh. (no, not talk-like-a-pirate day, frustration) Anthony -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Fri Sep 26 12:46:41 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Sep 26 12:48:30 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <200309261640.h8QGeCb21645@12-236-84-31.client.attbi.com> Message-ID: <200309261646.h8QGkf4h020451@localhost.localdomain> >>> Guido van Rossum wrote > That doesn't fit in the numbering scheme, it would have to be 2.3.2. > Given the amount of traffic about this I'm tempted to agree that we > need to move quickly and issue a fix for this issue (only). We really should fix the HP/UX build problem as well. Anthony From anthony at interlink.com.au Fri Sep 26 12:59:28 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Sep 26 13:01:10 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: Message-ID: <200309261659.h8QGxSTc020775@localhost.localdomain> >>> "Jason R. Mastaler" wrote > There's a workaround, but the problem is that I think I'll be the one > bearing the brunt of this bug for who knows how long. I've gotten > about a half dozen problem reports about it already on the TMDA lists. > TMDA uses os.fsync() in its mbox and Maildir delivery code, so > practically every user will find their TMDA broken as soon as they > install or upgrade to 2.3.1. Since Python point releases are usually > so solid, they don't consider it might be a bug in the interpreter, > and assume it's a TMDA bug. TMDA is just one example. Repeat for the > other packages out there which rely on os.fsync(). Hm. I ran the Zope2, Zope3, and Twisted test suites against the release build before the release. I'm open to additional suggestions (I just grabbed tmda, but it doesn't have a test suite). In this case, the problem hopefully would've been picked up with a release candidate, but I'm also happy to add additional slabs of test suites to run against code before it's released... Anthony -- Anthony Baxter It's never too late to have a happy childhood. From tim.one at comcast.net Fri Sep 26 13:23:13 2003 From: tim.one at comcast.net (Tim Peters) Date: Fri Sep 26 13:30:13 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <200309261646.h8QGkf4h020451@localhost.localdomain> Message-ID: [Guido] >> That doesn't fit in the numbering scheme, it would have to be 2.3.2. >> Given the amount of traffic about this I'm tempted to agree that we >> need to move quickly and issue a fix for this issue (only). [Anthony Baxter] > We really should fix the HP/UX build problem as well. That sounds dangerous to me: there's *always* an HP/UX build problem, if you count (as I do) getting threads to work. Every release, some HP/UX user submits a patch that breaks the build for other HP/UX users. So unless we've got a bona fide HP/UX expert on tap now (do we? that would be way cool ), don't hold up solving a clear and widespread Linux problem by waiting for one to appear. From anthony at interlink.com.au Fri Sep 26 13:42:45 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Sep 26 13:44:39 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: Message-ID: <200309261742.h8QHgkHM029591@localhost.localdomain> >>> "Tim Peters" wrote > That sounds dangerous to me: there's *always* an HP/UX build problem, if > you count (as I do) getting threads to work. Every release, some HP/UX user > submits a patch that breaks the build for other HP/UX users. So unless > we've got a bona fide HP/UX expert on tap now (do we? that would be way cool > ), don't hold up solving a clear and widespread Linux problem by > waiting for one to appear. On the scale of "HP/UX build problems we've known and loved", this one's a trivial one. https://sourceforge.net/tracker/?func=detail&atid=105470&aid=811160&group_id=5470 has a patch (from the gcc bugtracker - they ran into the exact same problem) Now, the readline+no-threads=boom coredump, well, that involves threads, and the original reporter even mentions HP/UX, but it's more a simple-ish one - missing #ifdef WITH_THREAD calls, as far as I can see. https://sourceforge.net/tracker/?func=detail&atid=105470&aid=811844&group_id=5470 (Someone should probably do a scan of the code in Modules to make sure all the threading API calls are wrapped in WITH_THREAD #ifdefs. Not me, now, though - way past bedtime). -- Anthony Baxter It's never too late to have a happy childhood. From jeremy at zope.com Fri Sep 26 13:43:31 2003 From: jeremy at zope.com (Jeremy Hylton) Date: Fri Sep 26 13:45:32 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <200309261640.h8QGeCb21645@12-236-84-31.client.attbi.com> References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> <20030926103846.A26782@discworld.dyndns.org> <200309261640.h8QGeCb21645@12-236-84-31.client.attbi.com> Message-ID: <1064598210.19498.6.camel@localhost.localdomain> On Fri, 2003-09-26 at 12:40, Guido van Rossum wrote: > > Too late: 2.3.1 is in at least some versions of Debian already -- > > I'm getting reports on the os.fsync() issue already from my users > > (it breaks getmail). Could 2.3.1 be revoked or replaced ASAP? > > "2.3.1b" or similar with only this change would suit me. > > That doesn't fit in the numbering scheme, it would have to be 2.3.2. > Given the amount of traffic about this I'm tempted to agree that we > need to move quickly and issue a fix for this i +1. We should treat 2.3.1 like 2.3.1rc1 and aim for Sep. 30 for a 2.3.2. Jeremy From anthony at interlink.com.au Fri Sep 26 13:52:19 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Sep 26 13:54:10 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <1064598210.19498.6.camel@localhost.localdomain> Message-ID: <200309261752.h8QHqJgD029823@localhost.localdomain> >>> Jeremy Hylton wrote > We should treat 2.3.1 like 2.3.1rc1 and aim for Sep. 30 for a 2.3.2. Wouldn't it be better to cut a 2.3.2rc before the final 2.3.2? once-bitten-ly, Anthony From jeremy at zope.com Fri Sep 26 13:57:55 2003 From: jeremy at zope.com (Jeremy Hylton) Date: Fri Sep 26 13:59:36 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <200309261752.h8QHqJgD029823@localhost.localdomain> References: <200309261752.h8QHqJgD029823@localhost.localdomain> Message-ID: <1064599075.19498.14.camel@localhost.localdomain> On Fri, 2003-09-26 at 13:52, Anthony Baxter wrote: > >>> Jeremy Hylton wrote > > We should treat 2.3.1 like 2.3.1rc1 and aim for Sep. 30 for a 2.3.2. > > Wouldn't it be better to cut a 2.3.2rc before the final 2.3.2? If we only fix the few critical problems -- CVS directories, fsync problem, Mac version numbers (?) -- then there doesn't seem to be much risk. But it wouldn't be too hard to convince me. (I gave my all clear on 2.3.1, after all.) Jeremy From tim.one at comcast.net Fri Sep 26 14:24:27 2003 From: tim.one at comcast.net (Tim Peters) Date: Fri Sep 26 14:24:33 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <200309261752.h8QHqJgD029823@localhost.localdomain> Message-ID: [Jeremy Hylton] >> We should treat 2.3.1 like 2.3.1rc1 and aim for Sep. 30 for a 2.3.2. [Anthony Baxter] > Wouldn't it be better to cut a 2.3.2rc before the final 2.3.2? > > once-bitten-ly, Well, Jeremy's "Sep. 30" is only 4 days away. If 2.3.2 is restricted to fixing just the obvious big problems already reported against 2.3.1, in reality it's like Jeremy said: 2.3.1 was really 2.3.2c1 <0.5 wink>. Don't get too gun-shy. There were lots of changes in who did what for the 2.3.1 release, and some glitches were inevitable. It *had* gotten so smooth with the same people doing the same things all the time that, e.g., you must have noticed that Guido stopped doing anything for releases. I would have liked to pay a lot more attention but just couldn't make time for it; I'm sure the same is true of the other PythonLabs alumni (including Guido). So some things could have gone better. It's not that big a deal -- next time they will go better. Remember that we pioneered all possible ways to screw up a release already, and discovered that it's a finite set . From barry at python.org Fri Sep 26 14:30:33 2003 From: barry at python.org (Barry Warsaw) Date: Fri Sep 26 14:30:44 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: References: Message-ID: <1064601033.31604.41.camel@anthem> On Fri, 2003-09-26 at 14:24, Tim Peters wrote: > Don't get too gun-shy. Plus there's always 2.3.3 :) -Barry From nas at arctrix.com Fri Sep 26 14:48:54 2003 From: nas at arctrix.com (Neil Schemenauer) Date: Fri Sep 26 14:48:43 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: References: <200309261752.h8QHqJgD029823@localhost.localdomain> Message-ID: <20030926184853.GB22837@mems-exchange.org> On Fri, Sep 26, 2003 at 02:24:27PM -0400, Tim Peters wrote: > Don't get too gun-shy. There were lots of changes in who did what for the > 2.3.1 release, and some glitches were inevitable. Some humble advice from someone who has done quite a few software releases in the last few years (mostly internal). The release process should be automated as much as possible. I think the Python project could do a bit better in this regard. Some ideas: * try to reduce the number of version numbers in different files need to be adjusted. Also, try to automate the adjustment. * write scripts to generate the final package and also verify its sanity (e.g. check version numbers, dates, detect files that should not be included in the release) * write scripts to upload the package for distribution. Ensure proper naming, permissions, etc. Python already has a release checklist. That's good. Be sure to follow it when doing a release and make sure it's up-to-date. Unfortunately Python doesn't do releases often enough for the process to be really polished. One effective way of improving the process is to have a new person do it with a veteran looking on. The new person should try to complete the release using only the checklist as the guide. The veteran is on hand to make sure nothing goes wrong. The veteran should also take notes; documenting any errors or omissions in the release checklist or ideas on ways to further automate the process. Neil From barry at python.org Fri Sep 26 15:08:54 2003 From: barry at python.org (Barry Warsaw) Date: Fri Sep 26 15:09:09 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <20030926184853.GB22837@mems-exchange.org> References: <200309261752.h8QHqJgD029823@localhost.localdomain> <20030926184853.GB22837@mems-exchange.org> Message-ID: <1064603333.31604.47.camel@anthem> On Fri, 2003-09-26 at 14:48, Neil Schemenauer wrote: > Some humble advice from someone who has done quite a few software > releases in the last few years (mostly internal). The release > process should be automated as much as possible. I agree. It's what I do for Mailman, although 1) the release process is much simpler and 2) I don't have to coordinate with anyone else. Both are becoming less true though with each release. -Barry From theller at python.net Fri Sep 26 15:52:03 2003 From: theller at python.net (Thomas Heller) Date: Fri Sep 26 15:52:17 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <20030926184853.GB22837@mems-exchange.org> (Neil Schemenauer's message of "Fri, 26 Sep 2003 11:48:54 -0700") References: <200309261752.h8QHqJgD029823@localhost.localdomain> <20030926184853.GB22837@mems-exchange.org> Message-ID: Neil Schemenauer writes: > * try to reduce the number of version numbers in different files need to > be adjusted. I suggest to update the version numbers in these files to the next version *after* a release is done. So, in the release23-maint branch the version numbers could *now* be adjusted to 2.3.2. And it would help if I could build the HTML docs myself from CVS. I did manage to create the pdf files with TeTex under windows, but I didn't succeed with the html pages so far. This way, a Windows release could be built at any time, and I would not only install and test it, but also use it as my default version. Thomas From python at rcn.com Fri Sep 26 15:58:05 2003 From: python at rcn.com (Raymond Hettinger) Date: Fri Sep 26 15:58:22 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <200309261640.h8QGeCb21645@12-236-84-31.client.attbi.com> Message-ID: <001201c38468$8077b5a0$e841fea9@oemcomputer> Should there be an announcement that 2.3.2 is forthcoming and hold off on further downloads or distribution of 2.3.1? Raymond -----Original Message----- From: python-dev-bounces@python.org [mailto:python-dev-bounces@python.org] On Behalf Of Guido van Rossum Sent: Friday, September 26, 2003 12:40 PM To: Charles Cazabon Cc: python-dev@python.org Subject: Re: [Python-Dev] Re: RELEASED Python 2.3.1 > Too late: 2.3.1 is in at least some versions of Debian already -- > I'm getting reports on the os.fsync() issue already from my users > (it breaks getmail). Could 2.3.1 be revoked or replaced ASAP? > "2.3.1b" or similar with only this change would suit me. That doesn't fit in the numbering scheme, it would have to be 2.3.2. Given the amount of traffic about this I'm tempted to agree that we need to move quickly and issue a fix for this issue (only). --Guido van Rossum (home page: http://www.python.org/~guido/) _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev From jason at mastaler.com Fri Sep 26 17:06:28 2003 From: jason at mastaler.com (Jason R. Mastaler) Date: Fri Sep 26 17:06:33 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> <20030926103846.A26782@discworld.dyndns.org> Message-ID: Charles Cazabon writes: > Too late: 2.3.1 is in at least some versions of Debian already The Debian situation is a bit better because they can incorporate patches and issue a point release of the package itself. I contacted the Debian/Python maintainer, and he plans to incorporate the os.fsync() fix as soon as the sourceforge CVS delay catches up to him. More troublesome are distros like Redhat whose packages are not dynamically updateable. They are sealed in stone once they become part of a release. > Could 2.3.1 be revoked or replaced ASAP? "2.3.1b" or similar with > only this change would suit me. I'd be in favour of revoking 2.3.1 as well because of the severity of this bug. I don't think we'll be completely rid of it otherwise. From jason at mastaler.com Fri Sep 26 17:09:07 2003 From: jason at mastaler.com (Jason R. Mastaler) Date: Fri Sep 26 17:10:31 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 References: <200309261646.h8QGkf4h020451@localhost.localdomain> Message-ID: "Tim Peters" writes: > So unless we've got a bona fide HP/UX expert on tap now (do we? that > would be way cool ), don't hold up solving a clear and > widespread Linux problem by waiting for one to appear. Just to be clear, the os.fsync() problem is not Linux-only. It's also on FreeBSD (probably every Unix platform actually). From skip at pobox.com Fri Sep 26 17:14:13 2003 From: skip at pobox.com (Skip Montanaro) Date: Fri Sep 26 17:14:22 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: References: <200309261646.h8QGkf4h020451@localhost.localdomain> Message-ID: <16244.44069.798860.174782@montanaro.dyndns.org> >> So unless we've got a bona fide HP/UX expert on tap now (do we? that >> would be way cool ), don't hold up solving a clear and >> widespread Linux problem by waiting for one to appear. Jason> Just to be clear, the os.fsync() problem is not Linux-only. It's Jason> also on FreeBSD (probably every Unix platform actually). Correct. There are two new C macros in pyconfig.h, HAVE_FSYNC and HAVE_FDATASYNC. When originally added, the former was mistyped as HAVE_SYNC. Skip From jason at mastaler.com Fri Sep 26 17:14:46 2003 From: jason at mastaler.com (Jason R. Mastaler) Date: Fri Sep 26 17:20:31 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 References: <200309261659.h8QGxSTc020775@localhost.localdomain> Message-ID: Anthony Baxter writes: > Hm. I ran the Zope2, Zope3, and Twisted test suites against the > release build before the release. I'm open to additional suggestions > (I just grabbed tmda, but it doesn't have a test suite). TMDA doesn't have a test suite, but it really should. It's on the TODO list. :-) > In this case, the problem hopefully would've been picked up with a > release candidate, but I'm also happy to add additional slabs of > test suites to run against code before it's released... I'll try to get more involved with testing release candidates from now on. From mwh at python.net Fri Sep 26 17:34:41 2003 From: mwh at python.net (Michael Hudson) Date: Fri Sep 26 17:34:01 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: (Thomas Heller's message of "Fri, 26 Sep 2003 21:52:03 +0200") References: <200309261752.h8QHqJgD029823@localhost.localdomain> <20030926184853.GB22837@mems-exchange.org> Message-ID: <2mzngr1bf2.fsf@starship.python.net> Thomas Heller writes: > Neil Schemenauer writes: > >> * try to reduce the number of version numbers in different files need to >> be adjusted. > > I suggest to update the version numbers in these files to the next > version *after* a release is done. So, in the release23-maint branch > the version numbers could *now* be adjusted to 2.3.2. > > And it would help if I could build the HTML docs myself from CVS. I did > manage to create the pdf files with TeTex under windows, but I didn't > succeed with the html pages so far. It occurs to me that I don't know *why* Fred is so much the documentation man; I've not had any trouble processing the docs into HTML lately (haven't tried on Windows, admittedly, and I haven't tried to make info ever). What else needs to be done? There must be quite a bit of mucking about on creosote to do, I guess. Finding a substitue Jack might be harder (though given Bob Ippolito's work rate at the moment we could probably make him MacPython maintainer without him noticing ). Cheers, mwh -- In short, just business as usual in the wacky world of floating point . -- Tim Peters, comp.lang.python From fdrake at acm.org Fri Sep 26 18:17:37 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Sep 26 18:17:54 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <200309261640.h8QGeCb21645@12-236-84-31.client.attbi.com> References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> <20030926103846.A26782@discworld.dyndns.org> <200309261640.h8QGeCb21645@12-236-84-31.client.attbi.com> Message-ID: <16244.47873.343272.206145@grendel.zope.com> Guido van Rossum writes: > That doesn't fit in the numbering scheme, it would have to be 2.3.2. > Given the amount of traffic about this I'm tempted to agree that we > need to move quickly and issue a fix for this issue (only). I think the documentation versioning problem should be fixed as well. I presume we can start making changes to the release23-maint branch now? -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From greg at electricrain.com Fri Sep 26 19:28:40 2003 From: greg at electricrain.com (Gregory P. Smith) Date: Fri Sep 26 19:28:49 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: References: <200309241550.h8OFojlx031706@localhost.localdomain> Message-ID: <20030926232840.GB17491@zot.electricrain.com> On Wed, Sep 24, 2003 at 12:01:04PM -0400, Tim Peters wrote: > [Anthony Baxter] > > I haven't been following the spambayes lists too closely. Are there > > concrete problems with bsddb that are cropping up, or just a general > > wariness of it? > > > > If there _is_ a problem with bsddb, it needs to be addressed. Too > > many things depend on it. > > Reports of database corruption are common in spambayes. At least one > knowledgable tester reported his problems went away after moving to a recent > Sleepycat release (4.1.25, IIRC). ... > They seem to come from non-Outlook people > using Berkeley for the message info database. Richie got a whittled down > threaded test that fails on Windows and Linux, and there's already a > (Python) bug report open on that; it's not thought to be relevant to how > spambayes uses Berkeley, though. Its been my impression that the sporatic bsddb testsuite failures are BerkeleyDB related rather than anything the python module can be responsible for (other than the previously mentioned needed improvements in cleaning up the on disk "temporary" test environment before launching tests). For anyone reporting berkeleydb issues, its important to find out the version of BerkeleyDB they're using and on what platform. When I find time I want to take the simple test that Richie created, try it in C and if it still fails there (it should) on the 4.2.xx release canidate i'll submit it to sleepycat. The full BerkeleyDB library is unfortunately complicated because it supports so much more than most people need. -g From greg at electricrain.com Fri Sep 26 19:39:59 2003 From: greg at electricrain.com (Gregory P. Smith) Date: Fri Sep 26 19:40:44 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <16241.45106.296743.387428@montanaro.dyndns.org> References: <200309241233.h8OCXcbg028668@localhost.localdomain> <1064409627.1958.90.camel@anthem> <16241.45106.296743.387428@montanaro.dyndns.org> Message-ID: <20030926233959.GC17491@zot.electricrain.com> > For long-running SpamBayes apps like pop3proxy and the > Outlook plugin, pickles make sense if shared access to the scoring database > is not needed. Combine that with data event/change logging and you have effectively the same thing as what many apps think they need to use a database library for. Replay the entire log of data changes internally on infrequent app startup to generate your internal state. when the log file is getting to big, write a full state snapshot and continue logging changes from it. no need to save the entire pickle everytime something changes. From skip at pobox.com Fri Sep 26 19:06:49 2003 From: skip at pobox.com (Skip Montanaro) Date: Fri Sep 26 23:11:46 2003 Subject: [Python-Dev] How to test for stuff like fsync? Message-ID: <16244.50825.588063.154695@montanaro.dyndns.org> When I checked in the HAVE_SYNC -> HAVE_FSYNC change I thought briefly about including a test to test_os.py. I eventually decided against it because it was clear that a simpleminded test for os.fsync's existence was useless. Jason seems to have been bitten badly by it. I doubt that fsync() is available everywhere (otherwise why test for it in the configure script?), so it seems that Jason should have been testing for its presence (though see below) and worming around its abscence if he wanted TMDA to run on as many platforms as possible. This suggests the following questions: * Is there a reasonable way to write tests which will detect this sort of error? * Should there be special documentation about os-dependent functions which are not universally available? The fsync() doc says nothing about it not always being available (thus Jason can be forgiven for not calling hasattr(os, "fsync")). Skip From anthony at interlink.com.au Sat Sep 27 00:33:20 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat Sep 27 00:35:16 2003 Subject: [Python-Dev] How to test for stuff like fsync? In-Reply-To: <16244.50825.588063.154695@montanaro.dyndns.org> Message-ID: <200309270433.h8R4XKwl006833@localhost.localdomain> >>> Skip Montanaro wrote > * Is there a reasonable way to write tests which will detect this sort of > error? A thought that occurred to me a short time ago: write a script that outputs the contents of each builtin module (the names in the module) to a file. Run that under previous version of python. Run it under release candidate. Compare differences. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Sat Sep 27 00:37:59 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat Sep 27 00:39:44 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: Message-ID: <200309270437.h8R4bxk0006927@localhost.localdomain> >>> "Jason R. Mastaler" wrote > > Could 2.3.1 be revoked or replaced ASAP? "2.3.1b" or similar with > > only this change would suit me. > > I'd be in favour of revoking 2.3.1 as well because of the severity of > this bug. I don't think we'll be completely rid of it otherwise. I';m not sure what the term "revoke" would mean in this context. Replacing it soon would work for me, and a big warning on the 2.3.1 page about the fsync problem and the HP/UX build problem (these are the two that _must_ be fixed in 2.3.2, as far as I'm concerned). I can whack a bit of text on the 2.3.1 page saying "there will be a 2.3.2 release shortly - stay tuned" if people think it's appropriate. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Sat Sep 27 01:09:09 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat Sep 27 01:10:55 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <200309270437.h8R4bxk0006927@localhost.localdomain> Message-ID: <200309270509.h8R59ArD007367@localhost.localdomain> >>> Anthony Baxter wrote > I can whack a bit of text on the 2.3.1 page saying "there will be a 2.3.2 > release shortly - stay tuned" if people think it's appropriate. This is now done. -- Anthony Baxter It's never too late to have a happy childhood. From tjreedy at udel.edu Sat Sep 27 01:16:40 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Sat Sep 27 01:16:48 2003 Subject: [Python-Dev] Re: Re: RELEASED Python 2.3.1 References: <200309270437.h8R4bxk0006927@localhost.localdomain> Message-ID: "Anthony Baxter" wrote in message news:200309270437.h8R4bxk0006927@localhost.localdomain... > I'm not sure what the term "revoke" would mean in this context. Stop distribution by removing from download page . Given that it works fine for some platforms and most uses on the rest, a warning message seems ok to me. TJR From anthony at interlink.com.au Sat Sep 27 01:16:10 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat Sep 27 01:18:18 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <20030926232840.GB17491@zot.electricrain.com> Message-ID: <200309270516.h8R5GAT7007511@localhost.localdomain> >>> "Gregory P. Smith" wrote > Its been my impression that the sporatic bsddb testsuite failures > are BerkeleyDB related rather than anything the python module can be > responsible for (other than the previously mentioned needed improvements > in cleaning up the on disk "temporary" test environment before launching > tests). For anyone reporting berkeleydb issues, its important to find > out the version of BerkeleyDB they're using and on what platform. I seem to recall some concern that using bsddb without a DBEnv being created could be causing problems - if this is the case, should we consider putting a deprecation warning into the code for people doing this? Anthony -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Sat Sep 27 01:43:48 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat Sep 27 01:46:11 2003 Subject: [Python-Dev] plans for 2.3.2 Message-ID: <200309270543.h8R5hmhH008225@localhost.localdomain> Ok, here's my plans for 2.3.2. Cut a release candidate on Tuesday (AU time, so Monday for merkins) Cut a release Thursday (AU time). I don't think the additional two days will hurt that badly, and I'd strongly prefer to have a release candidate before the final build. I don't know if the RC needs a windows installer built, as the problems are in non-windows platforms. Anthony From martin at v.loewis.de Sat Sep 27 02:55:33 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sat Sep 27 02:55:40 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> <20030926103846.A26782@discworld.dyndns.org> Message-ID: "Jason R. Mastaler" writes: > I'd be in favour of revoking 2.3.1 as well because of the severity of > this bug. I don't think the problem is severe; os.fsync is rarely used. It is just that people running into the problem are very vocal about it. Regards, Martin From martin at v.loewis.de Sat Sep 27 02:58:13 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sat Sep 27 02:58:44 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <200309270516.h8R5GAT7007511@localhost.localdomain> References: <200309270516.h8R5GAT7007511@localhost.localdomain> Message-ID: Anthony Baxter writes: > I seem to recall some concern that using bsddb without a DBEnv being > created could be causing problems - if this is the case, should we > consider putting a deprecation warning into the code for people > doing this? Not until we have determined with certainty that this *is* the case. Regards, Martin From martin at v.loewis.de Sat Sep 27 03:02:00 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sat Sep 27 03:02:16 2003 Subject: [Python-Dev] How to test for stuff like fsync? In-Reply-To: <200309270433.h8R4XKwl006833@localhost.localdomain> References: <200309270433.h8R4XKwl006833@localhost.localdomain> Message-ID: Anthony Baxter writes: > A thought that occurred to me a short time ago: write a script that outputs > the contents of each builtin module (the names in the module) to a file. > > Run that under previous version of python. Run it under release candidate. > Compare differences. In this specific case, it would have triggered a false alarm on some systems. I rewrote the test for fsync to overcome a build problem on some system, where fsync was present in the library, but not present in the header files. Since we are taking fsync's address, this causes a compile-time problem. So it *is* intentional that fsync is now gone for systems on which it was previously (2.2) present. (More specifically, fsync might disappear on systems that cannot stand _XOPEN_SOURCE being defined) Regards, Martin From jason at mastaler.com Sat Sep 27 03:21:51 2003 From: jason at mastaler.com (Jason R. Mastaler) Date: Sat Sep 27 03:21:58 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> <20030926103846.A26782@discworld.dyndns.org> Message-ID: martin@v.loewis.de (Martin v. L?wis) writes: > I don't think the problem is severe; os.fsync is rarely used. How did you arrive at this notion? From anthony at interlink.com.au Sat Sep 27 05:06:15 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat Sep 27 05:11:54 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: Message-ID: <200309270906.h8R96FCT010918@localhost.localdomain> >>> "Jason R. Mastaler" wrote > martin@v.loewis.de (Martin v. L?wis) writes: > > > I don't think the problem is severe; os.fsync is rarely used. > > How did you arrive at this notion? Hm. The only usage of it I can find in the Zope2, Zope3 and Twisted codebases is a single use inside zodb, and it's wrapped in a check that os.fsync exists (since it's not available on all platforms). I also did a grep over all the python packages I've downloaded or installed (which is a _lot_) and the TMDA code is the only package to use it. I'm still planning on cutting a 2.3.2 to fix the problem, but I see no reason to make any further steps to "revoke" 2.3.1 further than the warning message on the release page - which is mostly aimed at people who might be planning to package the code. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From martin at v.loewis.de Sat Sep 27 06:30:53 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sat Sep 27 06:31:08 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> <20030926103846.A26782@discworld.dyndns.org> Message-ID: "Jason R. Mastaler" writes: > > I don't think the problem is severe; os.fsync is rarely used. > > How did you arrive at this notion? >From four considerations: 1. It did not show up in any of the tests. 2. To use it, you have to use "bare" file descriptors; this is relatively un-pythonic. 3. To use it, you have to worry about data getting on the disk - this really requires some kind of expert application. 4. It is not universally available, so the expert application I had to assume in 3) would most likely test for presence of fsync, and fall back to just not use it (and perhaps invoke sync(1) instead). Regards, Martin From python at discworld.dyndns.org Sat Sep 27 12:09:40 2003 From: python at discworld.dyndns.org (Charles Cazabon) Date: Sat Sep 27 12:04:55 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: ; from martin@v.loewis.de on Sat, Sep 27, 2003 at 12:30:53PM +0200 References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> <20030926103846.A26782@discworld.dyndns.org> Message-ID: <20030927100940.E3923@discworld.dyndns.org> Martin v. L?wis wrote: > "Jason R. Mastaler" writes: > > > > I don't think the problem is severe; os.fsync is rarely used. > > > > How did you arrive at this notion? > > >From four considerations: [...] > 3. To use it, you have to worry about data getting on the disk - > this really requires some kind of expert application. Yes, like delivering mail, or writing to a database, or doing any of a thousand other tasks that use the filesystem to store and synchronize data. os.fsync() breakage is a /big/ problem, particularly because the resulting traceback makes users think the application is broken, not the interpreter. Charles -- ----------------------------------------------------------------------- Charles Cazabon GPL'ed software available at: http://www.qcc.ca/~charlesc/software/ ----------------------------------------------------------------------- From jason at mastaler.com Sat Sep 27 12:07:18 2003 From: jason at mastaler.com (Jason R. Mastaler) Date: Sat Sep 27 12:07:23 2003 Subject: [Python-Dev] Re: How to test for stuff like fsync? References: <16244.50825.588063.154695@montanaro.dyndns.org> Message-ID: Skip Montanaro writes: > I doubt that fsync() is available everywhere (otherwise why test for > it in the configure script?), so it seems that Jason should have > been testing for its presence (though see below) and worming around > its abscence if he wanted TMDA to run on as many platforms as > possible. fsync() is POSIX, and TMDA assumes a POSIX-complaint UNIX. Most mail servers tend to be such, which is probably why I've never had a report of an installation lacking fsync(). That's just me though, in general I see your point here. From jason at mastaler.com Sat Sep 27 12:14:48 2003 From: jason at mastaler.com (Jason R. Mastaler) Date: Sat Sep 27 12:14:54 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 References: <200309270906.h8R96FCT010918@localhost.localdomain> Message-ID: Anthony Baxter writes: > Hm. The only usage of it I can find in the Zope2, Zope3 and Twisted > codebases is a single use inside zodb, and it's wrapped in a check > that os.fsync exists (since it's not available on all platforms). I > also did a grep over all the python packages I've downloaded or > installed (which is a _lot_) and the TMDA code is the only package > to use it. Also, getmail, as Charles Cazabon mentioned earlier in this thread. Also, Barry is considering using fsync() in the next release of Mailman, albeit as an option turned off by default. TMDA has thousands of users, Mailman has many more. So, this potentially affects many individuals regardless of whether a large number of packages use fsync() or not. > I'm still planning on cutting a 2.3.2 to fix the problem, but I see > no reason to make any further steps to "revoke" 2.3.1 further than > the warning message on the release page - which is mostly aimed at > people who might be planning to package the code. Yeah, I think what you are doing is fine. If 2.3.2 is available soon, I think people will just tend to pass over 2.3.1. From martin at v.loewis.de Sat Sep 27 12:30:45 2003 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Sep 27 12:30:59 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <20030927100940.E3923@discworld.dyndns.org> References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> <20030926103846.A26782@discworld.dyndns.org> <20030927100940.E3923@discworld.dyndns.org> Message-ID: <3F75BB35.8090106@v.loewis.de> Charles Cazabon wrote: > os.fsync() breakage is a /big/ problem, particularly because the resulting > traceback makes users think the application is broken, not the interpreter. It's not broken: it's absent. In a sense, the application /is/ broken, as it should have dealt with os.fsync being absent in the first place. Applications should be prepared that POSIX functions are not available in Python even if the C library provides them on the system, as the system headers may fail to declare a prototype for the function (a problem which triggered that erroneous change in 2.3.1 in the first place). Python is taking the address of fsync, so merely having the function in the C library is not enough, nor is it to define fsync as a function-style macro. In either case, os.fsync will be rightfully absent. FWIW, it was a Redhat system that failed to provide a fsync prototype. Regards, Martin From guido at python.org Sat Sep 27 12:47:03 2003 From: guido at python.org (Guido van Rossum) Date: Sat Sep 27 12:47:43 2003 Subject: [Python-Dev] How to test for stuff like fsync? In-Reply-To: Your message of "27 Sep 2003 09:02:00 +0200." References: <200309270433.h8R4XKwl006833@localhost.localdomain> Message-ID: <200309271647.h8RGl3X23590@12-236-54-216.client.attbi.com> > In this specific case, it would have triggered a false alarm on some > systems. I rewrote the test for fsync to overcome a build problem on > some system, where fsync was present in the library, but not present > in the header files. Since we are taking fsync's address, this causes > a compile-time problem. So it *is* intentional that fsync is now gone > for systems on which it was previously (2.2) present. (More > specifically, fsync might disappear on systems that cannot stand > _XOPEN_SOURCE being defined) I don't like this. We're not taking fsync's address for a very good reason, just to save some code duplication. Removing fsync from those platforms just because our code-saving trick doesn't work there doesn't strike me as good service for those platforms. --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Sat Sep 27 13:08:14 2003 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Sep 27 13:08:22 2003 Subject: [Python-Dev] How to test for stuff like fsync? In-Reply-To: <200309271647.h8RGl3X23590@12-236-54-216.client.attbi.com> References: <200309270433.h8R4XKwl006833@localhost.localdomain> <200309271647.h8RGl3X23590@12-236-54-216.client.attbi.com> Message-ID: <3F75C3FE.2050408@v.loewis.de> Guido van Rossum wrote: > I don't like this. We're not taking fsync's address for a very good > reason, just to save some code duplication. Removing fsync from > those platforms just because our code-saving trick doesn't work there > doesn't strike me as good service for those platforms. It's not only fsync, though - there would be a lot of duplicated code if we were to convert all function pointer passing back to function calls. Also, if the system is lacking a function prototype, at the minimum, the compiler will give a warning - at worst, we invoke the function incorrectly, causing a crash. Regards, Martin From jeremy at zope.com Sat Sep 27 13:28:39 2003 From: jeremy at zope.com (Jeremy Hylton) Date: Sat Sep 27 13:29:48 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> <20030926103846.A26782@discworld.dyndns.org> Message-ID: <1064683719.19498.68.camel@localhost.localdomain> On Sat, 2003-09-27 at 06:30, Martin v. L?wis wrote: > "Jason R. Mastaler" writes: > > > > I don't think the problem is severe; os.fsync is rarely used. > > > > How did you arrive at this notion? > > >From four considerations: > > 1. It did not show up in any of the tests. > 2. To use it, you have to use "bare" file descriptors; this is > relatively un-pythonic. > 3. To use it, you have to worry about data getting on the disk - > this really requires some kind of expert application. > 4. It is not universally available, so the expert application I had > to assume in 3) would most likely test for presence of fsync, and > fall back to just not use it (and perhaps invoke sync(1) instead). I suppose ZODB is such an expert application. It has to cope with systems that do not provide fsync(), but it provides degraded service on such platforms. It is very important for the database to call fsync() when it commits a transaction. Jeremy From jason at mastaler.com Sat Sep 27 13:17:32 2003 From: jason at mastaler.com (Jason R. Mastaler) Date: Sat Sep 27 13:34:48 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> <20030926103846.A26782@discworld.dyndns.org> <20030927100940.E3923@discworld.dyndns.org> <3F75BB35.8090106@v.loewis.de> Message-ID: "Martin v. L?wis" writes: > It's not broken: it's absent. I'm not sure I see the distinction. The system provides fsync(), and the Python documentation says os.fsync() is available, but it isn't. Python therefore doesn't work as advertised. This is broken to me. From martin at v.loewis.de Sat Sep 27 13:48:48 2003 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Sep 27 13:49:00 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <1064683719.19498.68.camel@localhost.localdomain> References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> <20030926103846.A26782@discworld.dyndns.org> <1064683719.19498.68.camel@localhost.localdomain> Message-ID: <3F75CD80.3080008@v.loewis.de> Jeremy Hylton wrote: > I suppose ZODB is such an expert application. It has to cope with > systems that do not provide fsync(), but it provides degraded service on > such platforms. It is very important for the database to call fsync() > when it commits a transaction. I mostly agree: ZODB is indeed advanced, and it is indeed a good idea to check for presence of os.fsync before using it. While this is OT, I'd still like to question the usefulness of fsync(2) in the first place, for applications like ZODB. I assume fsync is used as a write barrier, to make sure old modifications are on disk, before making new changes. There are several cases in which this might be relevant: 1. The application will crash soon after performing fsync, leaving data potentially in an inconsistent state. Here, using fsync is not necessary, as the system will still perform all modifications on disk, even though the process has long terminated. 2. The application will be kill(2)ed soon after fsync completes (or even while fsync completes). Like 1), fsync is not needed. 3. There is an operating system crash (kernel panic or similar). fsync does not help, as, for a buggy kernel, anything might have happened to the data before. 4. There is a disk failure. fsync does not help, as the data on disk might not be recoverable. 5. There is a power outage. This is the case where fsync should help: everything up to the write barrier is on disk. Of course, if the disk drive itself has write caching, fsync may have completed without the data being on the disk (this would be an fsync bug, but I believe Linux suffers from this particular bug). So in short, fsync(2) helps only in case of a power outage; for normal operation, it is not needed. In the case of a power outage, it is doubtful whether it has the desired effect. Slightly more on-topic: os.fsync is even worse, as it cannot be used to implement a write barrier in case of multiple threads. It runs with the GIL released, so while fsync is running, other threads might change the file. The semantics of fsync in this case is pretty unclear, but it is likely not a write barrier. Regards, Martin From skip at pobox.com Sat Sep 27 14:11:07 2003 From: skip at pobox.com (Skip Montanaro) Date: Sat Sep 27 14:12:23 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: References: <200309270516.h8R5GAT7007511@localhost.localdomain> Message-ID: <16245.53947.33728.86658@montanaro.dyndns.org> Martin> Anthony Baxter writes: >> I seem to recall some concern that using bsddb without a DBEnv being >> created could be causing problems - if this is the case, should we >> consider putting a deprecation warning into the code for people doing >> this? Martin> Not until we have determined with certainty that this *is* the Martin> case. According to info I got from Sleepycat, a DBEnv is required for using in a multi-threaded environment. Since the legacy bsddb module API is widely used and use of threading has increased in the past couple years, I think we need to figure out how to solve that problem. This is just a wild-ass guess (I think I posted something like it before), but maybe all that's needed is to extend the bsddb.(bt|hash|rn)open functions to accept a dbenv arg, define a module-level default environment in bsddb/__init__.py which is used as the dbenv arg if the caller doesn't provide one. The __init__.py code would look like this: _env = db.DBEnv() def hashopen(file, flag='c', mode=0666, pgsize=None, ffactor=None, nelem=None, cachesize=None, lorder=None, hflags=0, dbenv=_env): flags = _checkflag(flag) d = db.DB(dbenv) ... def btopen(file, flag='c', mode=0666, btflags=0, cachesize=None, maxkeypage=None, minkeypage=None, pgsize=None, lorder=None, dbenv=_env): flags = _checkflag(flag) d = db.DB(dbenv) ... def rnopen(file, flag='c', mode=0666, rnflags=0, cachesize=None, pgsize=None, lorder=None, rlen=None, delim=None, source=None, pad=None, dbenv=_env): flags = _checkflag(flag) d = db.DB(dbenv) ... My reading of the bsddb3 docs at suggests that should be sufficient (though certain args may need to be passed to the DBEnv() call). See . Attached is a modified version of the hammer.py script which seems to not fail for me on either Windows run from IDLE (Python 2.3, BDB 4.1.6) or Mac OS X (Python CVS, BDB 4.2.1). The original script failed for me on Windows but not Mac OS X. Can some other people for whom the original script fails please try it? (I also attached it to bug #775414.) Skip -------------- next part -------------- A non-text attachment was scrubbed... Name: studly_hammer.py Type: application/octet-stream Size: 1818 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20030927/b92fcc07/studly_hammer-0001.obj From theller at python.net Sat Sep 27 15:02:44 2003 From: theller at python.net (Thomas Heller) Date: Sat Sep 27 15:03:01 2003 Subject: [Python-Dev] Re: plans for 2.3.2 In-Reply-To: <200309270543.h8R5hmhH008225@localhost.localdomain> (Anthony Baxter's message of "Sat, 27 Sep 2003 15:43:48 +1000") References: <200309270543.h8R5hmhH008225@localhost.localdomain> Message-ID: <1xu2vyuj.fsf@python.net> Anthony Baxter writes: > Ok, here's my plans for 2.3.2. > > Cut a release candidate on Tuesday (AU time, so Monday for merkins) > > Cut a release Thursday (AU time). > > I don't think the additional two days will hurt that badly, and I'd > strongly prefer to have a release candidate before the final build. Will the release candidate have the same version number as the final version? > I don't know if the RC needs a windows installer built, as the problems > are in non-windows platforms. I don't mind building one. Thomas From fdrake at acm.org Sat Sep 27 15:10:52 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sat Sep 27 15:11:08 2003 Subject: [Python-Dev] Re: plans for 2.3.2 In-Reply-To: <1xu2vyuj.fsf@python.net> References: <200309270543.h8R5hmhH008225@localhost.localdomain> <1xu2vyuj.fsf@python.net> Message-ID: <16245.57532.609456.554472@grendel.zope.com> Thomas Heller writes: > Will the release candidate have the same version number as the final > version? The typical case has been that the candidate has a distinct version number. I think 2.3.1 has taught us to be very careful, so we should keep the distinction between the candidate and the actual release. My opinion: What's "candidate" isn't the actual file we post to python.org, but the bits in CVS that form the implementation and build process. The actual packages are a byproduct of that. > > I don't know if the RC needs a windows installer built, as the problems > > are in non-windows platforms. > > I don't mind building one. I think there should be one. They've paid off in the past. I'm planning to post the 2.3.2c1 documentation late Sunday night or early Monday morning. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From theller at python.net Sat Sep 27 15:16:50 2003 From: theller at python.net (Thomas Heller) Date: Sat Sep 27 15:17:06 2003 Subject: [Python-Dev] [development doc updates] In-Reply-To: <20030927190724.DDC8818F3E6@grendel.zope.com> (Fred L. Drake's message of "Sat, 27 Sep 2003 15:07:24 -0400 (EDT)") References: <20030927190724.DDC8818F3E6@grendel.zope.com> Message-ID: "Fred L. Drake" writes: > The development version of the documentation has been updated: > > http://www.python.org/dev/doc/devel/ > > The new glossary in the tutorial is really nice! Sure, but the hyperlinks don't work ;-) Thomas From theller at python.net Sat Sep 27 15:17:46 2003 From: theller at python.net (Thomas Heller) Date: Sat Sep 27 15:17:59 2003 Subject: [Python-Dev] Re: plans for 2.3.2 In-Reply-To: <1xu2vyuj.fsf@python.net> (Thomas Heller's message of "Sat, 27 Sep 2003 21:02:44 +0200") References: <200309270543.h8R5hmhH008225@localhost.localdomain> <1xu2vyuj.fsf@python.net> Message-ID: Thomas Heller writes: > Will the release candidate have the same version number as the final > version? Never mind, I just missed Fred's checkin to ptachlevel. Thomas From jafo at tummy.com Sat Sep 27 15:19:06 2003 From: jafo at tummy.com (Sean Reifschneider) Date: Sat Sep 27 15:19:37 2003 Subject: [Python-Dev] Re: plans for 2.3.2 In-Reply-To: <200309270543.h8R5hmhH008225@localhost.localdomain> References: <200309270543.h8R5hmhH008225@localhost.localdomain> Message-ID: <20030927191906.GB17740@tummy.com> On Sat, Sep 27, 2003 at 03:43:48PM +1000, Anthony Baxter wrote: >Ok, here's my plans for 2.3.2. >Cut a release candidate on Tuesday (AU time, so Monday for merkins) >Cut a release Thursday (AU time). That sounds fine, I can do a test build and put up sample RPMs for the RC, and get the final pushed up on Thursday-ish. Sean -- Men are from Mars. Women are from Venus. Computers are from hell. Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995. Qmail, Python, SysAdmin Back off man. I'm a scientist. http://HackingSociety.org/ From theller at python.net Sat Sep 27 15:21:26 2003 From: theller at python.net (Thomas Heller) Date: Sat Sep 27 15:21:40 2003 Subject: [Python-Dev] Re: plans for 2.3.2 In-Reply-To: <16245.57532.609456.554472@grendel.zope.com> (Fred L. Drake, Jr.'s message of "Sat, 27 Sep 2003 15:10:52 -0400") References: <200309270543.h8R5hmhH008225@localhost.localdomain> <1xu2vyuj.fsf@python.net> <16245.57532.609456.554472@grendel.zope.com> Message-ID: "Fred L. Drake, Jr." writes: > Thomas Heller writes: > > Will the release candidate have the same version number as the final > > version? > > The typical case has been that the candidate has a distinct version > number. I think 2.3.1 has taught us to be very careful, so we should > keep the distinction between the candidate and the actual release. > > My opinion: What's "candidate" isn't the actual file we post to > python.org, but the bits in CVS that form the implementation and build > process. The actual packages are a byproduct of that. We've seen with 2.3.1 that this is only theory, at least in part ;-) > > > I don't know if the RC needs a windows installer built, as the problems > > > are in non-windows platforms. > > > > I don't mind building one. > > I think there should be one. They've paid off in the past. Thomas From martin at v.loewis.de Sat Sep 27 16:39:32 2003 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Sep 27 16:39:49 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <16245.53947.33728.86658@montanaro.dyndns.org> References: <200309270516.h8R5GAT7007511@localhost.localdomain> <16245.53947.33728.86658@montanaro.dyndns.org> Message-ID: <3F75F584.8040207@v.loewis.de> Skip Montanaro wrote: > According to info I got from Sleepycat, a DBEnv is required for using in a > multi-threaded environment. Since the legacy bsddb module API is widely > used and use of threading has increased in the past couple years, I think we > need to figure out how to solve that problem. This is just a wild-ass guess > (I think I posted something like it before), but maybe all that's needed is > to extend the bsddb.(bt|hash|rn)open functions to accept a dbenv arg, define > a module-level default environment in bsddb/__init__.py which is used as the > dbenv arg if the caller doesn't provide one. The __init__.py code would > look like this: > > _env = db.DBEnv() I think we should define a set of flags for the "common case". In general, multiple threads may write to the same database, as might multiple instances of the application. IOW, we might want to create the environment with DB_INIT_CDB|DB_INIT_MPOOL|DB_THREAD. Applications that don't want to suffer from the possible serialization of CDB would need to use their own environment. Regards, Martin From theller at python.net Sat Sep 27 16:40:28 2003 From: theller at python.net (Thomas Heller) Date: Sat Sep 27 16:40:42 2003 Subject: [Python-Dev] PC/python_nt.rc Message-ID: The file PC/python_nt.rc defines the version information which will be embedded as resource in pythonXX.dll. Because of limitations in MS resource compiler, the info cannot be constructed from patchlevel.h at compile time, instead it must be changed manually. There is a python script PC/field3.py which helps, but it cannot be used by the build process because python is then not available. I'm working on a way to automate this: A new project is inserted into the MSVC project file, and the pythoncore project (which builds the python dll) will depend on it, so this new project is built first. This new project builds a small exe which will create (as custom build step) an include file for python_nt.rc, so no manual step is needed any more. Does this sound like a sensible approach, or are there any objections? Thomas From guido at python.org Sat Sep 27 16:49:54 2003 From: guido at python.org (Guido van Rossum) Date: Sat Sep 27 16:50:36 2003 Subject: [Python-Dev] How to test for stuff like fsync? In-Reply-To: Your message of "Sat, 27 Sep 2003 19:08:14 +0200." <3F75C3FE.2050408@v.loewis.de> References: <200309270433.h8R4XKwl006833@localhost.localdomain> <200309271647.h8RGl3X23590@12-236-54-216.client.attbi.com> <3F75C3FE.2050408@v.loewis.de> Message-ID: <200309272049.h8RKnso23818@12-236-54-216.client.attbi.com> > Guido van Rossum wrote: > > I don't like this. We're not taking fsync's address for a very good > > reason, just to save some code duplication. Removing fsync from > > those platforms just because our code-saving trick doesn't work there > > doesn't strike me as good service for those platforms. [MvL] > It's not only fsync, though - there would be a lot of duplicated code if > we were to convert all function pointer passing back to function calls. Well, too bad. We could use macros instead. I believe I started taking the function address many years ago, and I just did it as a trick to save some typing, not to save code size. I think it's wrong to use it as an argument against including a function on a platform because the platform doesn't support that trick. > Also, if the system is lacking a function prototype, at the minimum, the > compiler will give a warning - at worst, we invoke the function > incorrectly, causing a crash. For fsync(), invoking it wrongly sounds unlikely. I'm sure the users of fsync() would gladly take the warning if they can have the function. Is there really no function prototype in the system headers for fsync() on those systems, or don't we see it because we're using feature selection macros (isn't that the right term?) that hide it? --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at electricrain.com Sat Sep 27 16:51:47 2003 From: greg at electricrain.com (Gregory P. Smith) Date: Sat Sep 27 16:51:51 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <3F75F584.8040207@v.loewis.de> References: <200309270516.h8R5GAT7007511@localhost.localdomain> <16245.53947.33728.86658@montanaro.dyndns.org> <3F75F584.8040207@v.loewis.de> Message-ID: <20030927205147.GH17491@zot.electricrain.com> > >According to info I got from Sleepycat, a DBEnv is required for using in a > >multi-threaded environment. Since the legacy bsddb module API is widely > >used and use of threading has increased in the past couple years, I think > >we > >need to figure out how to solve that problem. This is just a wild-ass > >guess > >(I think I posted something like it before), but maybe all that's needed is > >to extend the bsddb.(bt|hash|rn)open functions to accept a dbenv arg, > >define > >a module-level default environment in bsddb/__init__.py which is used as > >the > >dbenv arg if the caller doesn't provide one. The __init__.py code would > >look like this: > > > > _env = db.DBEnv() > > I think we should define a set of flags for the "common case". In > general, multiple threads may write to the same database, as might > multiple instances of the application. IOW, we might want to create > the environment with DB_INIT_CDB|DB_INIT_MPOOL|DB_THREAD. It is worth noting that using a DBEnv in this manner will create a bunch of auxilary DBEnv related files on the disk. That has the potential of confusing people who expect only the database file. It also means that separate unrelated databases cannot exist in the same directory without being part of the same DBEnv (which BerkeleyDB multiprocess access should handle just fine; but it might not be what people expect). > Applications that don't want to suffer from the possible > serialization of CDB would need to use their own environment. An alternative would be to say that applications that want to use bsddb with threading need to use the *real* BerkeleyDB API rather than the ancient compatibility interface. (it'd be easy to check that bsddb doesn't get used by multiple threads, raising an exception if it does) greg From theller at python.net Sat Sep 27 16:51:52 2003 From: theller at python.net (Thomas Heller) Date: Sat Sep 27 16:52:07 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <2mzngr1bf2.fsf@starship.python.net> (Michael Hudson's message of "Fri, 26 Sep 2003 22:34:41 +0100") References: <200309261752.h8QHqJgD029823@localhost.localdomain> <20030926184853.GB22837@mems-exchange.org> <2mzngr1bf2.fsf@starship.python.net> Message-ID: <8yoauf87.fsf@python.net> [cc to webmaster@starship.python.net] Michael Hudson writes: > Thomas Heller writes: > >> And it would help if I could build the HTML docs myself from CVS. I did >> manage to create the pdf files with TeTex under windows, but I didn't >> succeed with the html pages so far. > > It occurs to me that I don't know *why* Fred is so much the > documentation man; I've not had any trouble processing the docs into > HTML lately (haven't tried on Windows, admittedly, and I haven't > tried to make info ever). I could do it myself if the tools needed to build the docs would be available on the starship. Last time I checked, neither Tex nor latex2html were installed. So, I hereby kindly request: Could the tools to build the Python docs from source please be installed on the starship? Thanks, Thomas From martin at v.loewis.de Sat Sep 27 16:57:36 2003 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Sep 27 16:57:49 2003 Subject: [Python-Dev] PC/python_nt.rc In-Reply-To: References: Message-ID: <3F75F9C0.8040504@v.loewis.de> Thomas Heller wrote: > Does this sound like a sensible approach, or are there any objections? Would it be possible to use VBScript instead? That should be available at any phase of the build process. While we are at it: At what point will we switch to VC7.1 for the Windows builds? Regards, Martin From martin at v.loewis.de Sat Sep 27 17:12:17 2003 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Sep 27 17:12:39 2003 Subject: [Python-Dev] How to test for stuff like fsync? In-Reply-To: <200309272049.h8RKnso23818@12-236-54-216.client.attbi.com> References: <200309270433.h8R4XKwl006833@localhost.localdomain> <200309271647.h8RGl3X23590@12-236-54-216.client.attbi.com> <3F75C3FE.2050408@v.loewis.de> <200309272049.h8RKnso23818@12-236-54-216.client.attbi.com> Message-ID: <3F75FD31.4010103@v.loewis.de> Guido van Rossum wrote: > Is there really no function prototype in the system headers for > fsync() on those systems, or don't we see it because we're using > feature selection macros (isn't that the right term?) that hide it? I really can't answer that for all systems that lost fsync; I only know about a single one. I earlier said it was Redhat, but I recalled that incorrectly; it was "a 2-year-old Linux box running slackware" (no idea what slackware version). It is also unclear what specific version of the C library headers where installed on the system; the compiler is gcc 2.7.2. On that system, fsync is only available if __USE_BSD is defined, see python.org/sf/800710. So on the surface, it looks like a feature selection macro which we fail to define (_BSD_SOURCE), instead of a feature selection macro that we do define. OTOH, it might be that the definition of _XOPEN_SOURCE on that system would cause __USE_BSD not to be defined - I don't have a copy of features.h of the particular system. In recent glibc versions, fsync is declared if either __USE_BSD or __USE_XOPEN is defined. Regards, Martin From theller at python.net Sat Sep 27 17:15:11 2003 From: theller at python.net (Thomas Heller) Date: Sat Sep 27 17:15:26 2003 Subject: [Python-Dev] PC/python_nt.rc In-Reply-To: <3F75F9C0.8040504@v.loewis.de> ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "Sat, 27 Sep 2003 22:57:36 +0200") References: <3F75F9C0.8040504@v.loewis.de> Message-ID: <1xu2ue5c.fsf@python.net> "Martin v. L?wis" writes: > Thomas Heller wrote: >> Does this sound like a sensible approach, or are there any objections? > > Would it be possible to use VBScript instead? That should be available > at any phase of the build process. I'm much more comfortable to write the following C program than to parse patchlevel.h in VB. What would the advantage of VB be? #include #include "patchlevel.h" int main(int argc, char **argv) { printf("#define FIELD3 %d\n", PY_MICRO_VERSION*1000 + PY_RELEASE_LEVEL*10 + PY_RELEASE_SERIAL); /* some more fields needed */ return 0; } > > While we are at it: At what point will we switch to VC7.1 for the > Windows builds? Hm, for 2.4, maybe. Or should there 7.1 compiled .msi installers in parallel for the 2.3 line? Thomas From martin at v.loewis.de Sat Sep 27 17:17:46 2003 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Sep 27 17:17:57 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <20030927205147.GH17491@zot.electricrain.com> References: <200309270516.h8R5GAT7007511@localhost.localdomain> <16245.53947.33728.86658@montanaro.dyndns.org> <3F75F584.8040207@v.loewis.de> <20030927205147.GH17491@zot.electricrain.com> Message-ID: <3F75FE7A.6070800@v.loewis.de> Gregory P. Smith wrote: > It is worth noting that using a DBEnv in this manner will create a bunch > of auxilary DBEnv related files on the disk. That has the potential of > confusing people who expect only the database file. It also means that > separate unrelated databases cannot exist in the same directory without > being part of the same DBEnv (which BerkeleyDB multiprocess access should > handle just fine; but it might not be what people expect). I see. I could accept that it is confusing; would it also be backwards compatible (i.e. would BerkeleyDB create those files on demand, and would old Python installation still be able to read the database even if those files where around)? > An alternative would be to say that applications that want to use bsddb > with threading need to use the *real* BerkeleyDB API rather than the > ancient compatibility interface. (it'd be easy to check that bsddb > doesn't get used by multiple threads, raising an exception if it does) But is it also easy to detect that multiple applications try to use the same database? Regards, Martin From greg at electricrain.com Sat Sep 27 17:33:32 2003 From: greg at electricrain.com (Gregory P. Smith) Date: Sat Sep 27 17:33:40 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <3F75FE7A.6070800@v.loewis.de> References: <200309270516.h8R5GAT7007511@localhost.localdomain> <16245.53947.33728.86658@montanaro.dyndns.org> <3F75F584.8040207@v.loewis.de> <20030927205147.GH17491@zot.electricrain.com> <3F75FE7A.6070800@v.loewis.de> Message-ID: <20030927213332.GI17491@zot.electricrain.com> On Sat, Sep 27, 2003 at 11:17:46PM +0200, "Martin v. L?wis" wrote: > Gregory P. Smith wrote: > > >It is worth noting that using a DBEnv in this manner will create a bunch > >of auxilary DBEnv related files on the disk. That has the potential of > >confusing people who expect only the database file. It also means that > >separate unrelated databases cannot exist in the same directory without > >being part of the same DBEnv (which BerkeleyDB multiprocess access should > >handle just fine; but it might not be what people expect). > > I see. I could accept that it is confusing; would it also be backwards > compatible (i.e. would BerkeleyDB create those files on demand, and > would old Python installation still be able to read the database even > if those files where around)? It would not. Its already not backwards compatible. Thats what the bsddb185 module is for. The BerkeleyDB file format changed between 3.1, 3.2 and 4.0 even. BerkeleyDB can upgrade from an older format to the current one (using the DB.upgrade method). > >An alternative would be to say that applications that want to use bsddb > >with threading need to use the *real* BerkeleyDB API rather than the > >ancient compatibility interface. (it'd be easy to check that bsddb > >doesn't get used by multiple threads, raising an exception if it does) > > But is it also easy to detect that multiple applications try to use > the same database? No. But it is easy to document. The old bsddb module never allowed it either. I only suggest detecting multithreaded access because it is possible to do. Anyone who wants multiprocess access should use the real DBEnv+DB interface and know what they're doing. -g From skip at pobox.com Sat Sep 27 15:50:49 2003 From: skip at pobox.com (Skip Montanaro) Date: Sat Sep 27 17:55:18 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: References: <200309270906.h8R96FCT010918@localhost.localdomain> Message-ID: <16245.59929.264386.827541@montanaro.dyndns.org> Jason> Also, Barry is considering using fsync() in the next release of Jason> Mailman, albeit as an option turned off by default. Something tells me Barry will probably guard its use with a hasattr(). ;-) Skip From skip at pobox.com Sat Sep 27 18:07:57 2003 From: skip at pobox.com (Skip Montanaro) Date: Sat Sep 27 18:08:11 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <3F75F584.8040207@v.loewis.de> References: <200309270516.h8R5GAT7007511@localhost.localdomain> <16245.53947.33728.86658@montanaro.dyndns.org> <3F75F584.8040207@v.loewis.de> Message-ID: <16246.2621.472392.671642@montanaro.dyndns.org> Martin> Applications that don't want to suffer from the possible Martin> serialization of CDB would need to use their own environment. I've given that a little more thought. Instead of encumbering the factory functions which implement the old API with an optional dbenv argument, I think if people want to provide their own environment it's reasonable to expect them to use the new bsddb3 API. All we should do with the old API is make it work in a multi-threaded environment if possible. Skip From skip at pobox.com Sat Sep 27 18:09:37 2003 From: skip at pobox.com (Skip Montanaro) Date: Sat Sep 27 18:09:49 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <20030927205147.GH17491@zot.electricrain.com> References: <200309270516.h8R5GAT7007511@localhost.localdomain> <16245.53947.33728.86658@montanaro.dyndns.org> <3F75F584.8040207@v.loewis.de> <20030927205147.GH17491@zot.electricrain.com> Message-ID: <16246.2721.881433.68068@montanaro.dyndns.org> Greg> An alternative would be to say that applications that want to use Greg> bsddb with threading need to use the *real* BerkeleyDB API rather Greg> than the ancient compatibility interface. (it'd be easy to check Greg> that bsddb doesn't get used by multiple threads, raising an Greg> exception if it does) Well, perhaps a warning. I doubt you can tell if the programmer has provided his own locks around db accesses. Skip From greg at electricrain.com Sat Sep 27 19:09:07 2003 From: greg at electricrain.com (Gregory P. Smith) Date: Sat Sep 27 19:09:11 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <16246.2621.472392.671642@montanaro.dyndns.org> References: <200309270516.h8R5GAT7007511@localhost.localdomain> <16245.53947.33728.86658@montanaro.dyndns.org> <3F75F584.8040207@v.loewis.de> <16246.2621.472392.671642@montanaro.dyndns.org> Message-ID: <20030927230907.GJ17491@zot.electricrain.com> On Sat, Sep 27, 2003 at 05:07:57PM -0500, Skip Montanaro wrote: > > Martin> Applications that don't want to suffer from the possible > Martin> serialization of CDB would need to use their own environment. > > I've given that a little more thought. Instead of encumbering the factory > functions which implement the old API with an optional dbenv argument, I > think if people want to provide their own environment it's reasonable to > expect them to use the new bsddb3 API. All we should do with the old API is > make it work in a multi-threaded environment if possible. > > Skip Agreed, that sounds like a good option. In my previous email I had forgotten about the DB_PRIVATE flag to DBEnv objects; that prevents them from writing extra DBEnv files to the filesystem for use when multi-process access is not needed. I just committed the small change needed to bsddb/__init__.py for it to use a DBEnv allowing multithreaded access. The original hammer.py from bug 775414 has been running for 15 minutes without problems on my alpha with BerkeleyDB 4.1.25. -g From tim.one at comcast.net Sat Sep 27 21:47:05 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Sep 27 21:53:13 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <1064683719.19498.68.camel@localhost.localdomain> Message-ID: [Jeremy Hylton] > I suppose ZODB is such an expert application. It has to cope with > systems that do not provide fsync(), but it provides degraded service > on such platforms. It is very important for the database to call > fsync() when it commits a transaction. I'm not sure that ZODB *intends* to run on any system w/o fsync anymore. It had to before 2.3 because os.fsync() didn't exist on Windows, and ZODB has to run on Windows. I implemented os.fysnc() for Windows in 2.3 precisely so that ZODB would work better on Windows. More mundanely, as I recall, we were having a specific problem wherein a backup script wasn't getting the correct size for a ZODB Data.fs file on Windows, and file.flush()'ing Data.fs didn't help enough -- backup kept screwing up until FileStorage's if fsync is not None: fsync(file.fileno()) actually did something on Windows. From guido at python.org Sat Sep 27 22:03:38 2003 From: guido at python.org (Guido van Rossum) Date: Sat Sep 27 22:04:17 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: Your message of "Sat, 27 Sep 2003 21:47:05 EDT." References: Message-ID: <200309280203.h8S23cA24143@12-236-54-216.client.attbi.com> > [Jeremy Hylton] > > I suppose ZODB is such an expert application. It has to cope with > > systems that do not provide fsync(), but it provides degraded service > > on such platforms. It is very important for the database to call > > fsync() when it commits a transaction. [Tim] > I'm not sure that ZODB *intends* to run on any system w/o fsync anymore. It > had to before 2.3 because os.fsync() didn't exist on Windows, and ZODB has > to run on Windows. I implemented os.fysnc() for Windows in 2.3 precisely so > that ZODB would work better on Windows. > > More mundanely, as I recall, we were having a specific problem wherein a > backup script wasn't getting the correct size for a ZODB Data.fs file on > Windows, and file.flush()'ing Data.fs didn't help enough -- backup kept > screwing up until FileStorage's > > if fsync is not None: fsync(file.fileno()) > > actually did something on Windows. Bizarre. On Unix, fsync()'s only point is to do something that is only ever noticeable if there's an OS or hardware failure. But on Windows, it was needed to get the OS to synchronize the directory contents so that another process or thread would see the right thing. Anyway, I think that on Unix, that problem doesn't exist, and there the only problem of not having fsync() is the minuscule risk of hardware or OS failure right after you wrote the data but before it actually was written to the media. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Sat Sep 27 22:08:31 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Sep 27 22:08:38 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <3F75CD80.3080008@v.loewis.de> Message-ID: [Martin v. L?wis] > I mostly agree: ZODB is indeed advanced, and it is indeed a good idea > to check for presence of os.fsync before using it. > > While this is OT, I'd still like to question the usefulness of > fsync(2) in the first place, for applications like ZODB. I assume > fsync is used as a write barrier, to make sure old modifications are > on disk, before making new changes. That's important, but not primarily for the catastrophic error-recovery scenarios you go on to sketch. The most important error-recovery procedure is preventative, running backups against a ZODB database while the database is active (Tools/repozo.py in a recent ZODB distribution is the right tool for this). Since a ZODB process may run for months, it's not practical to say that backups require shutting ZODB down. Without both flushing and fsync'ing, the backup process can't get at all the data that's "really" in the file. Here's a little Python driver: import os fp = file('test.dat', 'wb') guts = 'x' * 1000000 n = 0 while 1: fp.write(guts) fp.flush() os.fsync(fp.fileno()) n += len(guts) proceed = raw_input("wrote %d so far; hit key" % n) At least on Windows, both the flush and the fsync are necessary to see one million bytes (via a different process) at the first prompt, two million at the second, and so on. With neither, another process typically sees 0 bytes before the file gets huge. With just one of them, it seems hard to predict, ranging from 0 to "almost" a million additional bytes per prompt. ZODB does a flush and an fsync after each transaction, so that the backup script (or any other distinct process) sees the most recent data available. Besides missing large gobs of newer data, without the fsync the backup script may see incomplete data for the most recent transaction that managed to wind up on disk, and erroneously conclude that the database is corrupted. In short, fsync is necessary to support ZODB best practice. From barry at python.org Sun Sep 28 00:42:30 2003 From: barry at python.org (Barry Warsaw) Date: Sun Sep 28 00:42:38 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <16246.2621.472392.671642@montanaro.dyndns.org> References: <200309270516.h8R5GAT7007511@localhost.localdomain> <16245.53947.33728.86658@montanaro.dyndns.org> <3F75F584.8040207@v.loewis.de> <16246.2621.472392.671642@montanaro.dyndns.org> Message-ID: <1064724149.31604.83.camel@anthem> On Sat, 2003-09-27 at 18:07, Skip Montanaro wrote: > Martin> Applications that don't want to suffer from the possible > Martin> serialization of CDB would need to use their own environment. > > I've given that a little more thought. Instead of encumbering the factory > functions which implement the old API with an optional dbenv argument, I > think if people want to provide their own environment it's reasonable to > expect them to use the new bsddb3 API. All we should do with the old API is > make it work in a multi-threaded environment if possible. +1 -Barry From itamar at itamarst.org Sun Sep 28 00:52:52 2003 From: itamar at itamarst.org (Itamar Shtull-Trauring) Date: Sun Sep 28 00:53:03 2003 Subject: [Python-Dev] Procedures for submitting patches to pybsddb Message-ID: <20030928005252.6fd1b4b9.itamar@itamarst.org> I have a patch (DBCursor.get_current_size(), returns size of data for current entry) which I'd like to submit. This involves changes to pybsddb cvs as well as python cvs, from what I can tell (for tests and docs in the pybsddb repo). While I have developer access to pybsddb, I don't have it for Python. Submitting patches for two different repositories seems cumbersome, so where should I add it? Python SF tracker? -- Itamar Shtull-Trauring http://itamarst.org/ Available for Python & Twisted consulting From barry at python.org Sun Sep 28 01:01:19 2003 From: barry at python.org (Barry Warsaw) Date: Sun Sep 28 01:01:27 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: References: <200309270906.h8R96FCT010918@localhost.localdomain> Message-ID: <1064725279.31604.97.camel@anthem> On Sat, 2003-09-27 at 12:14, Jason R.Mastaler wrote: > Also, Barry is considering using fsync() in the next release of > Mailman, albeit as an option turned off by default. It'll be in Mailman 2.1.3, but turned off by default. I plan on adding something to the release notes about it, warning that it may not be available on every system (although we don't have to worry about Windows ). Its use shouldn't be enabled if it's not available. -Barry From anthony at interlink.com.au Sun Sep 28 01:03:39 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Sun Sep 28 01:05:55 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <1064724149.31604.83.camel@anthem> Message-ID: <200309280503.h8S53e3P005335@localhost.localdomain> >>> Barry Warsaw wrote > On Sat, 2003-09-27 at 18:07, Skip Montanaro wrote: > > I've given that a little more thought. Instead of encumbering the factory > > functions which implement the old API with an optional dbenv argument, I > > think if people want to provide their own environment it's reasonable to > > expect them to use the new bsddb3 API. All we should do with the old API is > > make it work in a multi-threaded environment if possible. > > +1 > -Barry I'm +1 on this as well, but with the caveat that if we _can't_ make it work reliably in a MT environment, we should either decline the opportunity to mangle the user's data, or at _least_ issue a warning. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From barry at python.org Sun Sep 28 01:06:55 2003 From: barry at python.org (Barry Warsaw) Date: Sun Sep 28 01:07:03 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <16245.59929.264386.827541@montanaro.dyndns.org> References: <200309270906.h8R96FCT010918@localhost.localdomain> <16245.59929.264386.827541@montanaro.dyndns.org> Message-ID: <1064725614.31604.99.camel@anthem> On Sat, 2003-09-27 at 15:50, Skip Montanaro wrote: > Jason> Also, Barry is considering using fsync() in the next release of > Jason> Mailman, albeit as an option turned off by default. > > Something tells me Barry will probably guard its use with a hasattr(). ;-) Actually, no, but as of right now, it has to be explicitly enabled, so it shouldn't be when there's no fsync . -Barry From barry at python.org Sun Sep 28 01:24:27 2003 From: barry at python.org (Barry Warsaw) Date: Sun Sep 28 01:24:35 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <3F75CD80.3080008@v.loewis.de> References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> <20030926103846.A26782@discworld.dyndns.org> <1064683719.19498.68.camel@localhost.localdomain> <3F75CD80.3080008@v.loewis.de> Message-ID: <1064726667.31604.105.camel@anthem> On Sat, 2003-09-27 at 13:48, "Martin v. L?wis" wrote: > 5. There is a power outage. This is the case where fsync should help: > everything up to the write barrier is on disk. Of course, if the > disk drive itself has write caching, fsync may have completed > without the data being on the disk (this would be an fsync bug, but > I believe Linux suffers from this particular bug). > > So in short, fsync(2) helps only in case of a power outage; for normal > operation, it is not needed. In the case of a power outage, it is > doubtful whether it has the desired effect. This was the situation that folks reported causing message loss in Mailman 2.1.2. Those folks also reported that adding the fsync helped them (I guess a UPS would have helped too ). I've added it to MM2.1.3, but I haven't enabled it by default. I'm still not sure the specific problem can be fixed in better ways (e.g. running on a more reliable host, running it on a sync'ing filesystem, or maybe opening the file with O_SYNC). Also, I did some benchmarking with fsync's added and on the system I tested I saw a 97% performance hit when always fsync'ing. That made me nervous. Before I enable it unconditionally, I'd want to be more certain that it's necessary, or useful for the majority of sites, and that the performance hit is either worth it, not as bad as my benchmarks showed, or at least not as observably bad under real-world conditions. -Barry From barry at python.org Sun Sep 28 01:26:46 2003 From: barry at python.org (Barry Warsaw) Date: Sun Sep 28 01:26:56 2003 Subject: [Python-Dev] Procedures for submitting patches to pybsddb In-Reply-To: <20030928005252.6fd1b4b9.itamar@itamarst.org> References: <20030928005252.6fd1b4b9.itamar@itamarst.org> Message-ID: <1064726806.31604.107.camel@anthem> On Sun, 2003-09-28 at 00:52, Itamar Shtull-Trauring wrote: > While I have developer access to pybsddb, I don't have it for Python. > Submitting patches for two different repositories seems cumbersome, so > where should I add it? Python SF tracker? Yes, because the pybsddb project weaves the actual bsddb files from the Python project. pybsddb is now just a distutils/documentation shell project. -Barry From pf_moore at yahoo.co.uk Sun Sep 28 07:47:31 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Sun Sep 28 07:47:32 2003 Subject: [Python-Dev] Re: PC/python_nt.rc References: <3F75F9C0.8040504@v.loewis.de> <1xu2ue5c.fsf@python.net> Message-ID: Thomas Heller writes: >> While we are at it: At what point will we switch to VC7.1 for the >> Windows builds? > > Hm, for 2.4, maybe. That sounds reasonable. The currently released mingw is close to being able to build MSVC7.1 compatible DLLs, but not quite there yet (it needs a workaround). I'd expect it to be there by the time 2.4 is released. > Or should there 7.1 compiled .msi installers in parallel for the 2.3 line? That would be extremely useful - my view on mingw above is based on import table inspection only. It would be nice to have a way of testing genuine Python extensions built with mingw. Of course, a 7.1-compiled installer would need to allow for installation in parallel with the 6.0-compiled one, because I can't imagine many people wanting to use it as their production version (most extensions will only be distributed as 6.0-compiled versions, so wouldn't be usable with the 7.1-compiled Python). Paul. -- This signature intentionally left blank From skip at manatee.mojam.com Sun Sep 28 08:00:36 2003 From: skip at manatee.mojam.com (Skip Montanaro) Date: Sun Sep 28 08:00:50 2003 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200309281200.h8SC0aAS011740@manatee.mojam.com> Bug/Patch Summary ----------------- 507 open / 4175 total bugs (+25) 201 open / 2395 total patches (+10) New Bugs -------- exit() raises exception (2003-09-21) http://python.org/sf/810214 mkstemp doesn't return abspath (2003-09-21) http://python.org/sf/810408 nested variables in 'class:' statements (2003-09-22) http://python.org/sf/810714 Support for non-string data in ConfigParser unclear/buggy (2003-09-22) http://python.org/sf/810843 shutil.copyfile fails when dst exists read-only (2003-09-22) http://python.org/sf/810879 os.stat returning a time = -1 (2003-09-22) http://python.org/sf/810887 2.3.1 configure bug (2003-09-23) http://python.org/sf/811028 Py2.2.3: Problem with Expat/XML/Zope on MacOSX 10.2.8 (2003-09-23) http://python.org/sf/811070 test_tempfile fails on windows if space in install path (2003-09-23) http://python.org/sf/811082 HP/UX vs configure (2003-09-23) http://python.org/sf/811160 dis.disassemble_string() broken (2003-09-23) http://python.org/sf/811294 ntohs on Solaris can return negative values (2003-09-23) http://python.org/sf/811295 "string".split behaviour for empty strings (2003-09-24) http://python.org/sf/811604 readline+no threads = core dump (2003-09-24) http://python.org/sf/811844 int ("ffffffd3", 16) gives error (2003-09-24) http://python.org/sf/811898 webbrowser.open_new() opens in an existing browser window (2003-09-24) http://python.org/sf/812089 randint is always even (2003-09-24) http://python.org/sf/812202 tarfile violates bufsize (2003-09-25) http://python.org/sf/812325 BasHTTPServer IE Mac 5.1 size problem (2003-09-25) http://python.org/sf/812340 a Py_DECREF() too much (2003-09-25) http://python.org/sf/812353 Problem with ftplib on HP-UX11i (2003-09-25) http://python.org/sf/812376 OSA support for properties broken (2003-09-25) http://python.org/sf/812750 2.3.1 re_test causes sigsegv (2003-09-25) http://python.org/sf/812804 turtle.py deferres exec of stmnts with tracer(0) (2003-09-26) http://python.org/sf/812986 Migrate PEP material to docs (2003-09-26) http://python.org/sf/813198 -Qnew switch doesn't work (2003-09-26) http://python.org/sf/813342 locale.getdefaultlocale doesnt handle all locales gracefully (2003-09-27) http://python.org/sf/813449 restrictions in _tkinter built with threaded tk undocumented (2003-09-27) http://python.org/sf/813453 New Patches ----------- Fix many doc typos (2003-09-22) http://python.org/sf/810751 socket.ssl should check certificates (2003-09-22) http://python.org/sf/810754 fix for mkstemp with relative paths (bug #810408) (2003-09-22) http://python.org/sf/810914 More precise realm parsing in AbstractBasicAuthHandler (2003-09-25) http://python.org/sf/812285 module shutdown procedure based on GC (2003-09-25) http://python.org/sf/812369 normalize whitespace (2003-09-25) http://python.org/sf/812378 Fix test_tempfile: space in Win32 install path bug #811082 (2003-09-26) http://python.org/sf/813200 _sre stack overflow on FreeBSD/amd64 and /sparc64 (2003-09-26) http://python.org/sf/813391 Scalable zipfile extension (2003-09-27) http://python.org/sf/813436 Add IPPROTO_IPV6 option to the socketmodule (2003-09-27) http://python.org/sf/813445 entry size for cursors (2003-09-27) http://python.org/sf/813877 Closed Bugs ----------- In 2.2, types are inheritable, &c (2001-08-21) http://python.org/sf/453684 Minor Python Intro update (2003-09-13) http://python.org/sf/805788 WinCVs & Python 2.3 (2003-09-16) http://python.org/sf/807253 Max and min are incorrect with one argument (2003-09-17) http://python.org/sf/807771 leak on lambda with duplicate arguments error (2003-09-18) http://python.org/sf/808594 Closed Patches -------------- Adding rsplit() to string and unicode objects. (2003-09-06) http://python.org/sf/801847 From Scott.Daniels at Acm.Org Sun Sep 28 19:00:53 2003 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Sun Sep 28 11:24:53 2003 Subject: [Python-Dev] Re:coping with a missing fsync In-Reply-To: References: Message-ID: <3F776825.6060302@Acm.Org> After seeing: >backup kept screwing up until FileStorage's > > if fsync is not None: fsync(file.fileno()) > >actually did something on Windows. > I suspect making the change where you obtain fsync, rather than where you call it makes the code clearer. Just after import os, you can do the appropriate bit of the following code: try: fsync = os.fsync except AttributeError: fsync = lambda filenumber: None # Unfortunately nothing to call -------- or -------- fsync = os.fsync or lambda filenumber: None # if we cannot sync, do nothing This (at least to me) emphasizes the lack of a system facility. The existing code emphasizes local solutions on a case-by-case basis. If the solution is always to skip the call, I'd find this method clearer. -Scott Daid Daniels Scott.Daniels@Acm.Org From guido at python.org Sun Sep 28 12:12:29 2003 From: guido at python.org (Guido van Rossum) Date: Sun Sep 28 12:13:17 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: Your message of "Sat, 27 Sep 2003 22:08:31 EDT." References: Message-ID: <200309281612.h8SGCTS00510@12-236-54-216.client.attbi.com> > [Martin v. Löwis] > > I mostly agree: ZODB is indeed advanced, and it is indeed a good idea > > to check for presence of os.fsync before using it. > > > > While this is OT, I'd still like to question the usefulness of > > fsync(2) in the first place, for applications like ZODB. I assume > > fsync is used as a write barrier, to make sure old modifications are > > on disk, before making new changes. [Tim] > That's important, but not primarily for the catastrophic error-recovery > scenarios you go on to sketch. Tim, trust me. The scenario you go on to sketch is unique to Windows. fsync() was invented on Unix. On that platform, there's *never* a need to do fsync() to allow another process to see the right data. fsync()'s effect is just not visible at the abstraction level presented by the kernel. fsync() is needed because sometimes (e.g. through a power or kernel failure) the kernel's abstraction is broken. Everybody except you who is arguing for fsync()'s importance does so because they have experienced (or fear to experience) crashes. It's a bug in the Windows implementation of Unix-style I/O that a commit() call (which is what fsync() is aliased to in posixmodule.c on Windows) is needed in order to ensure that other processes see what a process wrote. Given that fsync() on Windows doesn't have the configuration problem that started all this, I'm not sure how *your* argument for fsync() can help convince Martin that we should do a 2.3.2 release ASAP. --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Sun Sep 28 15:18:08 2003 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Sep 28 15:18:28 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <20030927213332.GI17491@zot.electricrain.com> References: <200309270516.h8R5GAT7007511@localhost.localdomain> <16245.53947.33728.86658@montanaro.dyndns.org> <3F75F584.8040207@v.loewis.de> <20030927205147.GH17491@zot.electricrain.com> <3F75FE7A.6070800@v.loewis.de> <20030927213332.GI17491@zot.electricrain.com> Message-ID: <3F7733F0.6060803@v.loewis.de> Gregory P. Smith wrote: >>I see. I could accept that it is confusing; would it also be backwards >>compatible (i.e. would BerkeleyDB create those files on demand, and >>would old Python installation still be able to read the database even >>if those files where around)? > > > It would not. Its already not backwards compatible. Thats what the > bsddb185 module is for. I should be more specific: If CDB is activated by default in 2.3.1, would that compatible with files created in 2.3.0 (which are not 1.85 files, but some 4.x files). >>But is it also easy to detect that multiple applications try to use >>the same database? > > > No. But it is easy to document. The old bsddb module never allowed > it either. I doubt many users are aware of that restriction (I, myself, wasn't). Regards, Martin From gward at python.net Sun Sep 28 15:18:28 2003 From: gward at python.net (Greg Ward) Date: Sun Sep 28 15:18:42 2003 Subject: [Starship] Re: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <8yoauf87.fsf@python.net> References: <200309261752.h8QHqJgD029823@localhost.localdomain> <20030926184853.GB22837@mems-exchange.org> <2mzngr1bf2.fsf@starship.python.net> <8yoauf87.fsf@python.net> Message-ID: <20030928191828.GA2852@cthulhu.gerg.ca> On 27 September 2003, Thomas Heller said: > I could do it myself if the tools needed to build the docs would be > available on the starship. Last time I checked, neither Tex nor > latex2html were installed. > > So, I hereby kindly request: Could the tools to build the Python docs > from source please be installed on the starship? OK, I'm installing tetex-base and latex2html now. No guarantees if the version of latex2html in Debian testing will actually work, of course. ;-) (No, wait, it should work: I have the same version (2000-beta1-8) on my PC, and the last time I did Python doc work, I *think* I was able to produce HTML output.) Greg -- Greg Ward http://www.gerg.ca/ A closed mouth gathers no foot. From martin at v.loewis.de Sun Sep 28 15:34:00 2003 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Sep 28 15:34:18 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <16246.2621.472392.671642@montanaro.dyndns.org> References: <200309270516.h8R5GAT7007511@localhost.localdomain> <16245.53947.33728.86658@montanaro.dyndns.org> <3F75F584.8040207@v.loewis.de> <16246.2621.472392.671642@montanaro.dyndns.org> Message-ID: <3F7737A8.8010400@v.loewis.de> Skip Montanaro wrote: > I've given that a little more thought. Instead of encumbering the factory > functions which implement the old API with an optional dbenv argument, I > think if people want to provide their own environment it's reasonable to > expect them to use the new bsddb3 API. All we should do with the old API is > make it work in a multi-threaded environment if possible. But wouldn't that precisely involve creating environments? If you merely propose that the environment is not a parameter - +1. Regards, Martin From martin at v.loewis.de Sun Sep 28 15:38:14 2003 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Sep 28 15:38:39 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: References: Message-ID: <3F7738A6.5020004@v.loewis.de> Tim Peters wrote: > At least on Windows, both the flush and the fsync are necessary to see one > million bytes (via a different process) at the first prompt, two million at > the second, and so on. With neither, another process typically sees 0 bytes > before the file gets huge. With just one of them, it seems hard to predict, > ranging from 0 to "almost" a million additional bytes per prompt. As Guido explains, fsync is not necessary for that kind of application on a POSIX system. Once write(2) has completed, all other processes immediately see the changed data. Regards, Martin From martin at v.loewis.de Sun Sep 28 15:52:44 2003 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Sep 28 15:52:48 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <1064726667.31604.105.camel@anthem> References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> <20030926103846.A26782@discworld.dyndns.org> <1064683719.19498.68.camel@localhost.localdomain> <3F75CD80.3080008@v.loewis.de> <1064726667.31604.105.camel@anthem> Message-ID: <3F773C0C.7060102@v.loewis.de> Barry Warsaw wrote: > Before I enable it unconditionally, I'd want to be more certain that > it's necessary, or useful for the majority of sites, and that the > performance hit is either worth it, not as bad as my benchmarks showed, > or at least not as observably bad under real-world conditions. You should first understand why message loss occurs if there is no fsync. For example, if the MTA stores incoming messages on disk, then invokes mailman, message loss may occur if the MTA deletes the message from the spool before mailman has fsync'ed its copy, and the power goes out at this moment. If so, users might be better of using a powerful filesystem. I *think* that, e.g. on Linux, ext3fs with ordered data writes might be sufficient, as deletion of the file should only occur after the data have been written (OTOH, it might be that ext3 in journalled mode is needed, if data and metadata changes are not mutually subject to the ordering). This should be less expensive than fsync, as the journalled file system will still perform lazy writes - just in the right order. Also notice that some file system drivers are incapable of implementing fsync correctly, so they fall back to treat fsync(2) like sync(2), which is quite expensive. Regards, Martin From Jack.Jansen at cwi.nl Sun Sep 28 16:52:42 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Sun Sep 28 16:52:48 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <2mzngr1bf2.fsf@starship.python.net> References: <200309261752.h8QHqJgD029823@localhost.localdomain> <20030926184853.GB22837@mems-exchange.org> <2mzngr1bf2.fsf@starship.python.net> Message-ID: On 26-sep-03, at 23:34, Michael Hudson wrote: > Finding a substitue Jack might be harder (though given Bob Ippolito's > work rate at the moment we could probably make him MacPython > maintainer without him noticing ). Not that I'm planning to get run over by a tram shortly, but just in case: I think Just or Ronald or yourself or any various other python-devvers would make a better substitute *at this point in time*. Bob does some truly wonderful things at terrific speed, but that has some drawbacks for maintainability, IMHO. Being MacPython maintainer is 10% doing fun stuff and 90% taking care of all the nitty-gritty details, making everything work in as many situations as possible, documenting, etc etc etc. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From tim.one at comcast.net Sun Sep 28 17:50:30 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Sep 28 17:50:34 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <3F7738A6.5020004@v.loewis.de> Message-ID: [Martin v. L?wis] > As Guido explains, fsync is not necessary for that kind of application > on a POSIX system. Once write(2) has completed, all other processes > immediately see the changed data. But we're not calling write(2) -- Python's file.write() (which my little driver used, and also what ZODB uses) calls the buffered fwrite. Ignoring that, some systems aren't "pure". I note that, on my Windows box, some Cygwin utilties "see" the hoped-for file sizes while running my driver without the fsync under Cygwin Python, but Windows utilities don't. I was using Cygwin 2.2.2; I don't know whether 2.3.1 under Cygwin thinks fsync has gone missing. From jeremy at zope.com Sun Sep 28 17:58:30 2003 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Sep 28 17:59:29 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <3F75CD80.3080008@v.loewis.de> References: <200309260524.h8Q5Ok1t013688@localhost.localdomain> <1064591608.30783.30.camel@anthem> <20030926103846.A26782@discworld.dyndns.org> <1064683719.19498.68.camel@localhost.localdomain> <3F75CD80.3080008@v.loewis.de> Message-ID: <1064786309.19498.93.camel@localhost.localdomain> On Sat, 2003-09-27 at 13:48, "Martin v. L?wis" wrote: > 3. There is an operating system crash (kernel panic or similar). > fsync does not help, as, for a buggy kernel, anything might have > happened to the data before. This is a vacuously true statement :-). Anything might have happened; among other things, the kernel buffers could have been flushed to disk before the kernel panic. I imagine that there are many bugs that could cause a kernel panic or deadlock that would not also corrupt data on its way to disk. That is, we can't avoid bugs that are going to cause problems writing to disk, but we can do our best to minimize the effects of other bugs. > 5. There is a power outage. This is the case where fsync should help: > everything up to the write barrier is on disk. Of course, if the > disk drive itself has write caching, fsync may have completed > without the data being on the disk (this would be an fsync bug, but > I believe Linux suffers from this particular bug). (There are OSes other than Linux.) > So in short, fsync(2) helps only in case of a power outage; for normal > operation, it is not needed. In the case of a power outage, it is > doubtful whether it has the desired effect. I see two cases where it improves the changes that data is on persistent storage before the database reports to the client that the write succeeded. > Slightly more on-topic: os.fsync is even worse, as it cannot be > used to implement a write barrier in case of multiple threads. It > runs with the GIL released, so while fsync is running, other > threads might change the file. The semantics of fsync in this case > is pretty unclear, but it is likely not a write barrier. Sure. ZODB only allows a single thread to write at a time. Jeremy From tim.one at comcast.net Sun Sep 28 18:11:41 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Sep 28 18:12:41 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <200309281612.h8SGCTS00510@12-236-54-216.client.attbi.com> Message-ID: [Guido] > Tim, trust me. The scenario you go on to sketch is unique to Windows. I would rather trust the test program, but I don't see much point to the argument (if that's what this is): if os.fsync has gone missing on platforms that "used to" support it, of course that's a serious regression. > fsync() was invented on Unix. On that platform, there's *never* a > need to do fsync() to allow another process to see the right data. > fsync()'s effect is just not visible at the abstraction level > presented by the kernel. fsync() is needed because sometimes > (e.g. through a power or kernel failure) the kernel's abstraction is > broken. Everybody except you who is arguing for fsync()'s importance > does so because they have experienced (or fear to experience) crashes. > > It's a bug in the Windows implementation of Unix-style I/O that a > commit() call (which is what fsync() is aliased to in posixmodule.c on > Windows) is needed in order to ensure that other processes see what a > process wrote. The test driver (and ZODB) use buffered I/O, not unbuffered POSIX I/O. Assuring that writes done to buffered streams are visible to other processes is trickier than Martin's reliance on the write(2) man page, since write(2) isn't involved in the chain. > Given that fsync() on Windows doesn't have the configuration problem > that started all this, I don't know whether Cygwin Python suffers the same problem. If it does, then people running ZODB under Cygwin 2.3.1 (which I don't believe has been released yet) are in for nasty surprises if they run repozo.py under the "native" Windows Python. I've tried this under Cygwin 2.2.2 (see other email). Since that did have os.fsync(), but the native Windows Python didn't at that time, my best guess is that Cygwin Python is affected. > I'm not sure how *your* argument for fsync() can help convince Martin > that we should do a 2.3.2 release ASAP. It's enough for me that a function that used to be there has gone away without good reason. It's easier to cut a new release than argue about how much trouble the missing os.fsync should cause, as people have already poured more energy into complaining about it than would be needed to cut a new release. From tim.one at comcast.net Sun Sep 28 18:32:51 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Sep 28 18:32:55 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <20030926233959.GC17491@zot.electricrain.com> Message-ID: [Skip] >> For long-running SpamBayes apps like pop3proxy and the >> Outlook plugin, pickles make sense if shared access to the scoring >> database is not needed. [Gregory P. Smith] > Combine that with data event/change logging and you have effectively > the same thing as what many apps think they need to use a database > library for. Replay the entire log of data changes internally on > infrequent app startup to generate your internal state. when the log > file is getting to big, write a full state snapshot and continue > logging changes from it. no need to save the entire pickle everytime > something changes. That may be OK for pop3proxy users. Alas, startup time and memory burden are keen topics for Outlook plugin users, and, e.g., the startup time to unpickle a large dict (let alone replay logs since the last snapshot) is unbearable for people with large training sets. When you start Outlook, it typically starts fetching email right away, and you typically want to see your email right away then. The relatively zippy startup time of a Berkeley DB is highly welcome for that reason. From guido at python.org Sun Sep 28 19:43:43 2003 From: guido at python.org (Guido van Rossum) Date: Sun Sep 28 19:44:36 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: Your message of "Sun, 28 Sep 2003 18:11:41 EDT." References: Message-ID: <200309282343.h8SNhht00773@12-236-54-216.client.attbi.com> > The test driver (and ZODB) use buffered I/O, not unbuffered POSIX I/O. > Assuring that writes done to buffered streams are visible to other processes > is trickier than Martin's reliance on the write(2) man page, since write(2) > isn't involved in the chain. All you need to do is call f.flush(). This calls write(2) internally (even on Windows). --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Sun Sep 28 20:33:05 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Sep 28 20:33:10 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <200309282343.h8SNhht00773@12-236-54-216.client.attbi.com> Message-ID: [Tim] >> The test driver (and ZODB) use buffered I/O, not unbuffered POSIX >> I/O. Assuring that writes done to buffered streams are visible to >> other processes is trickier than Martin's reliance on the write(2) >> man page, since write(2) isn't involved in the chain. [Guido] > All you need to do is call f.flush(). This calls write(2) internally > (even on Windows). Sorry, I don't understand the point of this Unixish chest-puffing <0.8 wink>. POSIX doesn't mandate that fflush() call write(), although it requires, as an extension to the C standard, pretty much the same end result. Nevertheless, calling f.flush() isn't enough on Windows, regardless of whether the native Windows Python or Cygwin Python is doing it. Without the os.fsync(f.fileno()) too, native Windows utilities and the Windows Python see a wrong current file size. This gets pretty bizarre too: after letting Cygwin Python write and flush a million bytes (i.e., at the first prompt in the test driver), the Cygwin Python (in a different process) os.path.getsize() returns a million, but the Windows Python returns 0. Put in the fsync too and everyone is happy. This can't work if fsynch goes missing in the Cygwin Python. From tim.one at comcast.net Sun Sep 28 21:38:44 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Sep 28 21:38:48 2003 Subject: [Python-Dev] 2.3.1 is (almost) a go In-Reply-To: <16245.53947.33728.86658@montanaro.dyndns.org> Message-ID: [Skip Montanaro] > ... > Attached is a modified version of the hammer.py script which seems to > not fail for me on either Windows run from IDLE (Python 2.3, BDB > 4.1.6) or Mac OS X (Python CVS, BDB 4.2.1). The original script > failed for me on Windows but not Mac OS X. Can some other people for > whom the original script fails please try it? (I also attached it to > bug #775414.) On Win98SE with current Python 2.3.1, it doesn't fail, but it never seemed to finish for me either. Staring at WinTop showed that the Python process stopped accumulating cycles. Can't be killed with Ctrl+C (no visible effect). Can be killed with Ctrl+Break. Dumping print "%s %s" % (thread.get_ident(), i) at the top of the hammer loop showed that the threads get through several hundred iterations, then all printing stops. Attaching to a debug-build Python from the debugger when a freeze occurs isn't terribly illuminating. One thread's stack shows _BSDDB_D! __db_win32_mutex_lock + 134 bytes _BSDDB_D! __lock_get + 2264 bytes _BSDDB_D! __lock_get + 197 bytes _BSDDB_D! __ham_get_meta + 120 bytes _BSDDB_D! __ham_c_dup + 4201 bytes _BSDDB_D! __db_c_put + 2544 bytes _BSDDB_D! __db_put + 507 bytes _DB_put(DBObject * 0x016cff88, __db_txn * 0x016d0000, __db_dbt * 0x016cc000, __db_dbt * 0x50d751fe, int 0) line 562 + 35 bytes The main thread's stack shows _BSDDB_D! __db_win32_mutex_lock + 134 bytes _BSDDB_D! __lock_get + 2264 bytes _BSDDB_D! __lock_get + 197 bytes _BSDDB_D! __db_lget + 365 bytes _BSDDB_D! __ham_lock_bucket + 105 bytes _BSDDB_D! __ham_get_cpage + 195 bytes _BSDDB_D! __ham_item_next + 25 bytes _BSDDB_D! __ham_call_hash + 2479 bytes _BSDDB_D! __ham_c_dup + 4307 bytes _BSDDB_D! __db_c_put + 2544 bytes _BSDDB_D! __db_put + 507 bytes _DB_put(DBObject * 0x008fe2e8, __db_txn * 0x00000000, __db_dbt * 0x0062f230, __db_dbt * 0x0062f248, int 0) line 562 + 35 bytes DB_ass_sub(DBObject * 0x008fe2e8, _object * 0x00b83178, _object * 0x00b83370) line 2330 + 23 bytes PyObject_SetItem(_object * 0x008fe2e8, _object * 0x00b83178, _object * 0x00b83370) line 123 + 18 bytes eval_frame(_frame * 0x00984948) line 1448 + 17 bytes ... The other threads are somewhere in the OS kernel and don't have useful tracebacks. This varies from run to run, but all threads with a useful stack are always stuck at the same place in __db_win32_mutex_lock. All in all, looks like it's simply deadlocked. Running the original hammer.py under current CVS Python freezes in the same way now. I added this info to the bug report: http://www.python.org/sf/775414 From bac at OCF.Berkeley.EDU Sun Sep 28 22:40:38 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sun Sep 28 22:40:36 2003 Subject: [Python-Dev] Good way of finding out what C functions we have? Message-ID: <3F779BA6.6050407@ocf.berkeley.edu> I just did a bunch of research into trying to find out if using strdup() in a patch was reasonable. I finally grepped enough to find strdup in pyconfig.h and configure.in . But that leads to a Brett Newbie Question. Does there happen to be a better way to find out what functions we check for beyond grepping configure.in? The entire reason I bothered to do so much research is I thought we were only supposed to assume ANSI C with POSIX (which, from what I can tell, strdup() is not). But with more Googling and grepping I discovered the meaning of AC_REPLACE_FUNCS() which strdup was being called in configure.in . I would never have known strdup was available to me for code had I not randomly come across this patch and done all of this research. Because of this I would like to list in the Python development guide I am writing up all of the places one can check for info; what configure.in checks for or handles and pyport.h come to mind. It would be nice to have a place that lists all the places one can check for C functionality that are not in the Python/C API directly. If people have other places in the code that one should check please let me know so I can make sure it makes it into the doc and prevents me from having to come up with another Brett Newbie Question. =) -Brett From martin at v.loewis.de Mon Sep 29 02:03:39 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Mon Sep 29 02:04:00 2003 Subject: [Python-Dev] Good way of finding out what C functions we have? In-Reply-To: <3F779BA6.6050407@ocf.berkeley.edu> References: <3F779BA6.6050407@ocf.berkeley.edu> Message-ID: "Brett C." writes: > Does there happen to be a better way to find out what functions we > check for beyond grepping configure.in? The entire reason I bothered > to do so much research is I thought we were only supposed to assume > ANSI C with POSIX (which, from what I can tell, strdup() is not). Actually, strdup is in POSIX: http://www.opengroup.org/onlinepubs/007904975/functions/strdup.html If we still check for it, it is because it might not have been available on some system, at some point in time. > Because of this I would like to list in the Python development guide I > am writing up all of the places one can check for info; what > configure.in checks for or handles and pyport.h come to mind. It > would be nice to have a place that lists all the places one can check > for C functionality that are not in the Python/C API directly. If > people have other places in the code that one should check please let > me know so I can make sure it makes it into the doc and prevents me > from having to come up with another Brett Newbie Question. =) I would look in pyconfig.h.in as the first thing. There is a pitfall, though: presence of some HAVE_foo in pyconfig.h does not mean that Python gracefully deals with the absence of foo. When converting configure to autoconf 2.5x, I checked all macros to determine whether they were still in use, but some may have become unused/not processed correctly since. That would be a bug, of course. Regards, Martin From sjoerd at acm.org Mon Sep 29 05:12:33 2003 From: sjoerd at acm.org (Sjoerd Mullender) Date: Mon Sep 29 05:12:43 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <20030926184853.GB22837@mems-exchange.org> References: <200309261752.h8QHqJgD029823@localhost.localdomain> <20030926184853.GB22837@mems-exchange.org> Message-ID: <3F77F781.1020803@acm.org> Neil Schemenauer wrote: > * try to reduce the number of version numbers in different files need to > be adjusted. Also, try to automate the adjustment. Check out the package vertooo (on SourceForge) for this. It does just this (automating version numbers in lots of files). It's even written in Python. :-) From anthony at interlink.com.au Mon Sep 29 04:05:34 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Sep 29 06:08:20 2003 Subject: [Python-Dev] Python2.3.2 and release23-maint branch Message-ID: <200309290805.h8T85ZxC025742@localhost.localdomain> I've not heard any complaints, so please consider the release23-maint branch frozen. Anthony From mwh at python.net Mon Sep 29 07:00:46 2003 From: mwh at python.net (Michael Hudson) Date: Mon Sep 29 07:00:05 2003 Subject: [Starship] Re: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <20030928191828.GA2852@cthulhu.gerg.ca> (Greg Ward's message of "Sun, 28 Sep 2003 15:18:28 -0400") References: <200309261752.h8QHqJgD029823@localhost.localdomain> <20030926184853.GB22837@mems-exchange.org> <2mzngr1bf2.fsf@starship.python.net> <8yoauf87.fsf@python.net> <20030928191828.GA2852@cthulhu.gerg.ca> Message-ID: <2mvfrb271d.fsf@starship.python.net> Greg Ward writes: > On 27 September 2003, Thomas Heller said: >> I could do it myself if the tools needed to build the docs would be >> available on the starship. Last time I checked, neither Tex nor >> latex2html were installed. >> >> So, I hereby kindly request: Could the tools to build the Python docs >> from source please be installed on the starship? > > OK, I'm installing tetex-base and latex2html now. No guarantees if the > version of latex2html in Debian testing will actually work, of course. > ;-) Somehow this seems to have avoided installing some important executables -- like, e.g., "latex". Are there some other packages that need to be installed too? Cheers, mwh -- 58. Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html From Jack.Jansen at cwi.nl Mon Sep 29 08:35:33 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Mon Sep 29 08:35:15 2003 Subject: [Python-Dev] Python2.3.2 and release23-maint branch In-Reply-To: <200309290805.h8T85ZxC025742@localhost.localdomain> Message-ID: <6B24E828-F279-11D7-BFE6-0030655234CE@cwi.nl> On Monday, September 29, 2003, at 10:05 AM, Anthony Baxter wrote: > > I've not heard any complaints, so please consider the release23-maint > branch > frozen. Uhm... I was busy doing all the version numbers and testing for MacPython, I thought I had most of today (Monday, european) for that??!? I won't check anything in unless you explicitly give me permission, -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From andrew-pythondev at puzzling.org Mon Sep 29 09:03:40 2003 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Mon Sep 29 09:03:45 2003 Subject: [Starship] Re: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <2mvfrb271d.fsf@starship.python.net> References: <200309261752.h8QHqJgD029823@localhost.localdomain> <20030926184853.GB22837@mems-exchange.org> <2mzngr1bf2.fsf@starship.python.net> <8yoauf87.fsf@python.net> <20030928191828.GA2852@cthulhu.gerg.ca> <2mvfrb271d.fsf@starship.python.net> Message-ID: <20030929130340.GF21156@frobozz> Michael Hudson wrote: > Greg Ward writes: > > > OK, I'm installing tetex-base and latex2html now. No guarantees if the > > version of latex2html in Debian testing will actually work, of course. > > ;-) > > Somehow this seems to have avoided installing some important > executables -- like, e.g., "latex". Are there some other packages > that need to be installed too? Looks like you need tetex-bin, judging from my debian (unstable) system: bash-2.05b$ dpkg -S `which latex` tetex-bin: /usr/bin/latex It's probably worth grabbing tetex-extra, too; its description says: ... Together with tetex-bin and tetex-base, this will give you a complete teTeX installation. -Andrew. From anthony at interlink.com.au Mon Sep 29 09:08:22 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Sep 29 09:10:35 2003 Subject: [Python-Dev] Python2.3.2 and release23-maint branch In-Reply-To: <6B24E828-F279-11D7-BFE6-0030655234CE@cwi.nl> Message-ID: <200309291308.h8TD8N8O030399@localhost.localdomain> >>> Jack Jansen wrote > Uhm... I was busy doing all the version numbers and testing for > MacPython, I thought > I had most of today (Monday, european) for that??!? > > I won't check anything in unless you explicitly give me permission, My bad - I was going from the release PEPs. All checkins, _other_ than those required for the release. ;) I was looking through the .plist files, and it seems like there's about 8 or so different places featuring the string "2.3". Is there a way to make this number less? (Obviously, not for 2.3.2)... Anthony -- Anthony Baxter It's never too late to have a happy childhood. From Jack.Jansen at cwi.nl Mon Sep 29 10:44:39 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Mon Sep 29 10:44:16 2003 Subject: [Python-Dev] Python2.3.2 and release23-maint branch In-Reply-To: <200309291308.h8TD8N8O030399@localhost.localdomain> Message-ID: <746AF574-F28B-11D7-BFE6-0030655234CE@cwi.nl> On Monday, September 29, 2003, at 03:08 PM, Anthony Baxter wrote: > >>>> Jack Jansen wrote >> Uhm... I was busy doing all the version numbers and testing for >> MacPython, I thought >> I had most of today (Monday, european) for that??!? >> >> I won't check anything in unless you explicitly give me permission, > > My bad - I was going from the release PEPs. All checkins, _other_ than > those required for the release. ;) Sigh... Don't wait for me to update the Mac-specific files: I've apparently messed up the developer tools on my system. It's now a mix between gcc3.3 and gcc3, and I keep running into problems. Reverting to gcc3.1 also doesn't seem to work. My home system has a virgin 10.2 partition, but I'm away tonight so I won't have the chance to build and test anything (and I assume tomorrow-morning-European is too late, right?). I'm 99% sure that just changing the version numbers in the .plist and .strings files won't break anything, but without testing I don't want to chance it. > I was looking through the .plist files, and it seems like there's about > 8 or so different places featuring the string "2.3". Is there a way to > make this number less? (Obviously, not for 2.3.2)... Eventually these files need to be generated (we have a plistlib module for that), and the same goes for the accompanying .strings files. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From anthony at interlink.com.au Mon Sep 29 10:45:05 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Sep 29 10:46:55 2003 Subject: [Python-Dev] 2.3.2 and bsddb Message-ID: <200309291445.h8TEj5hc002980@localhost.localdomain> For those of you not following every bug in the SF tracker closely, in http://www.python.org/sf/775414 it's been suggested that the docs for 2.3.2 include a warning about using the old-style interface to bsddb (without a DBEnv) with multi-threaded applications. This seems like a prudent suggestion - does someone want to supply some words? Anthony From anthony at interlink.com.au Mon Sep 29 10:47:54 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Sep 29 10:49:43 2003 Subject: [Python-Dev] Python2.3.2 and release23-maint branch In-Reply-To: <746AF574-F28B-11D7-BFE6-0030655234CE@cwi.nl> Message-ID: <200309291447.h8TElsJG003047@localhost.localdomain> >>> Jack Jansen wrote > Sigh... Don't wait for me to update the Mac-specific files: I've > apparently > messed up the developer tools on my system. It's now a mix between > gcc3.3 > and gcc3, and I keep running into problems. Reverting to gcc3.1 also > doesn't seem to work. My home system has a virgin 10.2 partition, but > I'm > away tonight so I won't have the chance to build and test anything > (and I assume tomorrow-morning-European is too late, right?). I'm 99% > sure that just changing the version numbers in the .plist and .strings > files > won't break anything, but without testing I don't want to chance it. I'm cutting a release candidate tomorrow morning, but the release itself on Friday AU time. If it's just a matter of updating the version numbers in the .plist files, I can do this. Assuming, of course, that just editing the .plist files with vi to s/2.3/2.3.2/ is going to Do The Right Thing. I know of a couple of folks with Macs who could test the framework build - aside from "./configure ; make ; make testall", what else do they need to test? Anthony -- Anthony Baxter It's never too late to have a happy childhood. From fdrake at acm.org Mon Sep 29 10:31:08 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon Sep 29 11:03:50 2003 Subject: [Python-Dev] PC/python_nt.rc In-Reply-To: References: Message-ID: <16248.16940.996691.656779@grendel.zope.com> Thomas Heller writes: > Does this sound like a sensible approach, or are there any objections? Not from me! If Tim doesn't have any objections, then you should proceed. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From Jack.Jansen at cwi.nl Mon Sep 29 11:13:31 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Mon Sep 29 11:13:08 2003 Subject: [Python-Dev] Python2.3.2 and release23-maint branch In-Reply-To: <200309291447.h8TElsJG003047@localhost.localdomain> Message-ID: <7C78AF06-F28F-11D7-BFE6-0030655234CE@cwi.nl> On Monday, September 29, 2003, at 04:47 PM, Anthony Baxter wrote: > I'm cutting a release candidate tomorrow morning, but the release > itself > on Friday AU time. If it's just a matter of updating the version > numbers in > the .plist files, I can do this. Assuming, of course, that just editing > the .plist files with vi to s/2.3/2.3.2/ is going to Do The Right > Thing. Nevermind, I just did the checkins. I got my build system in enough of a shape that I've built everything and I'm confident it'll work. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From theller at python.net Mon Sep 29 11:37:47 2003 From: theller at python.net (Thomas Heller) Date: Mon Sep 29 11:37:59 2003 Subject: [Python-Dev] Python2.3.2 and release23-maint branch In-Reply-To: <200309290805.h8T85ZxC025742@localhost.localdomain> (Anthony Baxter's message of "Mon, 29 Sep 2003 18:05:34 +1000") References: <200309290805.h8T85ZxC025742@localhost.localdomain> Message-ID: <3cef7ghg.fsf@python.net> Anthony Baxter writes: > I've not heard any complaints, so please consider the release23-maint branch > frozen. When will you tag the branch? And I'm still unable to create the html docs myself ;-( Thomas From skip at pobox.com Mon Sep 29 11:37:45 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Sep 29 11:38:06 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: References: <3F7738A6.5020004@v.loewis.de> Message-ID: <16248.20937.685975.429148@montanaro.dyndns.org> Tim> [Martin v. L?wis] >> As Guido explains, fsync is not necessary for that kind of >> application on a POSIX system. Once write(2) has completed, all other >> processes immediately see the changed data. Tim> But we're not calling write(2) -- Python's file.write() (which my Tim> little driver used, and also what ZODB uses) calls the buffered Tim> fwrite. Then shouldn't a file.flush() call be sufficient to force a call to write(2) on POSIX systems? Tim> Ignoring that, some systems aren't "pure". I note that, on my Tim> Windows box, some Cygwin utilties "see" the hoped-for file sizes Tim> while running my driver without the fsync under Cygwin Python, but Tim> Windows utilities don't. I was using Cygwin 2.2.2; I don't know Tim> whether 2.3.1 under Cygwin thinks fsync has gone missing. It's been established that os.fsync() is required for Windows in at least some circumstances, so you have to assume it's required under all circumstances, correct? Skip From tim.one at comcast.net Mon Sep 29 11:58:43 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Sep 29 11:58:54 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <16248.20937.685975.429148@montanaro.dyndns.org> Message-ID: [Skip Montanaro] > Then shouldn't a file.flush() call be sufficient to force a call to > write(2) on POSIX systems? The POSIX spec doesn't say that, but it does say enough that it shouldn't matter whether or not an fflush implementation happens to call write under the covers -- POSIX fflush is required to write data "to the file" regardless of how it's implemented. A difficulty with Windows is that a file ain't an inode -- a file doesn't exist except in a directory, and the directory structure records the current length of the files in the directory. Writing data to a file doesn't necessarily update the file length stored in the file's directory entry, and flushing doesn't necessary update it either; _commit() does. Native Windows utilities get their idea of a file's length from reading the file's directory entry. > ... > It's been established that os.fsync() is required for Windows in at > least some circumstances, so you have to assume it's required under > all circumstances, correct? Sorry, I didn't follow the question. From tim.one at comcast.net Mon Sep 29 12:05:03 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Sep 29 12:05:13 2003 Subject: [Python-Dev] PC/python_nt.rc In-Reply-To: <16248.16940.996691.656779@grendel.zope.com> Message-ID: [Thomas Heller] >> Does this sound like a sensible approach, or are there any >> objections? [Fred L. Drake, Jr.] > Not from me! If Tim doesn't have any objections, then you should > proceed. Too much context got snipped here. If this was about eliminating the need for the field3.py dance in building Windows Python, ya, Thomas's idea sounded fine to me. Silence is assent <0.9 wink>. From skip at pobox.com Mon Sep 29 12:20:11 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Sep 29 12:20:21 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: References: <16248.20937.685975.429148@montanaro.dyndns.org> Message-ID: <16248.23483.916661.489555@montanaro.dyndns.org> >> ... It's been established that os.fsync() is required for Windows in >> at least some circumstances, so you have to assume it's required >> under all circumstances, correct? Tim> Sorry, I didn't follow the question. We've already established that os.fsync() is required in at least some cases on Windows. It seemed to me that you were trying to construct some other cases (possibly involving Cygwin) where that wasn't necessary. I probably just didn't follow your intent. Skip From tim.one at comcast.net Mon Sep 29 12:38:19 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Sep 29 12:38:28 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <16248.23483.916661.489555@montanaro.dyndns.org> Message-ID: [Skip Montanaro] > We've already established that os.fsync() is required in at least > some cases on Windows. It seemed to me that you were trying to > construct some other cases (possibly involving Cygwin) where that > wasn't necessary. Cygwin doesn't appear to need fsync() so long as you stick purely to Cygwin tools in all processes -- Cygwin appears POSIX-like in that respect. But Cygwin runs on Windows, and fsync() is still needed in Cygwin Python programs if native Windows utilities (including the native Windows Python) are to see correct current file sizes too. If os.fsync() is missing in Cygwin 2.3.1 (I don't know whether it is, but suspect that it is), then problems follow from that. A specific example is ZODB running under Cygwin, while the ZODB live-backup script repozo.py runs under native Windows Python. If the former can't do an fsync(), the latter can't know the correct current file size. From fdrake at acm.org Mon Sep 29 13:00:38 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon Sep 29 13:00:55 2003 Subject: [Python-Dev] Python 2.3.2c1 documentation Message-ID: <16248.25910.512604.157482@grendel.zope.com> The Python 2.3.2c1 documentation packages are now available on the FTP site: ftp://ftp.python.org/pub/python/doc/2.3.2c1/ -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From shane at zope.com Mon Sep 29 16:57:49 2003 From: shane at zope.com (Shane Hathaway) Date: Mon Sep 29 16:58:19 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <20030926184853.GB22837@mems-exchange.org> References: <20030926184853.GB22837@mems-exchange.org> Message-ID: <3F789CCD.1010707@zope.com> Neil Schemenauer wrote: > * write scripts to generate the final package and also verify its > sanity (e.g. check version numbers, dates, detect files that > should not be included in the release) FWIW, one thing I like to do is compare the list of files in the new release with the list of files in the previous release. It's amazing what can creep in. I've attached the script I use--it's ugly but it works. Shane -------------- next part -------------- #!/bin/sh # This utility compares the directory listing of two tar files. # It detects .gz or .bz2 extensions automatically and acts accordingly. if [ "$1" == "" ] || [ "$2" == "" ]; then echo usage: $0 tarfile1 tarfile2 exit 1 fi f1=`mktemp /tmp/difftar1.XXXXXX` f2=`mktemp /tmp/difftar2.XXXXXX` if echo $1 | grep '\.gz' > /dev/null; then flags='tfz' elif echo $1 | grep '\.bz2' > /dev/null; then flags='tfj' else flags='tf' fi tar $flags $1 | sort > $f1 if echo $2 | grep '\.gz' > /dev/null; then flags='tfz' elif echo $2 | grep '\.bz2' > /dev/null; then flags='tfj' else flags='tf' fi tar $flags $2 | sort > $f2 diff -u $f1 $f2 rm -f $f1 $f2 From jeremy at zope.com Mon Sep 29 17:07:39 2003 From: jeremy at zope.com (Jeremy Hylton) Date: Mon Sep 29 17:09:18 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <3F789CCD.1010707@zope.com> References: <20030926184853.GB22837@mems-exchange.org> <3F789CCD.1010707@zope.com> Message-ID: <1064869658.19498.183.camel@localhost.localdomain> On Mon, 2003-09-29 at 16:57, Shane Hathaway wrote: > Neil Schemenauer wrote: > > * write scripts to generate the final package and also verify its > > sanity (e.g. check version numbers, dates, detect files that > > should not be included in the release) > > FWIW, one thing I like to do is compare the list of files in the new > release with the list of files in the previous release. It's amazing > what can creep in. I've attached the script I use--it's ugly but it works. Sure is! This is a little OT, but distutils MANIFEST files are very useful for this purpose. Jeremy From gward at python.net Mon Sep 29 21:25:56 2003 From: gward at python.net (Greg Ward) Date: Mon Sep 29 21:26:00 2003 Subject: [Starship] Re: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <2mvfrb271d.fsf@starship.python.net> References: <200309261752.h8QHqJgD029823@localhost.localdomain> <20030926184853.GB22837@mems-exchange.org> <2mzngr1bf2.fsf@starship.python.net> <8yoauf87.fsf@python.net> <20030928191828.GA2852@cthulhu.gerg.ca> <2mvfrb271d.fsf@starship.python.net> Message-ID: <20030930012556.GA6451@cthulhu.gerg.ca> On 29 September 2003, Michael Hudson said: > Somehow this seems to have avoided installing some important > executables -- like, e.g., "latex". Are there some other packages > that need to be installed too? Argh. Installing tetex-bin (and -doc, -extra, -lib just for fun) now. Greg -- Greg Ward http://www.gerg.ca/ A man without religion is like a fish without a bicycle. From bac at OCF.Berkeley.EDU Mon Sep 29 22:07:58 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon Sep 29 22:08:00 2003 Subject: [Python-Dev] Good way of finding out what C functions we have? In-Reply-To: References: <3F779BA6.6050407@ocf.berkeley.edu> Message-ID: <3F78E57E.8090403@ocf.berkeley.edu> Martin v. L?wis wrote: > "Brett C." writes: > > >>Does there happen to be a better way to find out what functions we >>check for beyond grepping configure.in? The entire reason I bothered >>to do so much research is I thought we were only supposed to assume >>ANSI C with POSIX (which, from what I can tell, strdup() is not). > > > Actually, strdup is in POSIX: > > http://www.opengroup.org/onlinepubs/007904975/functions/strdup.html > Shows how much I know; I found that last night but since it didn't say POSIX anywhere I thought it was part of another standard. I guess the "Issue 5" mention means that BASE is POSIX? Or is IEEE 1003.1 basically POSIX? > If we still check for it, it is because it might not have been > available on some system, at some point in time. > We don't only check for it, but we have a replacement implementation (the comment for Python/strdup.c seems to suggest it is needed by the parser). > >>Because of this I would like to list in the Python development guide I >>am writing up all of the places one can check for info; what >>configure.in checks for or handles and pyport.h come to mind. It >>would be nice to have a place that lists all the places one can check >>for C functionality that are not in the Python/C API directly. If >>people have other places in the code that one should check please let >>me know so I can make sure it makes it into the doc and prevents me >>from having to come up with another Brett Newbie Question. =) > > > I would look in pyconfig.h.in as the first thing. There is a pitfall, > though: presence of some HAVE_foo in pyconfig.h does not mean that > Python gracefully deals with the absence of foo. When converting > configure to autoconf 2.5x, I checked all macros to determine whether > they were still in use, but some may have become unused/not processed > correctly since. That would be a bug, of course. > What does happen if a HAVE_foo is actually required? Does the build fail or will configure halt and say that the build will fail if you proceed? And if no one objects I will mention that looking in pyconfig.h.in is a good place to look to see what C functions that are non-POSIX and non-ANSI C we check for and pyport.h has helpful stuff as well. -Brett From anthony at interlink.com.au Tue Sep 30 03:01:30 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Sep 30 03:03:25 2003 Subject: [Python-Dev] Lib/idlelib/NEWS.txt Message-ID: <200309300701.h8U71VG7017578@localhost.localdomain> Idle still has it's own NEWS.txt file, and it's own version number (for instance, 2.3.1's version of idle is "1.0.1". Does this make sense? What should happen for 2.3.2, where no idle changes have been made? Anthony From anthony at python.org Tue Sep 30 06:37:16 2003 From: anthony at python.org (Anthony Baxter) Date: Tue Sep 30 06:39:13 2003 Subject: [Python-Dev] RELEASED Python 2.3.2, release candidate 1 Message-ID: <200309301037.h8UAbGhY029290@localhost.localdomain> On behalf of the Python development team and the Python community, I'm happy to announce the release of Python 2.3.2 (release candidate 1). Python 2.3.2 is a bug-fix release, to repair a couple of build problems and packaging errors in Python 2.3.1. Assuming no major problems crop up, a final release of Python 2.3.2 will follow later this week. For more information on Python 2.3.2, including download links for various platforms, release notes, and known issues, please see: http://www.python.org/2.3.2 Highlights of this new release include: - A bug in autoconf that broke building on HP/UX systems is fixed. - A bug in the Python configure script that meant os.fsync() was never available is fixed. Highlights of the previous major Python release (2.3) are available from the Python 2.3 page, at http://www.python.org/2.3/highlights.html Many apologies for the flaws in 2.3.1 - the final 2.3.2 release should address the problems. Enjoy the new release, Anthony Anthony Baxter anthony@python.org Python 2.3.2 Release Manager (on behalf of the entire python-dev team) From skip at pobox.com Tue Sep 30 09:12:09 2003 From: skip at pobox.com (Skip Montanaro) Date: Tue Sep 30 09:12:26 2003 Subject: [Python-Dev] Good way of finding out what C functions we have? In-Reply-To: <3F78E57E.8090403@ocf.berkeley.edu> References: <3F779BA6.6050407@ocf.berkeley.edu> <3F78E57E.8090403@ocf.berkeley.edu> Message-ID: <16249.33065.797263.217111@montanaro.dyndns.org> >> I would look in pyconfig.h.in as the first thing. There is a pitfall, >> though: presence of some HAVE_foo in pyconfig.h does not mean that >> Python gracefully deals with the absence of foo. When converting >> configure to autoconf 2.5x, I checked all macros to determine whether >> they were still in use, but some may have become unused/not processed >> correctly since. That would be a bug, of course. Brett> What does happen if a HAVE_foo is actually required? Does the Brett> build fail or will configure halt and say that the build will Brett> fail if you proceed? It's not exactly clear what you mean by "required". All the HAVE_foo macros are supposed to indicate the presence (when defined) or absence (when not defined) of a particular "feature" which is not felt by one or more of the authors to be universally available. As far as I know, a HAVE_foo macro should never be required to be defined, though I suppose if we had an infinite amount of time on our hands be might code a configure test for HAVE_WRITE. HAVE_* macros should only be used in a conditional way like #ifdef HAVE_FOO /* assume foo is available */ else /* assume foo is missing and code around its absence */ endif or as in the case of fsync() in posixmodule.c #ifdef HAVE_FSYNC /* define wrapper for fsync() endif The lack of a HAVE_foo macro definition might make an entire module unavailable. More often it causes a single module function to not be defined or for a messier or less efficient implementation to be used. A rather extreme case was strptime(). It's always been known to not be available everywhere, and in cases where it was available it didn't always present the same semantics to the Python programmer. You can to the rescue and removed the need for that macro altogether (I think it's too ill-defined for the interpreter itself to rely on a consistent definition). Skip From tim.one at comcast.net Tue Sep 30 09:45:18 2003 From: tim.one at comcast.net (Tim Peters) Date: Tue Sep 30 09:45:25 2003 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Tools/scripts md5sum.py, 1.3, 1.3.8.1 In-Reply-To: <16249.34285.353517.140250@montanaro.dyndns.org> Message-ID: > ! -b : read files in binary mode (default) > ! -t : read files in text mode (you almost certainly don't want > this!) > > When is -t useful? I don't know that it ever is. md5sum.py before 2.3 didn't have such an option, and always used binary mode. md5sum.py in 2.3 suddenly grew this option, and *defaulted* to text mode, which is a disaster for normal usage on Windows. On the CVS head I switched the default back to binary mode, but left -t in since we had released a version supporting a -t option (and as the default, no less). I can invent a use case for -t (a text file checksum independent of platform line-end convention), but it's strained (it's never been a real use case in my protracted life ). From skip at pobox.com Tue Sep 30 09:53:56 2003 From: skip at pobox.com (Skip Montanaro) Date: Tue Sep 30 09:56:21 2003 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Tools/scripts md5sum.py, 1.3, 1.3.8.1 In-Reply-To: References: <16249.34285.353517.140250@montanaro.dyndns.org> Message-ID: <16249.35572.260244.137333@montanaro.dyndns.org> Tim> ... (it's never been a real use case in my protracted life ). On the other hand, you never said you were good at geometry. (Sorry for the bad pun.) Skip From guido at python.org Tue Sep 30 10:11:52 2003 From: guido at python.org (Guido van Rossum) Date: Tue Sep 30 10:12:53 2003 Subject: [Python-Dev] Lib/idlelib/NEWS.txt In-Reply-To: Your message of "Tue, 30 Sep 2003 17:01:30 +1000." <200309300701.h8U71VG7017578@localhost.localdomain> References: <200309300701.h8U71VG7017578@localhost.localdomain> Message-ID: <200309301411.h8UEBqv03072@12-236-54-216.client.attbi.com> > Idle still has it's own NEWS.txt file, and it's own version number > (for instance, 2.3.1's version of idle is "1.0.1". Does this make > sense? What should happen for 2.3.2, where no idle changes have been > made? I thin it's appropriate for IDLE to have its own version, news etc. Many modules in the stdlib have their own __version__ and docstring. --Guido van Rossum (home page: http://www.python.org/~guido/) From anthony at interlink.com.au Tue Sep 30 10:39:36 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Sep 30 10:41:37 2003 Subject: [Python-Dev] RELEASED Python 2.3.2, release candidate 1 In-Reply-To: <200309301037.h8UAbGhY029290@localhost.localdomain> Message-ID: <200309301439.h8UEdaJv009639@localhost.localdomain> As previously mentioned, I plan to do the 2.3.2 final release on Friday morning AU time. Thomas will probably be doing the windows installer on Thursday night (EU time), so please test test and test again and get your feedback in before then. I've opened bugs against most of the problems I've found on the HP testdrive systems (which covers a lot of systems). Anthony -- Anthony Baxter It's never too late to have a happy childhood. From theller at python.net Tue Sep 30 12:29:06 2003 From: theller at python.net (Thomas Heller) Date: Tue Sep 30 12:29:33 2003 Subject: [Python-Dev] OpenSSL vulnerability Message-ID: <7k3q8ckt.fsf@python.net> Is this enough reason to use OpenSSL version 0.9.7c instead of 0.9.7b for the 2.3.2 final windows installer, or should the release candidate remain unchanged? Thomas From skip at pobox.com Tue Sep 30 12:58:39 2003 From: skip at pobox.com (Skip Montanaro) Date: Tue Sep 30 12:59:00 2003 Subject: [Python-Dev] OpenSSL vulnerability In-Reply-To: <7k3q8ckt.fsf@python.net> References: <7k3q8ckt.fsf@python.net> Message-ID: <16249.46655.874245.937560@montanaro.dyndns.org> Thomas> Is this enough reason to use OpenSSL version 0.9.7c instead of Thomas> 0.9.7b for the 2.3.2 final windows installer, or should the Thomas> release candidate remain unchanged? Thomas> At this point I'm inclined to let it go. There are many other vulnerable SS[LH targets out there, and you can't wait forever until the OpenSS[LH] folks stop emitting patches. Skip From anthony at interlink.com.au Tue Sep 30 13:12:00 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Sep 30 13:14:06 2003 Subject: [Python-Dev] OpenSSL vulnerability In-Reply-To: <7k3q8ckt.fsf@python.net> Message-ID: <200309301712.h8UHC0sJ012578@localhost.localdomain> >>> Thomas Heller wrote > Is this enough reason to use OpenSSL version 0.9.7c instead of 0.9.7b > for the 2.3.2 final windows installer, or should the release candidate > remain unchanged? > > I'd say build with the patched libraries, but only if you can test them before then. OTOH, most of the advisory seems to be about server-side problems, and the inbuilt SSL stuff in python is client stuff. go-not-to-australians-for-answers-for-they-will-answer-both-yes-and-no, Anthony -- Anthony Baxter It's never too late to have a happy childhood. From fdrake at acm.org Tue Sep 30 17:10:38 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue Sep 30 17:10:59 2003 Subject: [Python-Dev] Documentation packages Message-ID: <16249.61774.732640.328478@grendel.zope.com> After a brief discussion on the Doc-SIG, it looks like I can reasonably drop the .tar.gz packaging for the documentation, leaving only .zip and .tar.bz2 formats. Are there any strong objections to this change? (If this flies here, I'll ask the larger community, but I wanted to get and idea of the reaction here first.) -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From bac at OCF.Berkeley.EDU Tue Sep 30 17:41:12 2003 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Tue Sep 30 17:41:18 2003 Subject: [Python-Dev] Good way of finding out what C functions we have? In-Reply-To: <16249.33065.797263.217111@montanaro.dyndns.org> References: <3F779BA6.6050407@ocf.berkeley.edu> <3F78E57E.8090403@ocf.berkeley.edu> <16249.33065.797263.217111@montanaro.dyndns.org> Message-ID: <3F79F878.4070805@ocf.berkeley.edu> Skip Montanaro wrote: > >> I would look in pyconfig.h.in as the first thing. There is a pitfall, > >> though: presence of some HAVE_foo in pyconfig.h does not mean that > >> Python gracefully deals with the absence of foo. When converting > >> configure to autoconf 2.5x, I checked all macros to determine whether > >> they were still in use, but some may have become unused/not processed > >> correctly since. That would be a bug, of course. > > Brett> What does happen if a HAVE_foo is actually required? Does the > Brett> build fail or will configure halt and say that the build will > Brett> fail if you proceed? > > It's not exactly clear what you mean by "required". As in Python will not be at all functional without the functionality being required. > All the HAVE_foo macros > are supposed to indicate the presence (when defined) or absence (when not > defined) of a particular "feature" which is not felt by one or more of the > authors to be universally available. As far as I know, a HAVE_foo macro > should never be required to be defined, though I suppose if we had an > infinite amount of time on our hands be might code a configure test for > HAVE_WRITE. HAVE_* macros should only be used in a conditional way like > > #ifdef HAVE_FOO > /* assume foo is available */ > else > /* assume foo is missing and code around its absence */ > endif > > or as in the case of fsync() in posixmodule.c > > #ifdef HAVE_FSYNC > /* define wrapper for fsync() > endif > > The lack of a HAVE_foo macro definition might make an entire module > unavailable. More often it causes a single module function to not be > defined or for a messier or less efficient implementation to be used. > Right. I was wondering if there are any checks in configure.in that if something was not available that Python itself would not compile *at all*. I would suspect not since that is what the ANSI C/POSIX coding requirement is supposed to handle, right? -Brett From martin at v.loewis.de Tue Sep 30 17:58:07 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Sep 30 17:58:18 2003 Subject: [Python-Dev] Good way of finding out what C functions we have? In-Reply-To: <3F78E57E.8090403@ocf.berkeley.edu> References: <3F779BA6.6050407@ocf.berkeley.edu> <3F78E57E.8090403@ocf.berkeley.edu> Message-ID: "Brett C." writes: > Shows how much I know; I found that last night but since it didn't say > POSIX anywhere I thought it was part of another standard. I guess the > "Issue 5" mention means that BASE is POSIX? Or is IEEE 1003.1 > basically POSIX? In the past, the various standards different slightly. Today, IEEE 1003.1 is the same as ISO/IEC 9945:2003 (parts 1..4), is the same as Single Unix version 3, is the same as XPG/5. It contains the OpenGroup's XBD, XSH, XCU, and XRAT documents. The UNIX product standard (which apparently hasn't been revised since '98) also covers XCURSES, XTI, and a few others; UNIX98 workstation also supports X11 and CDE. > What does happen if a HAVE_foo is actually required? Does the build > fail or will configure halt and say that the build will fail if you > proceed? configure only checks for the feature. If the feature is absent, yet we use the feature without an #ifdef, the build just fails; that would be a bug. Regards, Martin From fdrake at acm.org Tue Sep 30 18:11:50 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue Sep 30 18:13:04 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: <2mzngr1bf2.fsf@starship.python.net> References: <200309261752.h8QHqJgD029823@localhost.localdomain> <20030926184853.GB22837@mems-exchange.org> <2mzngr1bf2.fsf@starship.python.net> Message-ID: <16249.65446.699521.103706@grendel.zope.com> Michael Hudson writes: > It occurs to me that I don't know *why* Fred is so much the > documentation man; I've not had any trouble processing the docs into > HTML lately (haven't tried on Windows, admittedly, and I haven't > tried to make info ever). It's certainly gotten easier to deal with the documentation on modern Linux distributions. At CNRI, we used mostly Solaris boxes, and I have to build my own teTeX installations from source, and hand-select a version of LaTeX2HTML that worked for me. At this point, all the software that I can't just install from a RedHat CD is part of what gets pulled down from CVS. I've been able to build the docs on Cygwin as well, though I've not tried lately. A lot of what it takes to build the docs is written into Doc/Makefile, but it does require a solid make (it even uses $(shell ...) now, so maybe only GNU make will do; not sure). > What else needs to be done? There must be quite a bit of mucking > about on creosote to do, I guess. There's a bit, but that's getting easier and easier as I've gone through it a few times now. I updated PEP 101 the other evening so anyone can do what's needed to build the packages and get them in the download locations. There's more to be written to explain what else needs to be updated on the site. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From fdrake at acm.org Tue Sep 30 18:15:51 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue Sep 30 18:16:04 2003 Subject: [Python-Dev] Re: RELEASED Python 2.3.1 In-Reply-To: References: <200309261752.h8QHqJgD029823@localhost.localdomain> <20030926184853.GB22837@mems-exchange.org> Message-ID: <16250.151.309386.606310@grendel.zope.com> Thomas Heller writes: > I suggest to update the version numbers in these files to the next > version *after* a release is done. So, in the release23-maint branch > the version numbers could *now* be adjusted to 2.3.2. The Doc/ tree no longer depends on version numbers stored in the Doc/ tree; the makefile causes the script Doc/tools/getversioninfo to pull the version numbers from Include/patchlevel.h. > And it would help if I could build the HTML docs myself from CVS. I did > manage to create the pdf files with TeTex under windows, but I didn't > succeed with the html pages so far. Are you using Cygwin? What problems did you encounter? I'll help if I can; I have a Windows machine (sometimes), but don't know anything about non-Cygwin Windows TeX systems (and I don't have Cygwin installed most of the time). -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tree at basistech.com Tue Sep 30 18:33:50 2003 From: tree at basistech.com (Tom Emerson) Date: Tue Sep 30 18:36:25 2003 Subject: [Python-Dev] imp.findmodule and zip files Message-ID: <16250.1230.305462.233362@magrathea.basistech.com> Should imp.find_module() work for modules that are packaged in a zip file in 2.3.x? I'm seeing that this doesn't, and before I dive in to figure out why, I want to see if this is the intent. -- Tom Emerson Basis Technology Corp. Software Architect http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever"